Package org.elasticsearch.xpack.esql.datasources

Class AsyncExternalSourceBuffer

java.lang.Object
org.elasticsearch.xpack.esql.datasources.AsyncExternalSourceBuffer
public final class AsyncExternalSourceBuffer extends Object
Thread-safe buffer for async external source data. Modeled after ExchangeBuffer. This buffer provides: - Thread-safe page queue for cross-thread communication - Byte-based backpressure control proportional to actual memory usage - Notification via SubscribableListener when data becomes available - Lifecycle management (finished state tracking)

  • Field Details

    • DEFAULT_MAX_BUFFER_BYTES

      public static final long DEFAULT_MAX_BUFFER_BYTES
      Default byte limit for the buffer, preserving the original "10 normal-sized pages" intent.

  • Constructor Details

    • AsyncExternalSourceBuffer

      public AsyncExternalSourceBuffer(long maxBufferBytes)

  • Method Details

    • addPage

      public void addPage(Page page)
      Add a page to the buffer. Called by the background reader thread.

    • pollPage

      public Page pollPage()
      Poll a page from the buffer. Called by the operator (driver thread).
      Returns:
      the next page, or null if no pages available

    • waitForWriting

      public IsBlockedResult waitForWriting()
      Returns an IsBlockedResult that completes when the buffer has space for writing. Used by background reader for backpressure.

    • waitForSpace

      public SubscribableListener<Void> waitForSpace()
      Returns a SubscribableListener that completes when the buffer has space for writing. This is the preferred method for producers to use for backpressure coordination.

      Unlike waitForWriting() which returns an IsBlockedResult, this method returns a SubscribableListener that can be used directly with ES async patterns.

      Returns:
      a listener that completes when space is available, or an already-completed listener if space exists

    • waitForReading

      public IsBlockedResult waitForReading()
      Returns an IsBlockedResult that completes when the buffer has data for reading. Used by operator to signal driver when waiting for data.

    • finish

      public void finish(boolean drainingPages)
      Mark the buffer as finished. Called when reading is done or an error occurs.

    • onFailure

      public void onFailure(Throwable t)
      Mark the buffer as failed. Called when the background reader encounters an error.

    • isFinished

      public boolean isFinished()

    • noMoreInputs

      public boolean noMoreInputs()

    • size

      public int size()

    • addCompletionListener

      public void addCompletionListener(ActionListener<Void> listener)
      Adds a listener that will be notified when this buffer is finished.

    • failure

      public Throwable failure()

    • bytesInBuffer

      public long bytesInBuffer()
      Returns the current number of bytes buffered, as measured by Page.ramBytesUsedByBlocks().