Module org.elasticsearch.compute
Package org.elasticsearch.compute.data.arrow

Class AbstractArrowBufBlock<V extends Vector,B extends Block>

java.lang.Object
org.elasticsearch.compute.data.AbstractNonThreadSafeRefCounted
org.elasticsearch.compute.data.arrow.AbstractArrowBufBlock<V,B>
All Implemented Interfaces:
Closeable, AutoCloseable, org.apache.lucene.util.Accountable, Writeable, Block, org.elasticsearch.core.RefCounted, org.elasticsearch.core.Releasable, BlockLoader.Block
Direct Known Subclasses:
BooleanArrowBufBlock, BytesRefArrowBufBlock, DoubleArrowBufBlock, Float16ArrowBufBlock, FloatArrowBufBlock, Int16ArrowBufBlock, Int8ArrowBufBlock, IntArrowBufBlock, LongArrowBufBlock, LongMul1kArrowBufBlock, UInt16ArrowBufBlock, UInt32ArrowBufBlock, UInt8ArrowBufBlock
public abstract class AbstractArrowBufBlock<V extends Vector,B extends Block> extends AbstractNonThreadSafeRefCounted implements Block

  • Field Details

    • valueCount

      protected final int valueCount

    • offsetCount

      protected final int offsetCount

    • valueBuffer

      protected final org.apache.arrow.memory.ArrowBuf valueBuffer

    • validityBuffer

      @Nullable protected final org.apache.arrow.memory.ArrowBuf validityBuffer

    • offsetBuffer

      @Nullable protected final org.apache.arrow.memory.ArrowBuf offsetBuffer

    • blockFactory

      protected final BlockFactory blockFactory

    • closed

      protected boolean closed

  • Constructor Details

    • AbstractArrowBufBlock

      public AbstractArrowBufBlock(org.apache.arrow.memory.ArrowBuf valueBuffer, @Nullable org.apache.arrow.memory.ArrowBuf validityBuffer, @Nullable org.apache.arrow.memory.ArrowBuf offsetBuffer, int valueCount, int offsetCount, BlockFactory blockFactory)
      Create an ArrowBuf block based on the constituents of an Arrow ValueVector. The caller must retain the buffers if they are shared with other blocks or Arrow vectors.

    • AbstractArrowBufBlock

      protected AbstractArrowBufBlock(org.apache.arrow.vector.ValueVector arrowVector, BlockFactory blockFactory)

  • Method Details

    • retainBuffers

      public void retainBuffers()
      Retains (increments the reference count) this block's Arrow buffers

    • releaseBuffers

      public void releaseBuffers()
      Releases (decrements the reference count) this block's Arrow buffers

    • byteSize

      protected abstract int byteSize()

    • vectorConstructor

      protected abstract ArrowBufVectorConstructor<V> vectorConstructor()

    • blockConstructor

      protected abstract ArrowBufBlockConstructor<B> blockConstructor()

    • validityBufferLength

      protected static int validityBufferLength(int positions)

    • setValidityBit

      protected static void setValidityBit(org.apache.arrow.memory.ArrowBuf buf, int position)

    • closeInternal

      protected void closeInternal()
      Description copied from class: AbstractNonThreadSafeRefCounted
      This is called when the number of references reaches zero. This is where resources should be released (adjusting circuit breakers if needed).
      Specified by:
      closeInternal in class AbstractNonThreadSafeRefCounted

    • asVector

      public V asVector()
      Description copied from interface: Block
      Returns an efficient dense single-value view of this block. Null, if the block is not dense single-valued. That is, if mayHaveNulls returns true, or getTotalValueCount is not equal to getPositionCount.
      Specified by:
      asVector in interface Block
      Returns:
      an efficient dense single-value view of this block

    • blockFactory

      public BlockFactory blockFactory()
      Description copied from interface: Block
      The block factory associated with this block.
      Specified by:
      blockFactory in interface Block

    • allowPassingToDifferentDriver

      public void allowPassingToDifferentDriver()
      Description copied from interface: Block
      Before passing a Block to another Driver, it is necessary to switch the owning block factory to its parent, which is associated with the global circuit breaker. This ensures that when the new driver releases this Block, it returns memory directly to the parent block factory instead of the local block factory of this Block. This is important because the local block factory is not thread safe and doesn't support simultaneous access by more than one thread.
      Specified by:
      allowPassingToDifferentDriver in interface Block

    • isNull

      public boolean isNull(int position)
      Specified by:
      isNull in interface Block
      Parameters:
      position - the position
      Returns:
      true if the value stored at the given position is null, false otherwise

    • mayHaveNulls

      public boolean mayHaveNulls()
      Specified by:
      mayHaveNulls in interface Block
      Returns:
      true if some values might be null. False, if all values are guaranteed to be not null.

    • areAllValuesNull

      public boolean areAllValuesNull()
      Specified by:
      areAllValuesNull in interface Block
      Returns:
      true if all values in this block are guaranteed to be null.

    • mayHaveMultivaluedFields

      public boolean mayHaveMultivaluedFields()
      Description copied from interface: Block
      Can this block have multivalued fields? Blocks that return false will never return more than one from Block.getValueCount(int). This may return true for Blocks that do not have multivalued fields, but it will always answer quickly.
      Specified by:
      mayHaveMultivaluedFields in interface Block

    • doesHaveMultivaluedFields

      public boolean doesHaveMultivaluedFields()
      Description copied from interface: Block
      Does this block have multivalued fields? Unlike Block.mayHaveMultivaluedFields() this will never return a false positive. In other words, if this returns true then there are positions for which Block.getValueCount(int) will return more than 1. This will answer quickly if it can but may have to check all positions.
      Specified by:
      doesHaveMultivaluedFields in interface Block

    • mvOrdering

      public Block.MvOrdering mvOrdering()
      Description copied from interface: Block
      How are multivalued fields ordered?
      Specified by:
      mvOrdering in interface Block

    • ramBytesUsed

      public long ramBytesUsed()
      Specified by:
      ramBytesUsed in interface org.apache.lucene.util.Accountable

    • filter

      public B filter(boolean mayContainDuplicates, int... positions)
      Description copied from interface: Block
      Creates a new block that only exposes the positions provided.
      Specified by:
      filter in interface Block
      Parameters:
      mayContainDuplicates - may the positions array contain duplicate positions?
      positions - the positions to retain
      Returns:
      a filtered block

    • keepMask

      public B keepMask(BooleanVector mask)
      Description copied from interface: Block
      Build a Block with the same values as this Block, but replacing all values for which mask.getBooleanValue(position) returns false with null. The mask vector must be at least as long as this Block.
      Specified by:
      keepMask in interface Block

    • lookup

      public org.elasticsearch.core.ReleasableIterator<? extends B> lookup(IntBlock positions, ByteSizeValue targetBlockSize)
      Description copied from interface: Block
      Builds an Iterator of new Blocks with the same Block.elementType() as this Block whose values are copied from positions in this Block. It has the same number of positions as the positions parameter.

      For example, if this block contained [a, b, [b, c]] and were called with the block [0, 1, 1, [1, 2]] then the result would be [a, b, b, [b, b, c]].

      This process produces count(this) * count(positions) values per positions which could be quite large. Instead of returning a single Block, this returns an Iterator of Blocks containing all of the promised values.

      The returned ReleasableIterator may retain a reference to the positions parameter. Close it to release those references.

      This block is built using the same BlockFactory as was used to build the positions parameter.

      Specified by:
      lookup in interface Block

    • expand

      public B expand()
      Description copied from interface: Block
      Expand multivalued fields into one row per value. Returns the same block if there aren't any multivalued fields to expand. The returned block needs to be closed by the caller to release the block's resources.
      Specified by:
      expand in interface Block

    • getPositionCount

      public int getPositionCount()
      Description copied from interface: Block
      Returns the number of positions (rows) in this block. See class javadoc for the usual way to iterate these positions.
      Specified by:
      getPositionCount in interface Block
      Returns:
      the number of positions (rows) in this block

    • getFirstValueIndex

      public int getFirstValueIndex(int position)
      Description copied from interface: Block
      Returns the index of the first value for the given position. See class javadoc for the usual way to iterate these positions.

      For densely packed data this will return its parameter unchanged. For fields with null values or multivalued fields, this will shift. Here's an example: 

      
     0   <---+
     1       | Values at first position
     2       |
     3   <---+
     5   <---- Value at second position
     6   <---+ Values at third position
     7   <---+

      This represents three rows. The first has the value [0, 1, 2, 3]. The second has the value 5. The third has the value [6, 7]. This method will return 0 for the first position, 4 for the second, and 5 for the third.

      Specified by:
      getFirstValueIndex in interface Block
      Returns:
      the index of the first value for the given position

    • getValueCount

      public int getValueCount(int position)
      Description copied from interface: Block
      Returns the number of values for the given position. See class javadoc for the usual way to iterate these positions.

      For densely packed data this will return 1. For nulls this will return 0. For multivalued fields, this will return the number of values. Here's an example: 

      
     0   <---+
     1       | Values at first position
     2       |
     3   <---+
     5   <---- Value at second position
     6   <---+ Values at third position
     7   <---+

      This represents three rows. The first has the value [0, 1, 2, 3]. The second has the value 5. The third has the value [6, 7]. This method will return 4 for the first position, 1 for the second, and 2 for the third.

      Specified by:
      getValueCount in interface Block
      Returns:
      the number of values for the given position

    • getTotalValueCount

      public int getTotalValueCount()
      Description copied from interface: Block
      Returns the total number of values in this block not counting nulls. This powers the COUNT aggregation and is used to report the number of fields loaded by ESQL.
      Specified by:
      getTotalValueCount in interface Block
      Returns:
      the total number of values in this block not counting nulls