Module org.elasticsearch.server

Package org.elasticsearch.index.codec.vectors.diskbbq.es94

Class ES940DiskBBQVectorsWriter

java.lang.Object

org.apache.lucene.codecs.KnnVectorsWriter

org.elasticsearch.index.codec.vectors.diskbbq.IVFVectorsWriter

org.elasticsearch.index.codec.vectors.diskbbq.es94.ES940DiskBBQVectorsWriter

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.lucene.util.Accountable

public class ES940DiskBBQVectorsWriter
extends IVFVectorsWriter

Default implementation of IVFVectorsWriter. It uses HierarchicalKMeans algorithm to partition the vector space, and then stores the centroids and posting list in a sequential fashion.

Nested Class Summary

Nested classes/interfaces inherited from class org.elasticsearch.index.codec.vectors.diskbbq.IVFVectorsWriter

IVFVectorsWriter.CentroidOffsetAndLength

Nested classes/interfaces inherited from class org.apache.lucene.codecs.KnnVectorsWriter

org.apache.lucene.codecs.KnnVectorsWriter.MergedVectorValues

Field Summary

Fields inherited from interface org.apache.lucene.util.Accountable

NULL_ACCOUNTABLE

Constructor Summary

Constructors

Constructor

Description

ES940DiskBBQVectorsWriter(org.apache.lucene.index.SegmentWriteState state, String rawVectorFormatName, boolean useDirectIOReads, org.apache.lucene.codecs.hnsw.FlatVectorsWriter rawVectorDelegate, ES940DiskBBQVectorsFormat.QuantEncoding encoding, int vectorPerCluster, int centroidsPerParentCluster, org.apache.lucene.search.TaskExecutor mergeExec, int numMergeWorkers, int blockDimension, boolean doPrecondition, int flatVectorThreshold)

Method Summary

Modifier and Type	Method	Description
IVFVectorsWriter.CentroidOffsetAndLength	buildAndWritePostingsLists(org.apache.lucene.index.FieldInfo fieldInfo, CentroidSupplier centroidSupplier, org.apache.lucene.index.FloatVectorValues floatVectorValues, org.apache.lucene.store.IndexOutput postingsOutput, long fileOffset, int[] assignments, int[] overspillAssignments)
IVFVectorsWriter.CentroidOffsetAndLength	buildAndWritePostingsLists(org.apache.lucene.index.FieldInfo fieldInfo, CentroidSupplier centroidSupplier, org.apache.lucene.index.FloatVectorValues floatVectorValues, org.apache.lucene.store.IndexOutput postingsOutput, long fileOffset, org.apache.lucene.index.MergeState mergeState, int[] assignments, int[] overspillAssignments)
CentroidAssignments	calculateCentroids(org.apache.lucene.index.FieldInfo fieldInfo, KMeansFloatVectorValues floatVectorValues)	Calculate the centroids for the given field.
CentroidAssignments	calculateCentroids(org.apache.lucene.index.FieldInfo fieldInfo, KMeansFloatVectorValues floatVectorValues, org.apache.lucene.index.MergeState mergeState)
CentroidSupplier	createCentroidSupplier(org.apache.lucene.index.FieldInfo info, float[][] centroids, float[] globalCentroid)
CentroidSupplier	createCentroidSupplier(org.apache.lucene.store.IndexInput centroidsInput, int numCentroids, org.apache.lucene.index.FieldInfo fieldInfo, float[] globalCentroid)
protected Preconditioner	createPreconditioner(int dimension)
protected void	doWriteMeta(org.apache.lucene.store.IndexOutput metaOutput, org.apache.lucene.index.FieldInfo field, int numCentroids, long preconditionerOffset, long preconditionerLength)
protected Preconditioner	inheritPreconditioner(org.apache.lucene.index.FieldInfo fieldInfo, org.apache.lucene.index.MergeState mergeState)
protected Consumer<List<float[]>>	preconditionVectors(Preconditioner preconditioner)
protected org.apache.lucene.index.FloatVectorValues	preconditionVectors(Preconditioner preconditioner, org.apache.lucene.index.FloatVectorValues vectors)
void	writeCentroids(org.apache.lucene.index.FieldInfo fieldInfo, CentroidSupplier centroidSupplier, int[] centroidAssignments, float[] globalCentroid, IVFVectorsWriter.CentroidOffsetAndLength centroidOffsetAndLength, org.apache.lucene.store.IndexOutput centroidOutput)
void	writeCentroids(org.apache.lucene.index.FieldInfo fieldInfo, CentroidSupplier centroidSupplier, int[] centroidAssignments, float[] globalCentroid, IVFVectorsWriter.CentroidOffsetAndLength centroidOffsetAndLength, org.apache.lucene.store.IndexOutput centroidOutput, org.apache.lucene.index.MergeState mergeState)
protected void	writePreconditioner(Preconditioner preconditioner, org.apache.lucene.store.IndexOutput out)

Methods inherited from class org.elasticsearch.index.codec.vectors.diskbbq.IVFVectorsWriter

addField, close, finish, flush, mergeOneField, ramBytesUsed

Methods inherited from class org.apache.lucene.codecs.KnnVectorsWriter

mapOldOrdToNewOrd, merge

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.lucene.util.Accountable

getChildResources

Constructor Details

ES940DiskBBQVectorsWriter

public ES940DiskBBQVectorsWriter(org.apache.lucene.index.SegmentWriteState state,
 String rawVectorFormatName,
 boolean useDirectIOReads,
 org.apache.lucene.codecs.hnsw.FlatVectorsWriter rawVectorDelegate,
 ES940DiskBBQVectorsFormat.QuantEncoding encoding,
 int vectorPerCluster,
 int centroidsPerParentCluster,
 org.apache.lucene.search.TaskExecutor mergeExec,
 int numMergeWorkers,
 int blockDimension,
 boolean doPrecondition,
 int flatVectorThreshold)
                          throws IOException

Throws:

IOException

Method Details

inheritPreconditioner

protected Preconditioner inheritPreconditioner(org.apache.lucene.index.FieldInfo fieldInfo,
 org.apache.lucene.index.MergeState mergeState)
                                        throws IOException

Specified by:

inheritPreconditioner in class IVFVectorsWriter

Throws:

IOException

createPreconditioner

protected Preconditioner createPreconditioner(int dimension)

Specified by:

createPreconditioner in class IVFVectorsWriter

writePreconditioner

protected void writePreconditioner(Preconditioner preconditioner,
 org.apache.lucene.store.IndexOutput out)
                            throws IOException

Specified by:

writePreconditioner in class IVFVectorsWriter

Throws:

IOException

preconditionVectors

protected Consumer<List<float[]>> preconditionVectors(Preconditioner preconditioner)

Specified by:

preconditionVectors in class IVFVectorsWriter

preconditionVectors

protected org.apache.lucene.index.FloatVectorValues preconditionVectors(Preconditioner preconditioner,
 org.apache.lucene.index.FloatVectorValues vectors)

Specified by:

preconditionVectors in class IVFVectorsWriter

buildAndWritePostingsLists

public IVFVectorsWriter.CentroidOffsetAndLength buildAndWritePostingsLists(org.apache.lucene.index.FieldInfo fieldInfo,
 CentroidSupplier centroidSupplier,
 org.apache.lucene.index.FloatVectorValues floatVectorValues,
 org.apache.lucene.store.IndexOutput postingsOutput,
 long fileOffset,
 int[] assignments,
 int[] overspillAssignments)
                                                                    throws IOException

Specified by:

buildAndWritePostingsLists in class IVFVectorsWriter

Throws:

IOException

buildAndWritePostingsLists

public IVFVectorsWriter.CentroidOffsetAndLength buildAndWritePostingsLists(org.apache.lucene.index.FieldInfo fieldInfo,
 CentroidSupplier centroidSupplier,
 org.apache.lucene.index.FloatVectorValues floatVectorValues,
 org.apache.lucene.store.IndexOutput postingsOutput,
 long fileOffset,
 org.apache.lucene.index.MergeState mergeState,
 int[] assignments,
 int[] overspillAssignments)
                                                                    throws IOException

Specified by:

buildAndWritePostingsLists in class IVFVectorsWriter

Throws:

IOException

createCentroidSupplier

public CentroidSupplier createCentroidSupplier(org.apache.lucene.store.IndexInput centroidsInput,
 int numCentroids,
 org.apache.lucene.index.FieldInfo fieldInfo,
 float[] globalCentroid)
                                        throws IOException

Specified by:

createCentroidSupplier in class IVFVectorsWriter

Throws:

IOException

createCentroidSupplier

public CentroidSupplier createCentroidSupplier(org.apache.lucene.index.FieldInfo info,
 float[][] centroids,
 float[] globalCentroid)
                                        throws IOException

Specified by:

createCentroidSupplier in class IVFVectorsWriter

Throws:

IOException

doWriteMeta

protected void doWriteMeta(org.apache.lucene.store.IndexOutput metaOutput,
 org.apache.lucene.index.FieldInfo field,
 int numCentroids,
 long preconditionerOffset,
 long preconditionerLength)
                    throws IOException

Specified by:

doWriteMeta in class IVFVectorsWriter

Throws:

IOException

writeCentroids

public void writeCentroids(org.apache.lucene.index.FieldInfo fieldInfo,
 CentroidSupplier centroidSupplier,
 int[] centroidAssignments,
 float[] globalCentroid,
 IVFVectorsWriter.CentroidOffsetAndLength centroidOffsetAndLength,
 org.apache.lucene.store.IndexOutput centroidOutput)
                    throws IOException

Specified by:

writeCentroids in class IVFVectorsWriter

Throws:

IOException

writeCentroids

public void writeCentroids(org.apache.lucene.index.FieldInfo fieldInfo,
 CentroidSupplier centroidSupplier,
 int[] centroidAssignments,
 float[] globalCentroid,
 IVFVectorsWriter.CentroidOffsetAndLength centroidOffsetAndLength,
 org.apache.lucene.store.IndexOutput centroidOutput,
 org.apache.lucene.index.MergeState mergeState)
                    throws IOException

Specified by:

writeCentroids in class IVFVectorsWriter

Throws:

IOException

calculateCentroids

public CentroidAssignments calculateCentroids(org.apache.lucene.index.FieldInfo fieldInfo,
 KMeansFloatVectorValues floatVectorValues,
 org.apache.lucene.index.MergeState mergeState)
                                       throws IOException

Specified by:

calculateCentroids in class IVFVectorsWriter

Throws:

IOException

calculateCentroids

public CentroidAssignments calculateCentroids(org.apache.lucene.index.FieldInfo fieldInfo,
 KMeansFloatVectorValues floatVectorValues)
                                       throws IOException

Calculate the centroids for the given field. We use the HierarchicalKMeans algorithm to partition the space of all vectors across merging segments

Specified by:

calculateCentroids in class IVFVectorsWriter

Parameters:

fieldInfo - merging field info

floatVectorValues - the float vector values to merge

Returns:

the vector assignments, soar assignments, and if asked the centroids themselves that were computed

Throws:

IOException - if an I/O error occurs