Module org.elasticsearch.server

Package org.elasticsearch.index.codec.vectors.diskbbq.next

Class ESNextDiskBBQVectorsFormat

java.lang.Object

org.apache.lucene.codecs.KnnVectorsFormat

org.elasticsearch.index.codec.vectors.diskbbq.next.ESNextDiskBBQVectorsFormat

All Implemented Interfaces:

org.apache.lucene.util.NamedSPILoader.NamedSPI

public class ESNextDiskBBQVectorsFormat
extends org.apache.lucene.codecs.KnnVectorsFormat

Codec format for Inverted File Vector indexes. This index expects to break the dimensional space into clusters and assign each vector to a cluster generating a posting list of vectors. Clusters are represented by centroids. The vector quantization format used here is a per-vector optimized scalar quantization. Also see OptimizedScalarQuantizer. Some of key features are: The format is stored in three files:

.cenivf (centroid data) file

Which stores the raw and quantized centroid vectors.

.clivf (cluster data) file

Stores the quantized vectors for each cluster, inline and stored in blocks. Additionally, the docIds of each vector is stored.

.mivf (centroid metadata) file

Stores metadata including the number of centroids and their offsets in the clivf file

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static enum	ESNextDiskBBQVectorsFormat.QuantEncoding

Field Summary

Fields

Modifier and Type	Field	Description
static final String	CENTROID_EXTENSION
static final String	CLUSTER_EXTENSION
static final int	DEFAULT_CENTROIDS_PER_PARENT_CLUSTER
static final int	DEFAULT_PRECONDITIONING_BLOCK_DIMENSION
static final int	DEFAULT_VECTORS_PER_CLUSTER
static final float	DYNAMIC_VISIT_RATIO
static final String	IVF_META_EXTENSION
static final int	MAX_CENTROIDS_PER_PARENT_CLUSTER
static final int	MAX_DIMENSIONS
static final int	MAX_PRECONDITIONING_BLOCK_DIMS
static final int	MAX_VECTORS_PER_CLUSTER
static final int	MIN_CENTROIDS_PER_PARENT_CLUSTER
static final int	MIN_PRECONDITIONING_BLOCK_DIMS
static final int	MIN_VECTORS_PER_CLUSTER
static final String	NAME
static final int	VERSION_CURRENT
static final int	VERSION_DIRECT_IO
static final int	VERSION_START

Fields inherited from class org.apache.lucene.codecs.KnnVectorsFormat

DEFAULT_MAX_DIMENSIONS, EMPTY

Constructor Summary

Constructors

Constructor	Description
ESNextDiskBBQVectorsFormat()	Constructs a format using the given graph construction parameters and scalar quantization.
ESNextDiskBBQVectorsFormat(int vectorPerCluster, int centroidsPerParentCluster)
ESNextDiskBBQVectorsFormat(ESNextDiskBBQVectorsFormat.QuantEncoding quantEncoding, int vectorPerCluster, int centroidsPerParentCluster)
ESNextDiskBBQVectorsFormat(ESNextDiskBBQVectorsFormat.QuantEncoding quantEncoding, int vectorPerCluster, int centroidsPerParentCluster, DenseVectorFieldMapper.ElementType elementType, boolean useDirectIO, ExecutorService mergingExecutorService, int maxMergingWorkers, boolean doPrecondition, int preconditioningBlockDimension)
ESNextDiskBBQVectorsFormat(ESNextDiskBBQVectorsFormat.QuantEncoding quantEncoding, int vectorPerCluster, int centroidsPerParentCluster, DenseVectorFieldMapper.ElementType elementType, boolean useDirectIO, ExecutorService mergingExecutorService, int maxMergingWorkers, boolean doPrecondition, int preconditioningBlockDimension, int flatVectorThreshold)

Method Summary

Modifier and Type	Method	Description
static int	defaultFlatThreshold(int configuredClusterSize)	Returns the default flat index threshold for the given cluster size.
org.apache.lucene.codecs.KnnVectorsReader	fieldsReader(org.apache.lucene.index.SegmentReadState state)
org.apache.lucene.codecs.KnnVectorsWriter	fieldsWriter(org.apache.lucene.index.SegmentWriteState state)
int	getMaxDimensions(String fieldName)
String	toString()

Methods inherited from class org.apache.lucene.codecs.KnnVectorsFormat

availableKnnVectorsFormats, forName, getName, reloadKnnVectorsFormat

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

NAME

public static final String NAME

See Also:

• Constant Field Values

CENTROID_EXTENSION

public static final String CENTROID_EXTENSION

See Also:

• Constant Field Values

CLUSTER_EXTENSION

public static final String CLUSTER_EXTENSION

See Also:

• Constant Field Values

IVF_META_EXTENSION

public static final String IVF_META_EXTENSION

See Also:

• Constant Field Values

VERSION_START

public static final int VERSION_START

See Also:

• Constant Field Values

VERSION_DIRECT_IO

public static final int VERSION_DIRECT_IO

See Also:

• Constant Field Values

VERSION_CURRENT

public static final int VERSION_CURRENT

See Also:

• Constant Field Values

DYNAMIC_VISIT_RATIO

public static final float DYNAMIC_VISIT_RATIO

See Also:

• Constant Field Values

DEFAULT_VECTORS_PER_CLUSTER

public static final int DEFAULT_VECTORS_PER_CLUSTER

See Also:

• Constant Field Values

MIN_VECTORS_PER_CLUSTER

public static final int MIN_VECTORS_PER_CLUSTER

See Also:

• Constant Field Values

MAX_VECTORS_PER_CLUSTER

public static final int MAX_VECTORS_PER_CLUSTER

See Also:

• Constant Field Values

DEFAULT_CENTROIDS_PER_PARENT_CLUSTER

public static final int DEFAULT_CENTROIDS_PER_PARENT_CLUSTER

See Also:

• Constant Field Values

MIN_CENTROIDS_PER_PARENT_CLUSTER

public static final int MIN_CENTROIDS_PER_PARENT_CLUSTER

See Also:

• Constant Field Values

MAX_CENTROIDS_PER_PARENT_CLUSTER

public static final int MAX_CENTROIDS_PER_PARENT_CLUSTER

See Also:

• Constant Field Values

DEFAULT_PRECONDITIONING_BLOCK_DIMENSION

public static final int DEFAULT_PRECONDITIONING_BLOCK_DIMENSION

See Also:

• Constant Field Values

MIN_PRECONDITIONING_BLOCK_DIMS

public static final int MIN_PRECONDITIONING_BLOCK_DIMS

See Also:

• Constant Field Values

MAX_PRECONDITIONING_BLOCK_DIMS

public static final int MAX_PRECONDITIONING_BLOCK_DIMS

See Also:

• Constant Field Values

MAX_DIMENSIONS

public static final int MAX_DIMENSIONS

See Also:

• Constant Field Values

Constructor Details

ESNextDiskBBQVectorsFormat

public ESNextDiskBBQVectorsFormat(int vectorPerCluster,
 int centroidsPerParentCluster)

ESNextDiskBBQVectorsFormat

public ESNextDiskBBQVectorsFormat(ESNextDiskBBQVectorsFormat.QuantEncoding quantEncoding,
 int vectorPerCluster,
 int centroidsPerParentCluster)

ESNextDiskBBQVectorsFormat

public ESNextDiskBBQVectorsFormat(ESNextDiskBBQVectorsFormat.QuantEncoding quantEncoding,
 int vectorPerCluster,
 int centroidsPerParentCluster,
 DenseVectorFieldMapper.ElementType elementType,
 boolean useDirectIO,
 ExecutorService mergingExecutorService,
 int maxMergingWorkers,
 boolean doPrecondition,
 int preconditioningBlockDimension)

ESNextDiskBBQVectorsFormat

public ESNextDiskBBQVectorsFormat(ESNextDiskBBQVectorsFormat.QuantEncoding quantEncoding,
 int vectorPerCluster,
 int centroidsPerParentCluster,
 DenseVectorFieldMapper.ElementType elementType,
 boolean useDirectIO,
 ExecutorService mergingExecutorService,
 int maxMergingWorkers,
 boolean doPrecondition,
 int preconditioningBlockDimension,
 int flatVectorThreshold)

ESNextDiskBBQVectorsFormat

public ESNextDiskBBQVectorsFormat()

Constructs a format using the given graph construction parameters and scalar quantization.

Method Details

defaultFlatThreshold

public static int defaultFlatThreshold(int configuredClusterSize)

Returns the default flat index threshold for the given cluster size.

Parameters:

configuredClusterSize - the configured cluster size

Returns:

the default flat index threshold

fieldsWriter

public org.apache.lucene.codecs.KnnVectorsWriter fieldsWriter(org.apache.lucene.index.SegmentWriteState state)
                                                       throws IOException

Specified by:

fieldsWriter in class org.apache.lucene.codecs.KnnVectorsFormat

Throws:

IOException

fieldsReader

public org.apache.lucene.codecs.KnnVectorsReader fieldsReader(org.apache.lucene.index.SegmentReadState state)
                                                       throws IOException

Specified by:

fieldsReader in class org.apache.lucene.codecs.KnnVectorsFormat

Throws:

IOException

getMaxDimensions

public int getMaxDimensions(String fieldName)

Specified by:

getMaxDimensions in class org.apache.lucene.codecs.KnnVectorsFormat

toString

public String toString()

Overrides:

toString in class Object