Package org.elasticsearch.xpack.esql.datasources.spi

Interface DataSourcePlugin

public interface DataSourcePlugin

Extension point for data source implementations. Plugins implementing this interface will be discovered by ESQL at startup via Elasticsearch's plugin discovery mechanism.

This interface allows plugins to provide:

• Storage providers (S3, GCS, Azure, HTTP) for accessing data - keyed by URI scheme
• Format readers (Parquet, CSV, ORC) for parsing data files - keyed by format name
• Table catalog connectors (Iceberg, Delta Lake) for table metadata - keyed by catalog type
• Custom operator factories for complex datasources - keyed by source type
• Filter pushdown support for predicate pushdown optimization - keyed by source type
• Decompression codecs for compound extensions (e.g. .csv.gz) - via decompressionCodecs(Settings)

All methods have default implementations returning empty maps/lists, allowing plugins to implement only the capabilities they provide.

Note: Method names follow the project convention of omitting the "get" prefix since there are no corresponding setters.

Method Summary

Modifier and Type	Method	Description
default Map<String,ConnectorFactory>	connectors(Settings settings)
default List<DecompressionCodec>	decompressionCodecs(Settings settings)	Decompression codecs this plugin provides (e.g.
default Map<String,FilterPushdownSupport>	filterPushdownSupport(Settings settings)
default Map<String,FormatReaderFactory>	formatReaders(Settings settings)
default Set<FormatSpec>	formatSpecs()	Format descriptors this plugin provides.
default List<NamedWriteableRegistry.Entry>	namedWriteables()
default Map<String,SourceOperatorFactoryProvider>	operatorFactories(Settings settings)
default Map<String,ExternalSourceFactory>	sourceFactories(Settings settings)
default Map<String,StorageProviderFactory>	storageProviders(Settings settings)
default Map<String,StorageProviderFactory>	storageProviders(Settings settings, ExecutorService executor)
default Set<String>	supportedCatalogs()	Catalog types this plugin provides (e.g.
default Set<String>	supportedConnectorSchemes()	URI schemes handled by this plugin's connectors (e.g.
default Set<String>	supportedSchemes()	URI schemes whose storage providers this plugin supplies (e.g.
default Map<String,TableCatalogFactory>	tableCatalogs(Settings settings)

Method Details

supportedSchemes

default Set<String> supportedSchemes()

URI schemes whose storage providers this plugin supplies (e.g. "s3", "gs", "http"). Called once at registration time; must be cheap (no I/O, no heavy deps). Connector plugins should use supportedConnectorSchemes() instead.

supportedConnectorSchemes

default Set<String> supportedConnectorSchemes()

URI schemes handled by this plugin's connectors (e.g. "flight", "grpc"). Separate from supportedSchemes() which is for storage providers.

formatSpecs

default Set<FormatSpec> formatSpecs()

Format descriptors this plugin provides. Each FormatSpec pairs a logical format name with the file extensions that select it. Format names must match the keys returned by formatReaders(Settings).

supportedCatalogs

default Set<String> supportedCatalogs()

Catalog types this plugin provides (e.g. "iceberg"). Keys must match tableCatalogs(Settings) keys.

storageProviders

default Map<String,StorageProviderFactory> storageProviders(Settings settings)

storageProviders

default Map<String,StorageProviderFactory> storageProviders(Settings settings,
 ExecutorService executor)

formatReaders

default Map<String,FormatReaderFactory> formatReaders(Settings settings)

decompressionCodecs

default List<DecompressionCodec> decompressionCodecs(Settings settings)

Decompression codecs this plugin provides (e.g. gzip for .gz, .gzip). Used for compound extensions like .csv.gz or .ndjson.gz.

sourceFactories

default Map<String,ExternalSourceFactory> sourceFactories(Settings settings)

tableCatalogs

default Map<String,TableCatalogFactory> tableCatalogs(Settings settings)

operatorFactories

default Map<String,SourceOperatorFactoryProvider> operatorFactories(Settings settings)

connectors

default Map<String,ConnectorFactory> connectors(Settings settings)

filterPushdownSupport

default Map<String,FilterPushdownSupport> filterPushdownSupport(Settings settings)

namedWriteables

default List<NamedWriteableRegistry.Entry> namedWriteables()