Package org.elasticsearch.xpack.esql.datasource.csv

Class CsvFormatReader

java.lang.Object

org.elasticsearch.xpack.esql.datasource.csv.CsvFormatReader

All Implemented Interfaces:

Closeable, AutoCloseable, FormatReader, SegmentableFormatReader

public class CsvFormatReader
extends Object
implements SegmentableFormatReader

CSV/TSV format reader for external datasources.

File format

• First non-comment line: schema — column:type pairs separated by the delimiter
• Subsequent lines: data rows
• Empty/missing values → null
• Lines starting with the comment prefix (default //) are skipped

Supported types

integer (int, i), long (l), double (d), keyword (k, string, s), text (txt), boolean (bool), datetime (date, dt), ip, null (n).

Configurable options

All options are set via the WITH clause and parsed by withConfig(java.util.Map).

CSV options

ES/ESQL key	Default	Description
delimiter	,	Field separator character
quote	"	Quoting character
escape	\	Escape character inside quoted fields
comment	//	Line comment prefix
null_value	(empty)	String representation of null
encoding	UTF-8	Character encoding
datetime_format	ISO-8601 / epoch	Custom datetime pattern
max_field_size	10 MB	OOM protection; max bytes per field
multi_value_syntax	brackets	Multi-value field syntax
schema_sample_size	20,000	Number of rows to sample for type inference

Bracket multi-value syntax

When multi_value_syntax is brackets, array-like values support:

• [a,b,c] — unquoted elements
• ["a","b","c"] — quoted elements (quotes stripped)
• [a,"b,c"] — mixed; commas inside quotes are literal

With comma delimiter, a cell like [hello,world] is treated as one column: commas inside [...] are not column delimiters.

Error handling

Controlled by ErrorPolicy and its ErrorPolicy.Mode:

Error modes

ES/ESQL key	Behaviour
fail_fast	Abort on first error (default)
skip_row	Drop the entire bad row
null_field	Null-fill unparseable fields, keep the row

Examples

  EXTERNAL "s3://bucket/data.tsv" WITH {"delimiter": "\t", "error_mode": "skip_row", "max_errors": 100}

  EXTERNAL "s3://bucket/employees.csv" WITH {"multi_value_syntax": "brackets"}

  EXTERNAL "s3://bucket/data.csv" WITH {"multi_value_syntax": "brackets", "error_mode": "skip_row"}

Works with any StorageProvider (HTTP, S3, local filesystem).

Nested Class Summary

Nested classes/interfaces inherited from interface org.elasticsearch.xpack.esql.datasources.spi.FormatReader

FormatReader.SchemaResolution

Field Summary

Fields inherited from interface org.elasticsearch.xpack.esql.datasources.spi.FormatReader

NO_LIMIT

Constructor Summary

Constructors

Constructor	Description
CsvFormatReader(BlockFactory blockFactory)
CsvFormatReader(BlockFactory blockFactory, String format, List<String> extensions)
CsvFormatReader(BlockFactory blockFactory, CsvFormatOptions options, String format, List<String> extensions)

Method Summary

Modifier and Type	Method	Description
void	close()
List<String>	fileExtensions()
long	findNextRecordBoundary(InputStream stream)
String	formatName()
SourceMetadata	metadata(StorageObject object)
CloseableIterator<Page>	read(StorageObject object, FormatReadContext context)
FormatReader	withConfig(Map<String,Object> config)
CsvFormatReader	withOptions(CsvFormatOptions newOptions)
CsvFormatReader	withSchema(List<Attribute> schema)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.elasticsearch.xpack.esql.datasources.spi.FormatReader

aggregatePushdownSupport, defaultErrorPolicy, defaultSchemaResolution, filterPushdownSupport, read, readAsync, schema, supportsNativeAsync, withPushedFilter

Methods inherited from interface org.elasticsearch.xpack.esql.datasources.spi.SegmentableFormatReader

minimumSegmentSize

Constructor Details

CsvFormatReader

public CsvFormatReader(BlockFactory blockFactory)

CsvFormatReader

public CsvFormatReader(BlockFactory blockFactory,
 String format,
 List<String> extensions)

CsvFormatReader

public CsvFormatReader(BlockFactory blockFactory,
 CsvFormatOptions options,
 String format,
 List<String> extensions)

Method Details

withOptions

public CsvFormatReader withOptions(CsvFormatOptions newOptions)

withSchema

public CsvFormatReader withSchema(List<Attribute> schema)

Specified by:

withSchema in interface FormatReader

withConfig

public FormatReader withConfig(Map<String,Object> config)

Specified by:

withConfig in interface FormatReader

metadata

public SourceMetadata metadata(StorageObject object)
                        throws IOException

Specified by:

metadata in interface FormatReader

Throws:

IOException

read

public CloseableIterator<Page> read(StorageObject object,
 FormatReadContext context)
                             throws IOException

Specified by:

read in interface FormatReader

Throws:

IOException

findNextRecordBoundary

public long findNextRecordBoundary(InputStream stream)
                            throws IOException

Specified by:

findNextRecordBoundary in interface SegmentableFormatReader

Throws:

IOException

formatName

public String formatName()

Specified by:

formatName in interface FormatReader

fileExtensions

public List<String> fileExtensions()

Specified by:

fileExtensions in interface FormatReader

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Throws:

IOException