Package org.elasticsearch.xpack.esql.plan.physical

Class ExternalSourceExec

java.lang.Object

org.elasticsearch.xpack.esql.core.tree.Node<PhysicalPlan>

org.elasticsearch.xpack.esql.plan.QueryPlan<PhysicalPlan>

org.elasticsearch.xpack.esql.plan.physical.PhysicalPlan

org.elasticsearch.xpack.esql.plan.physical.LeafExec

org.elasticsearch.xpack.esql.plan.physical.ExternalSourceExec

All Implemented Interfaces:

NamedWriteable, Writeable, DataSourceExec, EstimatesRowSize

public class ExternalSourceExec
extends LeafExec
implements EstimatesRowSize, DataSourceExec

Generic physical plan node for reading from external data sources (e.g., Iceberg tables, Parquet files).

This is the unified physical plan node for all external sources, replacing source-specific nodes. It uses generic maps for configuration and metadata to avoid leaking source-specific types (like S3Configuration) into core ESQL code.

Key design principles:

• Generic configuration: Uses Map<String, Object> for config instead of source-specific classes like S3Configuration
• Opaque metadata: Source-specific data (native schema, etc.) is stored in sourceMetadata() and passed through without core understanding it
• Opaque pushed filter: The pushedFilter() is an opaque Object that only the source-specific operator factory interprets. It is NOT serialized; it is created locally on each data node by the LocalPhysicalPlanOptimizer via FilterPushdownRegistry
• Data node execution: Created on data nodes by LocalMapper from ExternalRelation inside FragmentExec

Nested Class Summary

Nested classes/interfaces inherited from class org.elasticsearch.xpack.esql.core.tree.Node

Node.NodeStringFormat

Nested classes/interfaces inherited from interface org.elasticsearch.xpack.esql.plan.physical.EstimatesRowSize

EstimatesRowSize.State

Nested classes/interfaces inherited from interface org.elasticsearch.common.io.stream.Writeable

Writeable.Reader<V>, Writeable.Writer<V>

Field Summary

Fields

Modifier and Type	Field	Description
static final NamedWriteableRegistry.Entry	ENTRY

Fields inherited from class org.elasticsearch.xpack.esql.core.tree.Node

TO_STRING_MAX_LINES, TO_STRING_MAX_WIDTH

Constructor Summary

Constructors

Constructor	Description
ExternalSourceExec(Source source, String sourcePath, String sourceType, List<Attribute> attributes, Map<String,Object> config, Map<String,Object> sourceMetadata, Integer estimatedRowSize)
ExternalSourceExec(Source source, String sourcePath, String sourceType, List<Attribute> attributes, Map<String,Object> config, Map<String,Object> sourceMetadata, Object pushedFilter, int pushedLimit, Integer estimatedRowSize, FileList fileList, List<ExternalSplit> splits)
ExternalSourceExec(Source source, String sourcePath, String sourceType, List<Attribute> attributes, Map<String,Object> config, Map<String,Object> sourceMetadata, Object pushedFilter, Integer estimatedRowSize)
ExternalSourceExec(Source source, String sourcePath, String sourceType, List<Attribute> attributes, Map<String,Object> config, Map<String,Object> sourceMetadata, Object pushedFilter, Integer estimatedRowSize, FileList fileList)
ExternalSourceExec(Source source, String sourcePath, String sourceType, List<Attribute> attributes, Map<String,Object> config, Map<String,Object> sourceMetadata, Object pushedFilter, List<Expression> pushedExpressions, int pushedLimit, Integer estimatedRowSize, FileList fileList, List<ExternalSplit> splits)

Method Summary

Modifier and Type	Method	Description
Map<String,Object>	config()
boolean	equals(Object obj)
Integer	estimatedRowSize()
PhysicalPlan	estimateRowSize(EstimatesRowSize.State state)	Estimate the number of bytes that'll be loaded per position before the stream of pages is consumed.
FileList	fileList()
String	getWriteableName()
int	hashCode()
protected NodeInfo<? extends PhysicalPlan>	info()	Normally, you want to use one of the static create methods to implement this.
void	nodeString(StringBuilder sb, Node.NodeStringFormat format)	Append this Node's string representation to sb.
List<Attribute>	output()	The ordered list of attributes (i.e.
List<Expression>	pushedExpressions()
Object	pushedFilter()
int	pushedLimit()
Map<String,Object>	sourceMetadata()
String	sourcePath()
String	sourceType()
List<ExternalSplit>	splits()
protected ExternalSourceExec	withEstimatedRowSize(Integer newEstimatedRowSize)
ExternalSourceExec	withPushedFilter(Object newFilter)
ExternalSourceExec	withPushedFilterAndExpressions(Object newFilter, List<Expression> newPushedExpressions)
ExternalSourceExec	withPushedLimit(int newLimit)
ExternalSourceExec	withSplits(List<ExternalSplit> newSplits)
void	writeTo(StreamOutput out)

Methods inherited from class org.elasticsearch.xpack.esql.plan.physical.LeafExec

computeReferences, replaceChildren

Methods inherited from class org.elasticsearch.xpack.esql.plan.QueryPlan

computeExpressions, expressions, forEachExpression, forEachExpression, forEachExpressionDown, forEachExpressionUp, inputSet, outputSet, references, transformExpressionsDown, transformExpressionsDown, transformExpressionsOnly, transformExpressionsOnly, transformExpressionsOnlyUp, transformExpressionsUp, transformExpressionsUp

Methods inherited from class org.elasticsearch.xpack.esql.core.tree.Node

allMatch, anyMatch, children, collect, collect, collect, collectFirstChildren, collectLeaves, doCollectFirst, forEachDown, forEachDown, forEachDownMayReturnEarly, forEachProperty, forEachPropertyDown, forEachPropertyOnly, forEachPropertyUp, forEachUp, forEachUp, nodeName, nodeProperties, nodeString, propertiesToString, replaceChildrenSameSize, source, sourceLocation, sourceText, toString, toString, transformChildren, transformChildren, transformDown, transformDown, transformDown, transformDown, transformDownSkipBranch, transformNodeProps, transformPropertiesDown, transformPropertiesOnly, transformPropertiesUp, transformUp, transformUp, transformUp

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.elasticsearch.xpack.esql.plan.physical.DataSourceExec

source

Field Details

ENTRY

public static final NamedWriteableRegistry.Entry ENTRY

Constructor Details

ExternalSourceExec

public ExternalSourceExec(Source source,
 String sourcePath,
 String sourceType,
 List<Attribute> attributes,
 Map<String,Object> config,
 Map<String,Object> sourceMetadata,
 Object pushedFilter,
 Integer estimatedRowSize,
 FileList fileList)

ExternalSourceExec

public ExternalSourceExec(Source source,
 String sourcePath,
 String sourceType,
 List<Attribute> attributes,
 Map<String,Object> config,
 Map<String,Object> sourceMetadata,
 Object pushedFilter,
 int pushedLimit,
 Integer estimatedRowSize,
 FileList fileList,
 List<ExternalSplit> splits)

ExternalSourceExec

public ExternalSourceExec(Source source,
 String sourcePath,
 String sourceType,
 List<Attribute> attributes,
 Map<String,Object> config,
 Map<String,Object> sourceMetadata,
 Object pushedFilter,
 List<Expression> pushedExpressions,
 int pushedLimit,
 Integer estimatedRowSize,
 FileList fileList,
 List<ExternalSplit> splits)

ExternalSourceExec

public ExternalSourceExec(Source source,
 String sourcePath,
 String sourceType,
 List<Attribute> attributes,
 Map<String,Object> config,
 Map<String,Object> sourceMetadata,
 Object pushedFilter,
 Integer estimatedRowSize)

ExternalSourceExec

public ExternalSourceExec(Source source,
 String sourcePath,
 String sourceType,
 List<Attribute> attributes,
 Map<String,Object> config,
 Map<String,Object> sourceMetadata,
 Integer estimatedRowSize)

Method Details

writeTo

public void writeTo(StreamOutput out)
             throws IOException

Specified by:

writeTo in interface Writeable

Throws:

IOException

getWriteableName

public String getWriteableName()

Specified by:

getWriteableName in interface NamedWriteable

sourcePath

public String sourcePath()

sourceType

public String sourceType()

output

public List<Attribute> output()

Description copied from class: QueryPlan

The ordered list of attributes (i.e. columns) this plan produces when executed. Must be called only on resolved plans, otherwise may throw an exception or return wrong results.

Specified by:

output in interface DataSourceExec

Specified by:

output in class QueryPlan<PhysicalPlan>

config

public Map<String,Object> config()

sourceMetadata

public Map<String,Object> sourceMetadata()

pushedFilter

public Object pushedFilter()

pushedExpressions

public List<Expression> pushedExpressions()

pushedLimit

public int pushedLimit()

estimatedRowSize

public Integer estimatedRowSize()

fileList

public FileList fileList()

splits

public List<ExternalSplit> splits()

withSplits

public ExternalSourceExec withSplits(List<ExternalSplit> newSplits)

withPushedFilter

public ExternalSourceExec withPushedFilter(Object newFilter)

withPushedFilterAndExpressions

public ExternalSourceExec withPushedFilterAndExpressions(Object newFilter,
 List<Expression> newPushedExpressions)

withPushedLimit

public ExternalSourceExec withPushedLimit(int newLimit)

estimateRowSize

public PhysicalPlan estimateRowSize(EstimatesRowSize.State state)

Description copied from interface: EstimatesRowSize

Estimate the number of bytes that'll be loaded per position before the stream of pages is consumed.

Specified by:

estimateRowSize in interface EstimatesRowSize

withEstimatedRowSize

protected ExternalSourceExec withEstimatedRowSize(Integer newEstimatedRowSize)

info

protected NodeInfo<? extends PhysicalPlan> info()

Description copied from class: Node

Normally, you want to use one of the static create methods to implement this.

For QueryPlans, it is very important that the properties contain all of the expressions and references relevant to this node, and that all the properties are used in the provided constructor; otherwise query plan transformations like QueryPlan#transformExpressionsOnly(Function) will not have an effect.

Specified by:

info in class Node<PhysicalPlan>

hashCode

public int hashCode()

Specified by:

hashCode in class PhysicalPlan

equals

public boolean equals(Object obj)

Specified by:

equals in class PhysicalPlan

nodeString

public void nodeString(StringBuilder sb,
 Node.NodeStringFormat format)

Description copied from class: Node

Append this Node's string representation to sb. This does not include this node's Node.children().

Overrides:

nodeString in class Node<PhysicalPlan>

Parameters:

sb - target for the string

format - configuration for rendering the string representation