Type Members

1.  class AppendOnceFlow extends ResolvedFlow

   A Flow that reads source[s] completely and appends data to the target, just once.

2.  class BatchTableWrite extends FlowExecution

   A FlowExecution that writes a batch DataFrame to a Table.

3.  case class CircularDependencyException(downstreamTable: TableIdentifier, upstreamDataset: TableIdentifier) extends AnalysisException with Product with Serializable

   Raised when there's a circular dependency in the current pipeline.

   Raised when there's a circular dependency in the current pipeline. That is, a downstream table is referenced while creating a upstream table.

4.  class CompleteFlow extends ResolvedFlow

   A Flow that declares exactly what data should be in the target table.

5.  class CoreDataflowNodeProcessor extends AnyRef

   Processor that is responsible for analyzing each flow and sort the nodes in topological order

6.  case class DataflowGraph(flows: Seq[Flow], tables: Seq[Table], sinks: Seq[Sink], views: Seq[View]) extends GraphOperations with GraphValidations with Product with Serializable

   DataflowGraph represents the core graph structure for Spark declarative pipelines.

   DataflowGraph represents the core graph structure for Spark declarative pipelines. It manages the relationships between logical flows, tables, sinks, and views, providing operations for graph traversal, validation, and transformation.

7.  class DataflowGraphTransformer extends AutoCloseable

   Resolves the DataflowGraph by processing each node in the graph.

   Resolves the DataflowGraph by processing each node in the graph. This class exposes visitor functionality to resolve/analyze graph nodes. We only expose simple visitor abilities to transform different entities of the graph. For advanced transformations we also expose a mechanism to walk the graph over entity by entity.

   Assumptions: 1. Each output will have at-least 1 flow to it. 2. Each flow may or may not have a destination table. If a flow does not have a destination table, the destination is a view.

   The way graph is structured is that flows, tables and sinks all are graph elements or nodes. While we expose transformation functions for each of these entities, we also expose a way to process to walk over the graph.

   Constructor is private as all usages should be via DataflowGraphTransformer.withDataflowGraphTransformer.

8.  sealed trait Dataset extends Output

   A type of Output that represents a materialized dataset in a DataflowGraph.

9.  sealed trait ExecutionResult extends AnyRef

   A flow's execution may complete for two reasons: 1.

   A flow's execution may complete for two reasons: 1. it may finish performing all of its necessary work, or 2. it may be interrupted by a request from a user to stop it.

   We use this result to disambiguate these two cases, using 'ExecutionResult.FINISHED' for the former and 'ExecutionResult.STOPPED' for the latter.

10.  case class FailureStoppingFlow(flowIdentifiers: Seq[TableIdentifier]) extends FailureStoppingOperation with Product with Serializable

    Indicates that there was a failure while stopping the flow.

11.  abstract class FailureStoppingOperation extends RunFailure

    Abstract class used to identify failures related to failures stopping an operation/timeouts.

12.  trait Flow extends GraphElement with Logging

    A Flow is a node of data transformation in a dataflow graph.

    A Flow is a node of data transformation in a dataflow graph. It describes the movement of data into a particular dataset.

13.  trait FlowExecution extends AnyRef

    A FlowExecution specifies how to execute a flow and manages its execution.

14.  sealed trait FlowFilter extends GraphFilter[ResolvedFlow]

    Specifies how we should filter Flows.

15.  trait FlowFunction extends Logging

    A wrapper for the lambda function that defines a Flow.

16.  case class FlowFunctionResult(requestedInputs: Set[TableIdentifier], batchInputs: Set[ResolvedInput], streamingInputs: Set[ResolvedInput], usedExternalInputs: Set[TableIdentifier], dataFrame: Try[classic.DataFrame], sqlConf: Map[String, String], analysisWarnings: Seq[AnalysisWarning] = Nil) extends Product with Serializable

    Holds the DataFrame returned by a FlowFunction along with the inputs used to construct it.

    Holds the DataFrame returned by a FlowFunction along with the inputs used to construct it.

    batchInputs

    the complete inputs read by the flow

    streamingInputs

    the incremental inputs read by the flow

    usedExternalInputs

    the identifiers of the external inputs read by the flow

    dataFrame

    the DataFrame expression executed by the flow if the flow can be resolved

17.  case class FlowNode(identifier: TableIdentifier, inputs: Set[TableIdentifier], output: TableIdentifier) extends Product with Serializable
    identifier

    The identifier of the flow.

    inputs

    The identifiers of nodes used as inputs to this flow.

    output

    The identifier of the output that this flow writes to.

18.  class FlowPlanner extends AnyRef

    Plans execution of Flows in a DataflowGraph by converting Flows into 'FlowExecution's.

19.  case class FlowSystemMetadata(context: PipelineUpdateContext, flow: Flow, graph: DataflowGraph) extends SystemMetadata with Logging with Product with Serializable

    Represents the system metadata associated with a Flow.

20.  case class FlowsForTables(selectedTables: Set[TableIdentifier]) extends FlowFilter with Product with Serializable

    Used in partial graph updates to select flows that flow to "selectedTables".

21.  trait GraphElement extends AnyRef

    An element in a DataflowGraph.

22.  abstract class GraphExecution extends Logging
23.  sealed trait GraphFilter[E] extends AnyRef

    Specifies how we should filter Graph elements.

24.  trait GraphOperations extends AnyRef
25.  class GraphRegistrationContext extends AnyRef

    A mutable context for registering tables, views, and flows in a dataflow graph.

26.  trait GraphValidations extends Logging

    Validations performed on a DataflowGraph.

27.  trait Input extends GraphElement

    Specifies an input that can be referenced by another Dataset's query.

28.  case class LoadTableException(name: String, cause: Option[Throwable]) extends SparkException with Product with Serializable

    Exception raised when a flow fails to read from a table defined within the pipeline

    Exception raised when a flow fails to read from a table defined within the pipeline

    name

    The name of the table

    cause

    The cause of the failure

29.  sealed trait Output extends AnyRef

    Represents a node in a DataflowGraph that can be written to by a Flow.

    Represents a node in a DataflowGraph that can be written to by a Flow. Must be backed by a file source.

30.  case class PersistedView(identifier: TableIdentifier, properties: Map[String, String], sqlText: Option[String], comment: Option[String], origin: QueryOrigin) extends View with Product with Serializable

    Representing a persisted View in a DataflowGraph.

31.  class PipelineExecution extends AnyRef

    Executes a DataflowGraph by resolving the graph, materializing datasets, and running the flows.

32.  case class PipelineTableProperty[T](key: String, default: String, fromString: (String) => T) extends Product with Serializable
33.  trait PipelineUpdateContext extends AnyRef
34.  class PipelineUpdateContextImpl extends PipelineUpdateContext

    An implementation of the PipelineUpdateContext trait used in production.

35.  case class QueryContext(currentCatalog: Option[String], currentDatabase: Option[String]) extends Product with Serializable

    Contains the catalog and database context information for query execution.

36.  case class QueryExecutionFailure(flowName: String, maxRetries: Int, cause: Option[Throwable]) extends RunFailure with Product with Serializable

    Indicates that run has failed due to a query execution failure.

37.  case class QueryOrigin(language: Option[Language] = None, filePath: Option[String] = None, sqlText: Option[String] = None, line: Option[Int] = None, startPosition: Option[Int] = None, objectType: Option[String] = None, objectName: Option[String] = None) extends Product with Serializable

    Records information used to track the provenance of a given query to user code.

    Records information used to track the provenance of a given query to user code.

    language

    The language used by the user to define the query.

    filePath

    Path to the file of the user code that defines the query.

    sqlText

    The SQL text of the query.

    line

    The line number of the query in the user code. Line numbers are 1-indexed.

    startPosition

    The start position of the query in the user code.

    objectType

    The type of the object that the query is associated with. (Table, View, etc)

    objectName

    The name of the object that the query is associated with.

38.  trait ResolutionCompletedFlow extends Flow

    A Flow whose flow function has been invoked, meaning either:

    A Flow whose flow function has been invoked, meaning either:

    • Its output schema and dependencies are known.
    • It failed to resolve.
39.  class ResolutionFailedFlow extends ResolutionCompletedFlow

    A Flow whose flow function has failed to resolve.

40.  trait ResolvedFlow extends ResolutionCompletedFlow with Input

    A Flow whose flow function has successfully resolved.

41.  case class ResolvedInput(input: Input, aliasIdentifier: AliasIdentifier) extends Product with Serializable

    A wrapper for a resolved internal input that includes the alias provided by the user.

42.  case class RunCompletion() extends RunTerminationReason with Product with Serializable

    Indicates that a triggered run has successfully completed execution.

43.  sealed abstract class RunFailure extends RunTerminationReason

    Indicates that an run entered the failed state..

44.  case class RunTerminationException(reason: RunTerminationReason) extends Exception with Product with Serializable

    Helper exception class that indicates that a run has to be terminated and tracks the associated termination reason.

45.  sealed trait RunTerminationReason extends AnyRef
46.  trait Sink extends GraphElement with Output
47.  case class SinkImpl(identifier: TableIdentifier, format: String, options: Map[String, String], origin: QueryOrigin) extends Sink with Product with Serializable
48.  class SinkWrite extends StreamingFlowExecution

    A StreamingFlowExecution that writes a streaming DataFrame to a Sink.

49.  case class SomeTables(selectedTables: Set[TableIdentifier]) extends TableFilter with Product with Serializable

    Used in partial graph updates to select "selectedTables".

50.  case class SqlGraphElementRegistrationException(msg: String, queryOrigin: QueryOrigin) extends AnalysisException with Product with Serializable
51.  class SqlGraphRegistrationContext extends AnyRef

    SQL statement processor context.

    SQL statement processor context. At any instant, an instance of this class holds the "active" catalog/schema in use within this SQL statement processing context, and tables/views/flows that have been registered from SQL statements within this context.

52.  class SqlGraphRegistrationContextState extends AnyRef

    Data class for all state that is accumulated while processing a particular SqlGraphRegistrationContext.

53.  class StreamingFlow extends ResolvedFlow

    A Flow that represents stateful movement of data to some target.

54.  trait StreamingFlowExecution extends FlowExecution with Logging

    A 'FlowExecution' that processes data statefully using Structured Streaming.

55.  class StreamingTableWrite extends StreamingFlowExecution

    A StreamingFlowExecution that writes a streaming DataFrame to a Table.

56.  sealed trait SystemMetadata extends AnyRef
57.  case class Table(identifier: TableIdentifier, specifiedSchema: Option[StructType], partitionCols: Option[Seq[String]], normalizedPath: Option[String], properties: Map[String, String] = Map.empty, comment: Option[String], origin: QueryOrigin, isStreamingTable: Boolean, format: Option[String]) extends TableInput with Dataset with Product with Serializable

    A table representing a materialized dataset in a DataflowGraph.

    A table representing a materialized dataset in a DataflowGraph.

    identifier

    The identifier of this table within the graph.

    specifiedSchema

    The user-specified schema for this table.

    partitionCols

    What columns the table should be partitioned by when materialized.

    normalizedPath

    Normalized storage location for the table based on the user-specified table path (if not defined, we will normalize a managed storage path for it).

    properties

    Table Properties to set in table metadata.

    comment

    User-specified comment that can be placed on the table.

    isStreamingTable

    if the table is a streaming table, as defined by the source code.

58.  sealed trait TableFilter extends GraphFilter[Table]

    Specifies how we should filter Tables.

59.  sealed trait TableInput extends Input

    A type of Input where data is loaded from a table.

60.  case class TemporaryView(identifier: TableIdentifier, properties: Map[String, String], sqlText: Option[String], comment: Option[String], origin: QueryOrigin) extends View with Product with Serializable

    Representing a temporary View in a DataflowGraph.

61.  case class TriggeredFailureInfo(lastFailTimestamp: Long, numFailures: Int, lastException: Throwable, lastExceptionAction: FlowExecutionAction) extends Product with Serializable
62.  class TriggeredGraphExecution extends GraphExecution

    Executes all of the flows in the given graph in topological order.

    Executes all of the flows in the given graph in topological order. Each flow processes all available data before downstream flows are triggered.

63.  class UncaughtExceptionHandler extends java.lang.Thread.UncaughtExceptionHandler

    Uncaught exception handler which first calls the delegate and then calls the OnFailure function with the uncaught exception.

64.  case class UnexpectedRunFailure() extends RunFailure with Product with Serializable

    Run could not be associated with a proper root cause.

    Run could not be associated with a proper root cause. This is not expected and likely indicates a bug.

65.  case class UnionFlowFilter(oneFilter: FlowFilter, otherFilter: FlowFilter) extends FlowFilter with Product with Serializable

    Returns a flow filter that is a union of two flow filters

66.  case class UnresolvedDatasetException(identifier: TableIdentifier) extends AnalysisException with Product with Serializable

    Exception raised when a flow tries to read from a dataset that exists but is unresolved

    Exception raised when a flow tries to read from a dataset that exists but is unresolved

    identifier

    The identifier of the dataset

67.  case class UnresolvedFlow(identifier: TableIdentifier, destinationIdentifier: TableIdentifier, func: FlowFunction, queryContext: QueryContext, sqlConf: Map[String, String], once: Boolean, origin: QueryOrigin) extends Flow with Product with Serializable

    A Flow whose output schema and dependencies aren't known.

68.  case class UnresolvedPipelineException(graph: DataflowGraph, directFailures: Map[TableIdentifier, Throwable], downstreamFailures: Map[TableIdentifier, Throwable], additionalHint: Option[String] = None) extends AnalysisException with Product with Serializable

    Exception raised when a pipeline has one or more flows that cannot be resolved

    Exception raised when a pipeline has one or more flows that cannot be resolved

    directFailures

    Mapping between the name of flows that failed to resolve (due to an error in that flow) and the error that occurred when attempting to resolve them

    downstreamFailures

    Mapping between the name of flows that failed to resolve (because they failed to read from other unresolved flows) and the error that occurred when attempting to resolve them

69.  trait View extends GraphElement

    Representing a view in the DataflowGraph.

70.  case class VirtualTableInput(identifier: TableIdentifier, specifiedSchema: Option[StructType], incomingFlowIdentifiers: Set[TableIdentifier], availableFlows: Seq[ResolvedFlow] = Nil) extends TableInput with Logging with Product with Serializable

    A type of TableInput that returns data from a specified schema or from the inferred Flows that write to the table.