Interface Flow
- All Superinterfaces:
GraphElement
,org.apache.spark.internal.Logging
- All Known Subinterfaces:
ResolutionCompletedFlow
,ResolvedFlow
- All Known Implementing Classes:
AppendOnceFlow
,CompleteFlow
,ResolutionFailedFlow
,StreamingFlow
,UnresolvedFlow
A
Flow
is a node of data transformation in a dataflow graph. It describes the movement
of data into a particular dataset.-
Nested Class Summary
Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging
org.apache.spark.internal.Logging.LogStringContext, org.apache.spark.internal.Logging.SparkShellLoggingFilter
-
Method Summary
Modifier and TypeMethodDescriptionscala.Option<String>
comment()
The comment associated with this floworg.apache.spark.sql.catalyst.TableIdentifier
The dataset that this Flow represents a write to.func()
TheFlowFunction
containing the user's query.org.apache.spark.sql.catalyst.TableIdentifier
Returns the unique identifier for thisGraphElement
.boolean
once()
Whether this is a ONCE flow.The current query context (catalog and database) when the query is defined.sqlConf()
Methods inherited from interface org.apache.spark.sql.pipelines.graph.GraphElement
displayName, origin, spark
Methods inherited from interface org.apache.spark.internal.Logging
initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logBasedOnLevel, logDebug, logDebug, logDebug, logDebug, logError, logError, logError, logError, logInfo, logInfo, logInfo, logInfo, logName, LogStringContext, logTrace, logTrace, logTrace, logTrace, logWarning, logWarning, logWarning, logWarning, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, withLogContext
-
Method Details
-
comment
scala.Option<String> comment()The comment associated with this flow -
destinationIdentifier
org.apache.spark.sql.catalyst.TableIdentifier destinationIdentifier()The dataset that this Flow represents a write to.- Returns:
- (undocumented)
-
func
FlowFunction func()TheFlowFunction
containing the user's query. -
identifier
org.apache.spark.sql.catalyst.TableIdentifier identifier()Description copied from interface:GraphElement
Returns the unique identifier for thisGraphElement
.- Specified by:
identifier
in interfaceGraphElement
-
once
boolean once()Whether this is a ONCE flow. ONCE flows should run only once per full refresh.- Returns:
- (undocumented)
-
queryContext
QueryContext queryContext()The current query context (catalog and database) when the query is defined. -
sqlConf
-