org.apache.spark.sql.pipelines.graph
DataflowGraphTransformer
Companion object DataflowGraphTransformer
class DataflowGraphTransformer extends AutoCloseable
Resolves the DataflowGraph by processing each node in the graph. This class exposes visitor functionality to resolve/analyze graph nodes. We only expose simple visitor abilities to transform different entities of the graph. For advanced transformations we also expose a mechanism to walk the graph over entity by entity.
Assumptions: 1. Each output will have at-least 1 flow to it. 2. Each flow may or may not have a destination table. If a flow does not have a destination table, the destination is a temporary view.
The way graph is structured is that flows, tables and sinks all are graph elements or nodes. While we expose transformation functions for each of these entities, we also expose a way to process to walk over the graph.
Constructor is private as all usages should be via DataflowGraphTransformer.withDataflowGraphTransformer.
- Alphabetic
- By Inheritance
- DataflowGraphTransformer
- AutoCloseable
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new DataflowGraphTransformer(graph: DataflowGraph)
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- def close(): Unit
- Definition Classes
- DataflowGraphTransformer → AutoCloseable
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def getDataflowGraph: DataflowGraph
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- def transformDownNodes(transformer: (GraphElement, Seq[GraphElement]) => Seq[GraphElement], disableParallelism: Boolean = false): DataflowGraphTransformer
Example graph: [Flow1, Flow 2] -> ST -> Flow3 -> MV Order of processing: Flow1, Flow2, ST, Flow3, MV.
Example graph: [Flow1, Flow 2] -> ST -> Flow3 -> MV Order of processing: Flow1, Flow2, ST, Flow3, MV.
- transformer
function that transforms any graph entity. transformer( nodeToTransform: GraphElement, upstreamNodes: Seq[GraphElement] ) => transformedNodes: Seq[GraphElement]
- returns
this
- def transformTables(transformer: (Table) => Table): DataflowGraphTransformer
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)