org.apache.spark.sql.pipelines.graph
DataflowGraphTransformer
Companion object DataflowGraphTransformer
class DataflowGraphTransformer extends AutoCloseable
Resolves the DataflowGraph by processing each node in the graph. This class exposes visitor functionality to resolve/analyze graph nodes. We only expose simple visitor abilities to transform different entities of the graph. For advanced transformations we also expose a mechanism to walk the graph over entity by entity.
Assumptions: 1. Each output will have at-least 1 flow to it. 2. Each flow may or may not have a destination table. If a flow does not have a destination table, the destination is a view.
The way graph is structured is that flows, tables and sinks all are graph elements or nodes. While we expose transformation functions for each of these entities, we also expose a way to process to walk over the graph.
Constructor is private as all usages should be via DataflowGraphTransformer.withDataflowGraphTransformer.
- Alphabetic
- By Inheritance
- DataflowGraphTransformer
- AutoCloseable
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new DataflowGraphTransformer(graph: DataflowGraph)
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- def close(): Unit
- Definition Classes
- DataflowGraphTransformer → AutoCloseable
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def getDataflowGraph: DataflowGraph
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- def transformDownNodes(transformer: (GraphElement, Seq[GraphElement]) => Seq[GraphElement], disableParallelism: Boolean = false): DataflowGraphTransformer
Example graph: [Flow1, Flow 2] -> ST -> Flow3 -> MV Order of processing: Flow1, Flow2, ST, Flow3, MV.
Example graph: [Flow1, Flow 2] -> ST -> Flow3 -> MV Order of processing: Flow1, Flow2, ST, Flow3, MV.
- transformer
function that transforms any graph entity. transformer( nodeToTransform: GraphElement, upstreamNodes: Seq[GraphElement] ) => transformedNodes: Seq[GraphElement]
- returns
this
- def transformTables(transformer: (Table) => Table): DataflowGraphTransformer
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)