org.apache.spark.sql.SQLContext

SparkPlanner

class SparkPlanner extends SparkStrategies

Attributes
protected[org.apache.spark.sql]
Linear Supertypes
SparkStrategies, QueryPlanner[SparkPlan], AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. SparkPlanner
  2. SparkStrategies
  3. QueryPlanner
  4. AnyRef
  5. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new SparkPlanner()

Type Members

  1. case class CommandStrategy extends Strategy with Product with Serializable

    Definition Classes
    SparkStrategies
  2. abstract class Strategy extends Logging

    Attributes
    protected
    Definition Classes
    QueryPlanner

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. object BasicOperators extends Strategy

    Definition Classes
    SparkStrategies
  7. object BroadcastNestedLoopJoin extends Strategy

    Definition Classes
    SparkStrategies
  8. object CartesianProduct extends Strategy

    Definition Classes
    SparkStrategies
  9. object HashAggregation extends Strategy

    Definition Classes
    SparkStrategies
  10. object HashJoin extends Strategy with PredicateHelper

    Uses the ExtractEquiJoinKeys pattern to find joins where at least some of the predicates can be evaluated by matching hash keys.

    Uses the ExtractEquiJoinKeys pattern to find joins where at least some of the predicates can be evaluated by matching hash keys.

    This strategy applies a simple optimization based on the estimates of the physical sizes of the two join sides. When planning a execution.BroadcastHashJoin, if one side has an estimated physical size smaller than the user-settable threshold org.apache.spark.sql.SQLConf.AUTO_BROADCASTJOIN_THRESHOLD, the planner would mark it as the build relation and mark the other relation as the stream side. The build table will be broadcasted to all of the executors involved in the join, as a org.apache.spark.broadcast.Broadcast object. If both estimates exceed the threshold, they will instead be used to decide the build side in a execution.ShuffledHashJoin.

    Definition Classes
    SparkStrategies
  11. object InMemoryScans extends Strategy

    Definition Classes
    SparkStrategies
  12. object LeftSemiJoin extends Strategy with PredicateHelper

    Definition Classes
    SparkStrategies
  13. object ParquetOperations extends Strategy

    Definition Classes
    SparkStrategies
  14. object TakeOrdered extends Strategy

    Definition Classes
    SparkStrategies
  15. def apply(plan: LogicalPlan): Iterator[SparkPlan]

    Definition Classes
    QueryPlanner
  16. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  17. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  18. def codegenEnabled: Boolean

  19. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  20. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  21. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  22. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  23. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  24. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  25. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  26. final def notify(): Unit

    Definition Classes
    AnyRef
  27. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  28. def numPartitions: Int

  29. def planLater(plan: LogicalPlan): SparkPlan

    Attributes
    protected
    Definition Classes
    QueryPlanner
  30. def pruneFilterProject(projectList: Seq[NamedExpression], filterPredicates: Seq[Expression], prunePushedDownFilters: (Seq[Expression]) ⇒ Seq[Expression], scanBuilder: (Seq[Attribute]) ⇒ SparkPlan): SparkPlan

    Used to build table scan operators where complex projection and filtering are done using separate physical operators.

    Used to build table scan operators where complex projection and filtering are done using separate physical operators. This function returns the given scan operator with Project and Filter nodes added only when needed. For example, a Project operator is only used when the final desired output requires complex expressions to be evaluated or when columns can be further eliminated out after filtering has been done.

    The prunePushedDownFilters parameter is used to remove those filters that can be optimized away by the filter pushdown optimization.

    The required attributes for both filtering and expression evaluation are passed to the provided scanBuilder function so that it can avoid unnecessary column materialization.

  31. lazy val singleRowRdd: RDD[catalyst.expressions.Row]

    Attributes
    protected
    Definition Classes
    SparkStrategies
  32. val sparkContext: SparkContext

  33. val sqlContext: SQLContext

  34. val strategies: Seq[Strategy]

    Definition Classes
    SparkPlanner → QueryPlanner
  35. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  36. def toString(): String

    Definition Classes
    AnyRef → Any
  37. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  38. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  39. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from SparkStrategies

Inherited from QueryPlanner[SparkPlan]

Inherited from AnyRef

Inherited from Any

Ungrouped