Packages

  • package root
    Definition Classes
    root
  • package org
    Definition Classes
    root
  • package apache
    Definition Classes
    org
  • package spark

    Core Spark functionality.

    Core Spark functionality. org.apache.spark.SparkContext serves as the main entry point to Spark, while org.apache.spark.rdd.RDD is the data type representing a distributed collection, and provides most parallel operations.

    In addition, org.apache.spark.rdd.PairRDDFunctions contains operations available only on RDDs of key-value pairs, such as groupByKey and join; org.apache.spark.rdd.DoubleRDDFunctions contains operations available only on RDDs of Doubles; and org.apache.spark.rdd.SequenceFileRDDFunctions contains operations available on RDDs that can be saved as SequenceFiles. These operations are automatically available on any RDD of the right type (e.g. RDD[(Int, Int)] through implicit conversions.

    Java programmers should reference the org.apache.spark.api.java package for Spark programming APIs in Java.

    Classes and methods marked with Experimental are user-facing features which have not been officially adopted by the Spark project. These are subject to change or removal in minor releases.

    Classes and methods marked with Developer API are intended for advanced users want to extend Spark through lower level interfaces. These are subject to changes or removal in minor releases.

    Definition Classes
    apache
  • package sql

    Allows the execution of relational queries, including those expressed in SQL using Spark.

    Allows the execution of relational queries, including those expressed in SQL using Spark.

    Definition Classes
    spark
  • package api

    Contains API classes that are specific to a single language (i.e.

    Contains API classes that are specific to a single language (i.e. Java).

    Definition Classes
    sql
  • package catalog
    Definition Classes
    sql
  • package connector
    Definition Classes
    sql
  • package dynamicpruning
    Definition Classes
    sql
  • CleanupDynamicPruningFilters
  • PartitionPruning
  • PlanDynamicPruningFilters
  • package expressions
    Definition Classes
    sql
  • package hive

    Support for running Spark SQL queries using functionality from Apache Hive (does not require an existing Hive installation).

    Support for running Spark SQL queries using functionality from Apache Hive (does not require an existing Hive installation). Supported Hive features include:

    • Using HiveQL to express queries.
    • Reading metadata from the Hive Metastore using HiveSerDes.
    • Hive UDFs, UDAs, UDTs
    Definition Classes
    sql
  • package jdbc
    Definition Classes
    sql
  • package sources

    A set of APIs for adding data sources to Spark SQL.

    A set of APIs for adding data sources to Spark SQL.

    Definition Classes
    sql
  • package streaming
    Definition Classes
    sql
  • package types

    Contains a type system for attributes produced by relations, including complex types like structs, arrays and maps.

    Contains a type system for attributes produced by relations, including complex types like structs, arrays and maps.

    Definition Classes
    sql
  • package util
    Definition Classes
    sql
  • package vectorized
    Definition Classes
    sql
p

org.apache.spark.sql

dynamicpruning

package dynamicpruning

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. All

Type Members

  1. case class PlanDynamicPruningFilters(sparkSession: SparkSession) extends Rule[SparkPlan] with PredicateHelper with Product with Serializable

    This planner rule aims at rewriting dynamic pruning predicates in order to reuse the results of broadcast.

    This planner rule aims at rewriting dynamic pruning predicates in order to reuse the results of broadcast. For joins that are not planned as broadcast hash joins we keep the fallback mechanism with subquery duplicate.

Value Members

  1. object CleanupDynamicPruningFilters extends Rule[LogicalPlan] with PredicateHelper

    Removes the filter nodes with dynamic pruning that were not pushed down to the scan.

    Removes the filter nodes with dynamic pruning that were not pushed down to the scan. These nodes will not be pushed through projects and aggregates with non-deterministic expressions.

  2. object PartitionPruning extends Rule[LogicalPlan] with PredicateHelper

    Dynamic partition pruning optimization is performed based on the type and selectivity of the join operation.

    Dynamic partition pruning optimization is performed based on the type and selectivity of the join operation. During query optimization, we insert a predicate on the partitioned table using the filter from the other side of the join and a custom wrapper called DynamicPruning.

    The basic mechanism for DPP inserts a duplicated subquery with the filter from the other side, when the following conditions are met: (1) the table to prune is partitioned by the JOIN key (2) the join operation is one of the following types: INNER, LEFT SEMI (partitioned on left), LEFT OUTER (partitioned on right), or RIGHT OUTER (partitioned on left)

    In order to enable partition pruning directly in broadcasts, we use a custom DynamicPruning clause that incorporates the In clause with the subquery and the benefit estimation. During query planning, when the join type is known, we use the following mechanism: (1) if the join is a broadcast hash join, we replace the duplicated subquery with the reused results of the broadcast, (2) else if the estimated benefit of partition pruning outweighs the overhead of running the subquery query twice, we keep the duplicated subquery (3) otherwise, we drop the subquery.

Ungrouped