Package org.apache.spark.sql

Class SparkSession.Builder

Object
org.apache.spark.sql.SparkSession.Builder
Enclosing class:
SparkSession
public static class SparkSession.Builder extends Object

  • Constructor Details

    • Builder

      public Builder()

  • Method Details

    • appName

      public SparkSession.Builder appName(String name)
      Sets a name for the application, which will be shown in the Spark web UI. If no application name is set, a randomly generated name will be used.

      Parameters:
      name - (undocumented)
      Returns:
      (undocumented)
      Inheritdoc:

    • classic

      public SparkSession.Builder classic()
      Make the builder create a Classic SparkSession.
      Returns:
      (undocumented)

    • config

      public SparkSession.Builder config(String key, String value)
      Sets a config option. Options set using this method are automatically propagated to both SparkConf and SparkSession's own configuration.

      Parameters:
      key - (undocumented)
      value - (undocumented)
      Returns:
      (undocumented)
      Inheritdoc:

    • config

      public SparkSession.Builder config(String key, long value)
      Sets a config option. Options set using this method are automatically propagated to both SparkConf and SparkSession's own configuration.

      Parameters:
      key - (undocumented)
      value - (undocumented)
      Returns:
      (undocumented)
      Inheritdoc:

    • config

      public SparkSession.Builder config(String key, double value)
      Sets a config option. Options set using this method are automatically propagated to both SparkConf and SparkSession's own configuration.

      Parameters:
      key - (undocumented)
      value - (undocumented)
      Returns:
      (undocumented)
      Inheritdoc:

    • config

      public SparkSession.Builder config(String key, boolean value)
      Sets a config option. Options set using this method are automatically propagated to both SparkConf and SparkSession's own configuration.

      Parameters:
      key - (undocumented)
      value - (undocumented)
      Returns:
      (undocumented)
      Inheritdoc:

    • config

      public SparkSession.Builder config(scala.collection.immutable.Map<String,Object> map)
      Sets a config option. Options set using this method are automatically propagated to both SparkConf and SparkSession's own configuration.

      Parameters:
      map - (undocumented)
      Returns:
      (undocumented)
      Inheritdoc:

    • config

      public SparkSession.Builder config(Map<String,Object> map)
      Sets a config option. Options set using this method are automatically propagated to both SparkConf and SparkSession's own configuration.

      Parameters:
      map - (undocumented)
      Returns:
      (undocumented)
      Inheritdoc:

    • config

      public SparkSession.Builder config(SparkConf conf)
      Sets a list of config options based on the given SparkConf.

      Parameters:
      conf - (undocumented)
      Returns:
      (undocumented)
      Inheritdoc:

    • connect

      public SparkSession.Builder connect()
      Make the builder create a Connect SparkSession.
      Returns:
      (undocumented)

    • create

      public SparkSession create()
      Create a new SparkSession.

      This will always return a newly created session.

      This method will update the default and/or active session if they are not set.

      Returns:
      (undocumented)
      Inheritdoc:

    • enableHiveSupport

      public SparkSession.Builder enableHiveSupport()
      Enables Hive support, including connectivity to a persistent Hive metastore, support for Hive serdes, and Hive user-defined functions.

      Returns:
      (undocumented)
      Inheritdoc:

    • getOrCreate

      public SparkSession getOrCreate()
      Gets an existing SparkSession or, if there is no existing one, creates a new one based on the options set in this builder.

      This method first checks whether there is a valid thread-local SparkSession, and if yes, return that one. It then checks whether there is a valid global default SparkSession, and if yes, return that one. If no valid global default SparkSession exists, the method creates a new SparkSession and assigns the newly created SparkSession as the global default.

      In case an existing SparkSession is returned, the non-static config options specified in this builder will be applied to the existing SparkSession.

      Returns:
      (undocumented)
      Inheritdoc:

    • master

      public SparkSession.Builder master(String master)
      Sets the Spark master URL to connect to, such as "local" to run locally, "local[4]" to run locally with 4 cores, or "spark://master:7077" to run on a Spark standalone cluster.

      Parameters:
      master - (undocumented)
      Returns:
      (undocumented)
      Inheritdoc:

    • remote

      public SparkSession.Builder remote(String connectionString)
      Sets the Spark Connect remote URL.

      Parameters:
      connectionString - (undocumented)
      Returns:
      (undocumented)
      Inheritdoc:

    • withExtensions

      public SparkSession.Builder withExtensions(scala.Function1<SparkSessionExtensions,scala.runtime.BoxedUnit> f)
      Inject extensions into the SparkSession. This allows a user to add Analyzer rules, Optimizer rules, Planning Strategies or a customized parser.

      Parameters:
      f - (undocumented)
      Returns:
      (undocumented)
      Inheritdoc:

    • MASTER_KEY

      public static String MASTER_KEY()

    • APP_NAME_KEY

      public static String APP_NAME_KEY()

    • CATALOG_IMPL_KEY

      public static String CATALOG_IMPL_KEY()

    • CONNECT_REMOTE_KEY

      public static String CONNECT_REMOTE_KEY()

    • API_MODE_KEY

      public static String API_MODE_KEY()

    • API_MODE_CLASSIC

      public static String API_MODE_CLASSIC()

    • API_MODE_CONNECT

      public static String API_MODE_CONNECT()