Package org.apache.spark

Class SparkEnv

Object
org.apache.spark.SparkEnv
All Implemented Interfaces:
org.apache.spark.internal.Logging
public class SparkEnv extends Object implements org.apache.spark.internal.Logging
:: DeveloperApi :: Holds all the runtime environment objects for a running Spark instance (either master or worker), including the serializer, RpcEnv, block manager, map output tracker, etc. Currently Spark code finds the SparkEnv through a global variable, so all the threads can access the same SparkEnv. It can be accessed by SparkEnv.get (e.g. after creating a SparkContext).

  • Nested Class Summary

    Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

    org.apache.spark.internal.Logging.LogStringContext, org.apache.spark.internal.Logging.SparkShellLoggingFilter

  • Constructor Summary

    Constructors
    Constructor
    Description
    SparkEnv(String executorId, org.apache.spark.rpc.RpcEnv rpcEnv, Serializer serializer, Serializer closureSerializer, org.apache.spark.serializer.SerializerManager serializerManager, org.apache.spark.MapOutputTracker mapOutputTracker, org.apache.spark.broadcast.BroadcastManager broadcastManager, org.apache.spark.storage.BlockManager blockManager, org.apache.spark.SecurityManager securityManager, org.apache.spark.metrics.MetricsSystem metricsSystem, org.apache.spark.scheduler.OutputCommitCoordinator outputCommitCoordinator, SparkConf conf)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    org.apache.spark.storage.BlockManager
    blockManager()
     
    org.apache.spark.broadcast.BroadcastManager
    broadcastManager()
     
    Serializer
    closureSerializer()
     
    SparkConf
    conf()
     
    String
    executorId()
     
    static SparkEnv
    get()
    Returns the SparkEnv.
    static org.apache.spark.internal.Logging.LogStringContext
    LogStringContext(scala.StringContext sc)
     
    org.apache.spark.MapOutputTracker
    mapOutputTracker()
     
    org.apache.spark.memory.MemoryManager
    memoryManager()
     
    org.apache.spark.metrics.MetricsSystem
    metricsSystem()
     
    static org.slf4j.Logger
    org$apache$spark$internal$Logging$$log_()
     
    static void
    org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger x$1)
     
    org.apache.spark.scheduler.OutputCommitCoordinator
    outputCommitCoordinator()
     
    org.apache.spark.SecurityManager
    securityManager()
     
    Serializer
    serializer()
     
    org.apache.spark.serializer.SerializerManager
    serializerManager()
     
    static void
    set(SparkEnv e)
     
    org.apache.spark.shuffle.ShuffleManager
    shuffleManager()
    Deprecated.
    use shuffleManagerFor(dependency) to route a shuffle by type, or blockingShuffleManager for the blocking manager explicitly.

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.spark.internal.Logging

    initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logBasedOnLevel, logDebug, logDebug, logDebug, logDebug, logError, logError, logError, logError, logInfo, logInfo, logInfo, logInfo, logName, LogStringContext, logTrace, logTrace, logTrace, logTrace, logWarning, logWarning, logWarning, logWarning, MDC, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, withLogContext

  • Constructor Details

    • SparkEnv

      public SparkEnv(String executorId, org.apache.spark.rpc.RpcEnv rpcEnv, Serializer serializer, Serializer closureSerializer, org.apache.spark.serializer.SerializerManager serializerManager, org.apache.spark.MapOutputTracker mapOutputTracker, org.apache.spark.broadcast.BroadcastManager broadcastManager, org.apache.spark.storage.BlockManager blockManager, org.apache.spark.SecurityManager securityManager, org.apache.spark.metrics.MetricsSystem metricsSystem, org.apache.spark.scheduler.OutputCommitCoordinator outputCommitCoordinator, SparkConf conf)

  • Method Details

    • set

      public static void set(SparkEnv e)

    • get

      public static SparkEnv get()
      Returns the SparkEnv.
      Returns:
      (undocumented)

    • org$apache$spark$internal$Logging$$log_

      public static org.slf4j.Logger org$apache$spark$internal$Logging$$log_()

    • org$apache$spark$internal$Logging$$log__$eq

      public static void org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger x$1)

    • LogStringContext

      public static org.apache.spark.internal.Logging.LogStringContext LogStringContext(scala.StringContext sc)

    • executorId

      public String executorId()

    • serializer

      public Serializer serializer()

    • closureSerializer

      public Serializer closureSerializer()

    • serializerManager

      public org.apache.spark.serializer.SerializerManager serializerManager()

    • mapOutputTracker

      public org.apache.spark.MapOutputTracker mapOutputTracker()

    • broadcastManager

      public org.apache.spark.broadcast.BroadcastManager broadcastManager()

    • blockManager

      public org.apache.spark.storage.BlockManager blockManager()

    • securityManager

      public org.apache.spark.SecurityManager securityManager()

    • metricsSystem

      public org.apache.spark.metrics.MetricsSystem metricsSystem()

    • outputCommitCoordinator

      public org.apache.spark.scheduler.OutputCommitCoordinator outputCommitCoordinator()

    • conf

      public SparkConf conf()

    • shuffleManager

      public org.apache.spark.shuffle.ShuffleManager shuffleManager()
      Deprecated.
      use shuffleManagerFor(dependency) to route a shuffle by type, or blockingShuffleManager for the blocking manager explicitly. Since 4.3.0.
      Retained for binary compatibility; returns the blocking manager. Prefer shuffleManagerFor (route a shuffle by its dependency) or blockingShuffleManager (the blocking manager explicitly), so the routing decision is explicit at the call site.
      Returns:
      (undocumented)

    • memoryManager

      public org.apache.spark.memory.MemoryManager memoryManager()