Package org.apache.spark.storage

Class DefaultTopologyMapper

Object
org.apache.spark.storage.TopologyMapper
org.apache.spark.storage.DefaultTopologyMapper
All Implemented Interfaces:
org.apache.spark.internal.Logging
public class DefaultTopologyMapper extends TopologyMapper implements org.apache.spark.internal.Logging
A TopologyMapper that assumes all nodes are in the same rack

  • Nested Class Summary

    Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

    org.apache.spark.internal.Logging.LogStringContext, org.apache.spark.internal.Logging.SparkShellLoggingFilter

  • Constructor Summary

    Constructors
    Constructor
    Description
    DefaultTopologyMapper(SparkConf conf)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    scala.Option<String>
    getTopologyForHost(String hostname)
    Gets the topology information given the host name

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.spark.internal.Logging

    initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logDebug, logDebug, logDebug, logDebug, logError, logError, logError, logError, logInfo, logInfo, logInfo, logInfo, logName, LogStringContext, logTrace, logTrace, logTrace, logTrace, logWarning, logWarning, logWarning, logWarning, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, withLogContext

  • Constructor Details

    • DefaultTopologyMapper

      public DefaultTopologyMapper(SparkConf conf)

  • Method Details

    • getTopologyForHost

      public scala.Option<String> getTopologyForHost(String hostname)
      Description copied from class: TopologyMapper
      Gets the topology information given the host name

      Specified by:
      getTopologyForHost in class TopologyMapper
      Parameters:
      hostname - Hostname
      Returns:
      topology information for the given hostname. One can use a 'topology delimiter' to make this topology information nested. For example : &lsquo;/myrack/myhost&rsquo;, where &lsquo;/&rsquo; is the topology delimiter, &lsquo;myrack&rsquo; is the topology identifier, and &lsquo;myhost&rsquo; is the individual host. This function only returns the topology information without the hostname. This information can be used when choosing executors for block replication to discern executors from a different rack than a candidate executor, for example.

      An implementation can choose to use empty strings or None in case topology info is not available. This would imply that all such executors belong to the same rack.