Package org.apache.spark.storage

Class FileBasedTopologyMapper

Object
org.apache.spark.storage.TopologyMapper
org.apache.spark.storage.FileBasedTopologyMapper
All Implemented Interfaces:
org.apache.spark.internal.Logging
public class FileBasedTopologyMapper extends TopologyMapper implements org.apache.spark.internal.Logging
A simple file based topology mapper. This expects topology information provided as a java.util.Properties file. The name of the file is obtained from SparkConf property spark.storage.replication.topologyFile. To use this topology mapper, set the spark.storage.replication.topologyMapper property to FileBasedTopologyMapper param: conf SparkConf object

  • Nested Class Summary

    Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

    org.apache.spark.internal.Logging.SparkShellLoggingFilter

  • Constructor Summary

    Constructors
    Constructor
    Description
    FileBasedTopologyMapper(SparkConf conf)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    scala.Option<String>
    getTopologyForHost(String hostname)
    Gets the topology information given the host name
    scala.Option<String>
    topologyFile()
     
    scala.collection.Map<String,String>
    topologyMap()
     

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.spark.internal.Logging

    initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq

  • Constructor Details

    • FileBasedTopologyMapper

      public FileBasedTopologyMapper(SparkConf conf)

  • Method Details

    • getTopologyForHost

      public scala.Option<String> getTopologyForHost(String hostname)
      Description copied from class: TopologyMapper
      Gets the topology information given the host name

      Specified by:
      getTopologyForHost in class TopologyMapper
      Parameters:
      hostname - Hostname
      Returns:
      topology information for the given hostname. One can use a 'topology delimiter' to make this topology information nested. For example : &lsquo;/myrack/myhost&rsquo;, where &lsquo;/&rsquo; is the topology delimiter, &lsquo;myrack&rsquo; is the topology identifier, and &lsquo;myhost&rsquo; is the individual host. This function only returns the topology information without the hostname. This information can be used when choosing executors for block replication to discern executors from a different rack than a candidate executor, for example.

      An implementation can choose to use empty strings or None in case topology info is not available. This would imply that all such executors belong to the same rack.

    • topologyFile

      public scala.Option<String> topologyFile()

    • topologyMap

      public scala.collection.Map<String,String> topologyMap()