Package

org.apache.spark

storage

Permalink

package storage

Visibility
  1. Public
  2. All

Type Members

  1. sealed abstract class BlockId extends AnyRef

    Permalink

    :: DeveloperApi :: Identifies a particular Block of data, usually associated with a single file.

    :: DeveloperApi :: Identifies a particular Block of data, usually associated with a single file. A Block can be uniquely identified by its filename, but each type of Block has a different set of keys which produce its unique name.

    If your BlockId should be serializable, be sure to add it to the BlockId.apply() method.

    Annotations
    @DeveloperApi()
  2. class BlockManagerId extends Externalizable

    Permalink

    :: DeveloperApi :: This class represent an unique identifier for a BlockManager.

    :: DeveloperApi :: This class represent an unique identifier for a BlockManager.

    The first 2 constructors of this class are made private to ensure that BlockManagerId objects can be created only using the apply method in the companion object. This allows de-duplication of ID objects. Also, constructor parameters are private to ensure that parameters cannot be modified from outside this class.

    Annotations
    @DeveloperApi()
  3. class BlockNotFoundException extends Exception

    Permalink
  4. trait BlockReplicationPolicy extends AnyRef

    Permalink

    ::DeveloperApi:: BlockReplicationPrioritization provides logic for prioritizing a sequence of peers for replicating blocks.

    ::DeveloperApi:: BlockReplicationPrioritization provides logic for prioritizing a sequence of peers for replicating blocks. BlockManager will replicate to each peer returned in order until the desired replication order is reached. If a replication fails, prioritize() will be called again to get a fresh prioritization.

    Annotations
    @DeveloperApi()
  5. case class BlockStatus(storageLevel: StorageLevel, memSize: Long, diskSize: Long) extends Product with Serializable

    Permalink
    Annotations
    @DeveloperApi()
  6. case class BlockUpdatedInfo(blockManagerId: BlockManagerId, blockId: BlockId, storageLevel: StorageLevel, memSize: Long, diskSize: Long) extends Product with Serializable

    Permalink

    :: DeveloperApi :: Stores information about a block status in a block manager.

    :: DeveloperApi :: Stores information about a block status in a block manager.

    Annotations
    @DeveloperApi()
  7. case class BroadcastBlockId(broadcastId: Long, field: String = "") extends BlockId with Product with Serializable

    Permalink
    Annotations
    @DeveloperApi()
  8. class DefaultTopologyMapper extends TopologyMapper with Logging

    Permalink

    A TopologyMapper that assumes all nodes are in the same rack

    A TopologyMapper that assumes all nodes are in the same rack

    Annotations
    @DeveloperApi()
  9. class FileBasedTopologyMapper extends TopologyMapper with Logging

    Permalink

    A simple file based topology mapper.

    A simple file based topology mapper. This expects topology information provided as a java.util.Properties file. The name of the file is obtained from SparkConf property spark.storage.replication.topologyFile. To use this topology mapper, set the spark.storage.replication.topologyMapper property to org.apache.spark.storage.FileBasedTopologyMapper

    Annotations
    @DeveloperApi()
  10. case class RDDBlockId(rddId: Int, splitIndex: Int) extends BlockId with Product with Serializable

    Permalink
    Annotations
    @DeveloperApi()
  11. class RDDInfo extends Ordered[RDDInfo]

    Permalink
    Annotations
    @DeveloperApi()
  12. class RandomBlockReplicationPolicy extends BlockReplicationPolicy with Logging

    Permalink
    Annotations
    @DeveloperApi()
  13. case class ShuffleBlockId(shuffleId: Int, mapId: Int, reduceId: Int) extends BlockId with Product with Serializable

    Permalink
    Annotations
    @DeveloperApi()
  14. case class ShuffleDataBlockId(shuffleId: Int, mapId: Int, reduceId: Int) extends BlockId with Product with Serializable

    Permalink
    Annotations
    @DeveloperApi()
  15. case class ShuffleIndexBlockId(shuffleId: Int, mapId: Int, reduceId: Int) extends BlockId with Product with Serializable

    Permalink
    Annotations
    @DeveloperApi()
  16. class StorageLevel extends Externalizable

    Permalink

    :: DeveloperApi :: Flags for controlling the storage of an RDD.

    :: DeveloperApi :: Flags for controlling the storage of an RDD. Each StorageLevel records whether to use memory, or ExternalBlockStore, whether to drop the RDD to disk if it falls out of memory or ExternalBlockStore, whether to keep the data in memory in a serialized format, and whether to replicate the RDD partitions on multiple nodes.

    The org.apache.spark.storage.StorageLevel$ singleton object contains some static constants for commonly useful storage levels. To create your own storage level object, use the factory method of the singleton object (StorageLevel(...)).

    Annotations
    @DeveloperApi()
  17. class StorageStatus extends AnyRef

    Permalink

    :: DeveloperApi :: Storage information for each BlockManager.

    :: DeveloperApi :: Storage information for each BlockManager.

    This class assumes BlockId and BlockStatus are immutable, such that the consumers of this class cannot mutate the source of the information. Accesses are not thread-safe.

    Annotations
    @DeveloperApi()
  18. class StorageStatusListener extends SparkListener

    Permalink

    :: DeveloperApi :: A SparkListener that maintains executor storage status.

    :: DeveloperApi :: A SparkListener that maintains executor storage status.

    This class is thread-safe (unlike JobProgressListener)

    Annotations
    @DeveloperApi()
  19. case class StreamBlockId(streamId: Int, uniqueId: Long) extends BlockId with Product with Serializable

    Permalink
    Annotations
    @DeveloperApi()
  20. case class TaskResultBlockId(taskId: Long) extends BlockId with Product with Serializable

    Permalink
    Annotations
    @DeveloperApi()
  21. final class TimeTrackingOutputStream extends OutputStream

    Permalink
  22. abstract class TopologyMapper extends AnyRef

    Permalink

    ::DeveloperApi:: TopologyMapper provides topology information for a given host

    ::DeveloperApi:: TopologyMapper provides topology information for a given host

    Annotations
    @DeveloperApi()

Value Members

  1. object BlockId

    Permalink
    Annotations
    @DeveloperApi()
  2. object BlockStatus extends Serializable

    Permalink
    Annotations
    @DeveloperApi()
  3. object StorageLevel extends Serializable

    Permalink

    Various org.apache.spark.storage.StorageLevel defined and utility functions for creating new storage levels.

Ungrouped