Package org.apache.spark.resource

Class ResourceProfile

Object
org.apache.spark.resource.ResourceProfile
All Implemented Interfaces:
Serializable, org.apache.spark.internal.Logging, scala.Serializable
public class ResourceProfile extends Object implements scala.Serializable, org.apache.spark.internal.Logging
Resource profile to associate with an RDD. A ResourceProfile allows the user to specify executor and task requirements for an RDD that will get applied during a stage. This allows the user to change the resource requirements between stages. This is meant to be immutable so user can't change it after building. Users should use ResourceProfileBuilder to build it.

param: executorResources Resource requests for executors. Mapped from the resource name (e.g., cores, memory, CPU) to its specific request. param: taskResources Resource requests for tasks. Mapped from the resource name (e.g., cores, memory, CPU) to its specific request.

See Also:

  • Constructor Details

  • Method Details

    • CPUS

      public static String CPUS()
      built-in task resource: cpus
      Returns:
      (undocumented)

    • CORES

      public static String CORES()
      built-in executor resource: cores
      Returns:
      (undocumented)

    • MEMORY

      public static String MEMORY()
      built-in executor resource: cores
      Returns:
      (undocumented)

    • OFFHEAP_MEM

      public static String OFFHEAP_MEM()
      built-in executor resource: offHeap
      Returns:
      (undocumented)

    • OVERHEAD_MEM

      public static String OVERHEAD_MEM()
      built-in executor resource: memoryOverhead
      Returns:
      (undocumented)

    • PYSPARK_MEM

      public static String PYSPARK_MEM()
      built-in executor resource: pyspark.memory
      Returns:
      (undocumented)

    • allSupportedExecutorResources

      public static String[] allSupportedExecutorResources()
      Return all supported Spark built-in executor resources, custom resources like GPUs/FPGAs are excluded.
      Returns:
      (undocumented)

    • UNKNOWN_RESOURCE_PROFILE_ID

      public static int UNKNOWN_RESOURCE_PROFILE_ID()

    • DEFAULT_RESOURCE_PROFILE_ID

      public static int DEFAULT_RESOURCE_PROFILE_ID()

    • org$apache$spark$internal$Logging$$log_

      public static org.slf4j.Logger org$apache$spark$internal$Logging$$log_()

    • org$apache$spark$internal$Logging$$log__$eq

      public static void org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger x$1)

    • executorResources

      public scala.collection.immutable.Map<String,ExecutorResourceRequest> executorResources()

    • taskResources

      public scala.collection.immutable.Map<String,TaskResourceRequest> taskResources()

    • id

      public int id()
      A unique id of this ResourceProfile
      Returns:
      (undocumented)

    • taskResourcesJMap

      public Map<String,TaskResourceRequest> taskResourcesJMap()
      (Java-specific) gets a Java Map of resources to TaskResourceRequest
      Returns:
      (undocumented)

    • executorResourcesJMap

      public Map<String,ExecutorResourceRequest> executorResourcesJMap()
      (Java-specific) gets a Java Map of resources to ExecutorResourceRequest
      Returns:
      (undocumented)

    • equals

      public boolean equals(Object obj)
      Overrides:
      equals in class Object

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object

    • toString

      public String toString()
      Overrides:
      toString in class Object