Packages

class BarrierTaskContext extends TaskContext with Logging

:: Experimental :: A TaskContext with extra contextual info and tooling for tasks in a barrier stage. Use BarrierTaskContext#get to obtain the barrier context for a running barrier task.

Annotations
@Experimental() @Since( "2.4.0" )
Source
BarrierTaskContext.scala
Linear Supertypes
Logging, TaskContext, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. BarrierTaskContext
  2. Logging
  3. TaskContext
  4. Serializable
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def addTaskCompletionListener(listener: TaskCompletionListener): BarrierTaskContext.this.type

    Adds a (Java friendly) listener to be executed on task completion.

    Adds a (Java friendly) listener to be executed on task completion. This will be called in all situations - success, failure, or cancellation. Adding a listener to an already completed task will result in that listener being called immediately.

    An example use is for HadoopRDD to register a callback to close the input stream.

    Exceptions thrown by the listener will result in failure of the task.

    Definition Classes
    BarrierTaskContextTaskContext
  5. def addTaskCompletionListener[U](f: (TaskContext) ⇒ U): TaskContext

    Adds a listener in the form of a Scala closure to be executed on task completion.

    Adds a listener in the form of a Scala closure to be executed on task completion. This will be called in all situations - success, failure, or cancellation. Adding a listener to an already completed task will result in that listener being called immediately.

    An example use is for HadoopRDD to register a callback to close the input stream.

    Exceptions thrown by the listener will result in failure of the task.

    Definition Classes
    TaskContext
  6. def addTaskFailureListener(listener: TaskFailureListener): BarrierTaskContext.this.type

    Adds a listener to be executed on task failure.

    Adds a listener to be executed on task failure. Adding a listener to an already failed task will result in that listener being called immediately.

    Definition Classes
    BarrierTaskContextTaskContext
  7. def addTaskFailureListener(f: (TaskContext, Throwable) ⇒ Unit): TaskContext

    Adds a listener to be executed on task failure.

    Adds a listener to be executed on task failure. Adding a listener to an already failed task will result in that listener being called immediately.

    Definition Classes
    TaskContext
  8. def allGather(message: String): Array[String]

    :: Experimental :: Blocks until all tasks in the same stage have reached this routine.

    :: Experimental :: Blocks until all tasks in the same stage have reached this routine. Each task passes in a message and returns with a list of all the messages passed in by each of those tasks.

    CAUTION! The allGather method requires the same precautions as the barrier method

    The message is type String rather than Array[Byte] because it is more convenient for the user at the cost of worse performance.

    Annotations
    @Experimental() @Since( "3.0.0" )
  9. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  10. def attemptNumber(): Int

    How many times this task has been attempted.

    How many times this task has been attempted. The first task attempt will be assigned attemptNumber = 0, and subsequent attempts will have increasing attempt numbers.

    Definition Classes
    BarrierTaskContextTaskContext
  11. def barrier(): Unit

    :: Experimental :: Sets a global barrier and waits until all tasks in this stage hit this barrier.

    :: Experimental :: Sets a global barrier and waits until all tasks in this stage hit this barrier. Similar to MPI_Barrier function in MPI, the barrier() function call blocks until all tasks in the same stage have reached this routine.

    CAUTION! In a barrier stage, each task must have the same number of barrier() calls, in all possible code branches. Otherwise, you may get the job hanging or a SparkException after timeout. Some examples of misuses are listed below: 1. Only call barrier() function on a subset of all the tasks in the same barrier stage, it shall lead to timeout of the function call.

    rdd.barrier().mapPartitions { iter =>
        val context = BarrierTaskContext.get()
        if (context.partitionId() == 0) {
            // Do nothing.
        } else {
            context.barrier()
        }
        iter
    }

    2. Include barrier() function in a try-catch code block, this may lead to timeout of the second function call.

    rdd.barrier().mapPartitions { iter =>
        val context = BarrierTaskContext.get()
        try {
            // Do something that might throw an Exception.
            doSomething()
            context.barrier()
        } catch {
            case e: Exception => logWarning("...", e)
        }
        context.barrier()
        iter
    }
    Annotations
    @Experimental() @Since( "2.4.0" )
  12. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  13. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  14. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  15. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  16. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  17. def getLocalProperty(key: String): String

    Get a local property set upstream in the driver, or null if it is missing.

    Get a local property set upstream in the driver, or null if it is missing. See also org.apache.spark.SparkContext.setLocalProperty.

    Definition Classes
    BarrierTaskContextTaskContext
  18. def getMetricsSources(sourceName: String): Seq[Source]

    ::DeveloperApi:: Returns all metrics sources with the given name which are associated with the instance which runs the task.

    ::DeveloperApi:: Returns all metrics sources with the given name which are associated with the instance which runs the task. For more information see org.apache.spark.metrics.MetricsSystem.

    Definition Classes
    BarrierTaskContextTaskContext
  19. def getTaskInfos(): Array[BarrierTaskInfo]

    :: Experimental :: Returns BarrierTaskInfo for all tasks in this barrier stage, ordered by partition ID.

    :: Experimental :: Returns BarrierTaskInfo for all tasks in this barrier stage, ordered by partition ID.

    Annotations
    @Experimental() @Since( "2.4.0" )
  20. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  21. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  22. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  23. def isCompleted(): Boolean

    Returns true if the task has completed.

    Returns true if the task has completed.

    Definition Classes
    BarrierTaskContextTaskContext
  24. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  25. def isInterrupted(): Boolean

    Returns true if the task has been killed.

    Returns true if the task has been killed.

    Definition Classes
    BarrierTaskContextTaskContext
  26. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  27. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  28. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  29. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  30. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  31. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  32. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  33. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  34. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  35. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  36. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  37. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  38. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  39. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  40. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  41. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  42. def partitionId(): Int

    The ID of the RDD partition that is computed by this task.

    The ID of the RDD partition that is computed by this task.

    Definition Classes
    BarrierTaskContextTaskContext
  43. def resources(): Map[String, ResourceInformation]

    Resources allocated to the task.

    Resources allocated to the task. The key is the resource name and the value is information about the resource. Please refer to org.apache.spark.resource.ResourceInformation for specifics.

    Definition Classes
    BarrierTaskContextTaskContext
  44. def resourcesJMap(): Map[String, ResourceInformation]

    (java-specific) Resources allocated to the task.

    (java-specific) Resources allocated to the task. The key is the resource name and the value is information about the resource. Please refer to org.apache.spark.resource.ResourceInformation for specifics.

    Definition Classes
    BarrierTaskContextTaskContext
  45. def stageAttemptNumber(): Int

    How many times the stage that this task belongs to has been attempted.

    How many times the stage that this task belongs to has been attempted. The first stage attempt will be assigned stageAttemptNumber = 0, and subsequent attempts will have increasing attempt numbers.

    Definition Classes
    BarrierTaskContextTaskContext
  46. def stageId(): Int

    The ID of the stage that this task belong to.

    The ID of the stage that this task belong to.

    Definition Classes
    BarrierTaskContextTaskContext
  47. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  48. def taskAttemptId(): Long

    An ID that is unique to this task attempt (within the same SparkContext, no two task attempts will share the same attempt ID).

    An ID that is unique to this task attempt (within the same SparkContext, no two task attempts will share the same attempt ID). This is roughly equivalent to Hadoop's TaskAttemptID.

    Definition Classes
    BarrierTaskContextTaskContext
  49. def taskMetrics(): TaskMetrics
    Definition Classes
    BarrierTaskContextTaskContext
  50. def toString(): String
    Definition Classes
    AnyRef → Any
  51. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  52. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  53. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Logging

Inherited from TaskContext

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped