Packages

c

org.apache.spark.internal.io

HadoopMapRedCommitProtocol

class HadoopMapRedCommitProtocol extends HadoopMapReduceCommitProtocol

An FileCommitProtocol implementation backed by an underlying Hadoop OutputCommitter (from the old mapred API).

Unlike Hadoop's OutputCommitter, this implementation is serializable.

Source
HadoopMapRedCommitProtocol.scala
Linear Supertypes
HadoopMapReduceCommitProtocol, Serializable, Serializable, FileCommitProtocol, Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. HadoopMapRedCommitProtocol
  2. HadoopMapReduceCommitProtocol
  3. Serializable
  4. Serializable
  5. FileCommitProtocol
  6. Logging
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new HadoopMapRedCommitProtocol(jobId: String, path: String)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def abortJob(jobContext: JobContext): Unit

    Abort the job; log and ignore any IO exception thrown.

    Abort the job; log and ignore any IO exception thrown. This is invariably invoked in an exception handler; raising an exception here will lose the root cause of the failure.

    jobContext

    job context

    Definition Classes
    HadoopMapReduceCommitProtocol → FileCommitProtocol
  5. def abortTask(taskContext: TaskAttemptContext): Unit

    Abort the task; log and ignore any failure thrown.

    Abort the task; log and ignore any failure thrown. This is invariably invoked in an exception handler; raising an exception here will lose the root cause of the failure.

    taskContext

    context

    Definition Classes
    HadoopMapReduceCommitProtocol → FileCommitProtocol
  6. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  8. def commitJob(jobContext: JobContext, taskCommits: Seq[TaskCommitMessage]): Unit

    Commits a job after the writes succeed.

    Commits a job after the writes succeed. Must be called on the driver.

    Definition Classes
    HadoopMapReduceCommitProtocol → FileCommitProtocol
  9. def commitTask(taskContext: TaskAttemptContext): TaskCommitMessage

    Commits a task after the writes succeed.

    Commits a task after the writes succeed. Must be called on the executors when running tasks.

    Definition Classes
    HadoopMapReduceCommitProtocol → FileCommitProtocol
  10. def deleteWithJob(fs: FileSystem, path: Path, recursive: Boolean): Boolean

    Specifies that a file should be deleted with the commit of this job.

    Specifies that a file should be deleted with the commit of this job. The default implementation deletes the file immediately.

    Definition Classes
    FileCommitProtocol
  11. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  12. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  13. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  14. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  15. def getFilename(taskContext: TaskAttemptContext, ext: String): String
    Attributes
    protected
    Definition Classes
    HadoopMapReduceCommitProtocol
  16. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  17. def initializeForcefully(isInterpreter: Boolean, silent: Boolean): Unit
    Definition Classes
    Logging
  18. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean = false): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  19. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  20. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  21. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  22. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  23. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  24. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  25. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  26. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  27. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  28. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  29. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  30. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  31. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  32. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  33. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  34. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  35. def newTaskTempFile(taskContext: TaskAttemptContext, dir: Option[String], ext: String): String

    Notifies the commit protocol to add a new file, and gets back the full path that should be used.

    Notifies the commit protocol to add a new file, and gets back the full path that should be used. Must be called on the executors when running tasks.

    Note that the returned temp file may have an arbitrary path. The commit protocol only promises that the file will be at the location specified by the arguments after job commit.

    A full file path consists of the following parts:

    1. the base path 2. some sub-directory within the base path, used to specify partitioning 3. file prefix, usually some unique job id with the task id 4. bucket id 5. source specific file extension, e.g. ".snappy.parquet"

    The "dir" parameter specifies 2, and "ext" parameter specifies both 4 and 5, and the rest are left to the commit protocol implementation to decide.

    Important: it is the caller's responsibility to add uniquely identifying content to "ext" if a task is going to write out multiple files to the same dir. The file commit protocol only guarantees that files written by different tasks will not conflict.

    Definition Classes
    HadoopMapReduceCommitProtocol → FileCommitProtocol
  36. def newTaskTempFileAbsPath(taskContext: TaskAttemptContext, absoluteDir: String, ext: String): String

    Similar to newTaskTempFile(), but allows files to committed to an absolute output location.

    Similar to newTaskTempFile(), but allows files to committed to an absolute output location. Depending on the implementation, there may be weaker guarantees around adding files this way.

    Important: it is the caller's responsibility to add uniquely identifying content to "ext" if a task is going to write out multiple files to the same dir. The file commit protocol only guarantees that files written by different tasks will not conflict.

    Definition Classes
    HadoopMapReduceCommitProtocol → FileCommitProtocol
  37. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  38. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  39. def onTaskCommit(taskCommit: TaskCommitMessage): Unit

    Called on the driver after a task commits.

    Called on the driver after a task commits. This can be used to access task commit messages before the job has finished. These same task commit messages will be passed to commitJob() if the entire job succeeds.

    Definition Classes
    FileCommitProtocol
  40. def setupCommitter(context: TaskAttemptContext): OutputCommitter
    Definition Classes
    HadoopMapRedCommitProtocol → HadoopMapReduceCommitProtocol
  41. def setupJob(jobContext: JobContext): Unit

    Setups up a job.

    Setups up a job. Must be called on the driver before any other methods can be invoked.

    Definition Classes
    HadoopMapReduceCommitProtocol → FileCommitProtocol
  42. def setupTask(taskContext: TaskAttemptContext): Unit

    Sets up a task within a job.

    Sets up a task within a job. Must be called before any other task related methods can be invoked.

    Definition Classes
    HadoopMapReduceCommitProtocol → FileCommitProtocol
  43. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  44. def toString(): String
    Definition Classes
    AnyRef → Any
  45. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  46. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  47. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from HadoopMapReduceCommitProtocol

Inherited from Serializable

Inherited from Serializable

Inherited from FileCommitProtocol

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped