public abstract class OutputWriterFactory extends java.lang.Object implements scala.Serializable
OutputWriters. A new
OutputWriterFactoryis created on driver side for each write job issued when writing to a
HadoopFsRelation, and then gets serialized to executor side to create actual
OutputWriters on the fly.
|Constructor and Description|
|Modifier and Type||Method and Description|
public abstract OutputWriter newInstance(java.lang.String path, StructType dataSchema, org.apache.hadoop.mapreduce.TaskAttemptContext context)
HadoopFsRelation, this method gets called by each task on executor side to instantiate new
path- Path of the file to which this
OutputWriteris supposed to write. Note that this may not point to the final output file. For example,
FileOutputFormatwrites to temporary directories and then merge written files back to the final destination. In this case,
pathpoints to a temporary output file under the temporary directory.
dataSchema- Schema of the rows to be written. Partition columns are not included in the schema if the relation being written is partitioned.
context- The Hadoop MapReduce task context.