public abstract class OutputWriterFactory
extends java.lang.Object
implements scala.Serializable
OutputWriter
s. A new OutputWriterFactory
is created on driver
side for each write job issued when writing to a HadoopFsRelation
, and then gets serialized
to executor side to create actual OutputWriter
s on the fly.
Constructor and Description |
---|
OutputWriterFactory() |
Modifier and Type | Method and Description |
---|---|
abstract OutputWriter |
newInstance(java.lang.String path,
StructType dataSchema,
org.apache.hadoop.mapreduce.TaskAttemptContext context)
When writing to a
HadoopFsRelation , this method gets called by each task on executor side
to instantiate new OutputWriter s. |
public abstract OutputWriter newInstance(java.lang.String path, StructType dataSchema, org.apache.hadoop.mapreduce.TaskAttemptContext context)
HadoopFsRelation
, this method gets called by each task on executor side
to instantiate new OutputWriter
s.
path
- Path of the file to which this OutputWriter
is supposed to write. Note that
this may not point to the final output file. For example, FileOutputFormat
writes to
temporary directories and then merge written files back to the final destination. In
this case, path
points to a temporary output file under the temporary directory.dataSchema
- Schema of the rows to be written. Partition columns are not included in the
schema if the relation being written is partitioned.context
- The Hadoop MapReduce task context.