public class DataFrameReader
extends java.lang.Object
Dataset
from external storage systems (e.g. file systems,
key-value stores, etc) or data streams. Use SparkSession.read
to access this.
Modifier and Type | Method and Description |
---|---|
Dataset<Row> |
csv(scala.collection.Seq<java.lang.String> paths)
Loads a CSV file and returns the result as a
DataFrame . |
Dataset<Row> |
csv(java.lang.String... paths)
Loads a CSV file and returns the result as a
DataFrame . |
DataFrameReader |
format(java.lang.String source)
Specifies the input data source format.
|
Dataset<Row> |
jdbc(java.lang.String url,
java.lang.String table,
java.util.Properties properties)
Construct a
DataFrame representing the database table accessible via JDBC URL
url named table and connection properties. |
Dataset<Row> |
jdbc(java.lang.String url,
java.lang.String table,
java.lang.String[] predicates,
java.util.Properties connectionProperties)
Construct a
DataFrame representing the database table accessible via JDBC URL
url named table using connection properties. |
Dataset<Row> |
jdbc(java.lang.String url,
java.lang.String table,
java.lang.String columnName,
long lowerBound,
long upperBound,
int numPartitions,
java.util.Properties connectionProperties)
Construct a
DataFrame representing the database table accessible via JDBC URL
url named table. |
Dataset<Row> |
json(JavaRDD<java.lang.String> jsonRDD)
Loads an
JavaRDD[String] storing JSON objects (one object per record) and
returns the result as a DataFrame . |
Dataset<Row> |
json(RDD<java.lang.String> jsonRDD)
Loads an
RDD[String] storing JSON objects (one object per record) and
returns the result as a DataFrame . |
Dataset<Row> |
json(scala.collection.Seq<java.lang.String> paths)
Loads a JSON file (one object per line) and returns the result as a
DataFrame . |
Dataset<Row> |
json(java.lang.String... paths)
Loads a JSON file (one object per line) and returns the result as a
DataFrame . |
Dataset<Row> |
load()
Loads input in as a
DataFrame , for data sources that don't require a path (e.g. |
Dataset<Row> |
load(scala.collection.Seq<java.lang.String> paths)
Loads input in as a
DataFrame , for data sources that support multiple paths. |
Dataset<Row> |
load(java.lang.String... paths)
Loads input in as a
DataFrame , for data sources that support multiple paths. |
Dataset<Row> |
load(java.lang.String path)
Loads input in as a
DataFrame , for data sources that require a path (e.g. |
DataFrameReader |
option(java.lang.String key,
boolean value)
Adds an input option for the underlying data source.
|
DataFrameReader |
option(java.lang.String key,
double value)
Adds an input option for the underlying data source.
|
DataFrameReader |
option(java.lang.String key,
long value)
Adds an input option for the underlying data source.
|
DataFrameReader |
option(java.lang.String key,
java.lang.String value)
Adds an input option for the underlying data source.
|
DataFrameReader |
options(scala.collection.Map<java.lang.String,java.lang.String> options)
(Scala-specific) Adds input options for the underlying data source.
|
DataFrameReader |
options(java.util.Map<java.lang.String,java.lang.String> options)
Adds input options for the underlying data source.
|
Dataset<Row> |
orc(java.lang.String path)
Loads an ORC file and returns the result as a
DataFrame . |
Dataset<Row> |
parquet(scala.collection.Seq<java.lang.String> paths)
Loads a Parquet file, returning the result as a
DataFrame . |
Dataset<Row> |
parquet(java.lang.String... paths)
Loads a Parquet file, returning the result as a
DataFrame . |
DataFrameReader |
schema(StructType schema)
Specifies the input schema.
|
Dataset<Row> |
stream()
:: Experimental ::
Loads input data stream in as a
DataFrame , for data streams that don't require a path
(e.g. |
Dataset<Row> |
stream(java.lang.String path)
:: Experimental ::
Loads input in as a
DataFrame , for data streams that read from some path. |
Dataset<Row> |
table(java.lang.String tableName)
Returns the specified table as a
DataFrame . |
Dataset<java.lang.String> |
text(scala.collection.Seq<java.lang.String> paths)
Loads a text file and returns a
Dataset of String. |
Dataset<java.lang.String> |
text(java.lang.String... paths)
Loads a text file and returns a
Dataset of String. |
public Dataset<Row> load(java.lang.String... paths)
DataFrame
, for data sources that support multiple paths.
Only works if the source is a HadoopFsRelationProvider.
paths
- (undocumented)public Dataset<Row> json(java.lang.String... paths)
DataFrame
.
This function goes through the input once to determine the input schema. If you know the schema in advance, use the version that specifies the schema to avoid the extra scan.
You can set the following JSON-specific options to deal with non-standard JSON files:
primitivesAsString
(default false
): infers all primitive values as a string typeprefersDecimal
(default false
): infers all floating-point values as a decimal
type. If the values do not fit in decimal, then it infers them as doubles.allowComments
(default false
): ignores Java/C++ style comment in JSON recordsallowUnquotedFieldNames
(default false
): allows unquoted JSON field namesallowSingleQuotes
(default true
): allows single quotes in addition to double quotes
allowNumericLeadingZeros
(default false
): allows leading zeros in numbers
(e.g. 00012)allowBackslashEscapingAnyCharacter
(default false
): allows accepting quoting of all
character using backslash quoting mechanismmode
(default PERMISSIVE
): allows a mode for dealing with corrupt records
during parsing.PERMISSIVE
: sets other fields to null
when it meets a corrupted record, and puts the
malformed string into a new field configured by columnNameOfCorruptRecord
. When
a schema is set by user, it sets null
for extra fields.DROPMALFORMED
: ignores the whole corrupted records.FAILFAST
: throws an exception when it meets corrupted records.columnNameOfCorruptRecord
(default _corrupt_record
): allows renaming the new field
having malformed string created by PERMISSIVE
mode. This overrides
spark.sql.columnNameOfCorruptRecord
.paths
- (undocumented)public Dataset<Row> csv(java.lang.String... paths)
DataFrame
.
This function goes through the input once to determine the input schema. To avoid going
through the entire data once, specify the schema explicitly using schema
.
You can set the following CSV-specific options to deal with CSV files:
sep
(default ,
): sets the single character as a separator for each
field and value.encoding
(default UTF-8
): decodes the CSV files by the given encoding
type.quote
(default "
): sets the single character used for escaping quoted values where
the separator can be part of the value.escape
(default \
): sets the single character used for escaping quotes inside
an already quoted value.comment
(default empty string): sets the single character used for skipping lines
beginning with this character. By default, it is disabled.header
(default false
): uses the first line as names of columns.ignoreLeadingWhiteSpace
(default false
): defines whether or not leading whitespaces
from values being read should be skipped.ignoreTrailingWhiteSpace
(default false
): defines whether or not trailing
whitespaces from values being read should be skipped.nullValue
(default empty string): sets the string representation of a null value.nanValue
(default NaN
): sets the string representation of a non-number" value.positiveInf
(default Inf
): sets the string representation of a positive infinity
value.negativeInf
(default -Inf
): sets the string representation of a negative infinity
value.dateFormat
(default null
): sets the string that indicates a date format. Custom date
formats follow the formats at java.text.SimpleDateFormat
. This applies to both date type
and timestamp type. By default, it is null
which means trying to parse times and date by
java.sql.Timestamp.valueOf()
and java.sql.Date.valueOf()
.maxColumns
(default 20480
): defines a hard limit of how many columns
a record can have.maxCharsPerColumn
(default 1000000
): defines the maximum number of characters allowed
for any given value being read.mode
(default PERMISSIVE
): allows a mode for dealing with corrupt records
during parsing.PERMISSIVE
: sets other fields to null
when it meets a corrupted record. When
a schema is set by user, it sets null
for extra fields.DROPMALFORMED
: ignores the whole corrupted records.FAILFAST
: throws an exception when it meets corrupted records.paths
- (undocumented)public Dataset<Row> parquet(java.lang.String... paths)
DataFrame
. This function returns an empty
DataFrame
if no paths are passed in.
paths
- (undocumented)public Dataset<java.lang.String> text(java.lang.String... paths)
Dataset
of String. The underlying schema of the Dataset
contains a single string column named "value".
Each line in the text file is a new row in the resulting Dataset. For example:
// Scala:
spark.read.text("/path/to/spark/README.md")
// Java:
spark.read().text("/path/to/spark/README.md")
paths
- input pathpublic DataFrameReader format(java.lang.String source)
source
- (undocumented)public DataFrameReader schema(StructType schema)
schema
- (undocumented)public DataFrameReader option(java.lang.String key, java.lang.String value)
key
- (undocumented)value
- (undocumented)public DataFrameReader option(java.lang.String key, boolean value)
key
- (undocumented)value
- (undocumented)public DataFrameReader option(java.lang.String key, long value)
key
- (undocumented)value
- (undocumented)public DataFrameReader option(java.lang.String key, double value)
key
- (undocumented)value
- (undocumented)public DataFrameReader options(scala.collection.Map<java.lang.String,java.lang.String> options)
options
- (undocumented)public DataFrameReader options(java.util.Map<java.lang.String,java.lang.String> options)
options
- (undocumented)public Dataset<Row> load()
DataFrame
, for data sources that don't require a path (e.g. external
key-value stores).
public Dataset<Row> load(java.lang.String path)
DataFrame
, for data sources that require a path (e.g. data backed by
a local or distributed file system).
path
- (undocumented)public Dataset<Row> load(scala.collection.Seq<java.lang.String> paths)
DataFrame
, for data sources that support multiple paths.
Only works if the source is a HadoopFsRelationProvider.
paths
- (undocumented)public Dataset<Row> stream()
DataFrame
, for data streams that don't require a path
(e.g. external key-value stores).
public Dataset<Row> stream(java.lang.String path)
DataFrame
, for data streams that read from some path.
path
- (undocumented)public Dataset<Row> jdbc(java.lang.String url, java.lang.String table, java.util.Properties properties)
DataFrame
representing the database table accessible via JDBC URL
url named table and connection properties.
url
- (undocumented)table
- (undocumented)properties
- (undocumented)public Dataset<Row> jdbc(java.lang.String url, java.lang.String table, java.lang.String columnName, long lowerBound, long upperBound, int numPartitions, java.util.Properties connectionProperties)
DataFrame
representing the database table accessible via JDBC URL
url named table. Partitions of the table will be retrieved in parallel based on the parameters
passed to this function.
Don't create too many partitions in parallel on a large cluster; otherwise Spark might crash your external database systems.
url
- JDBC database url of the form jdbc:subprotocol:subname
.table
- Name of the table in the external database.columnName
- the name of a column of integral type that will be used for partitioning.lowerBound
- the minimum value of columnName
used to decide partition stride.upperBound
- the maximum value of columnName
used to decide partition stride.numPartitions
- the number of partitions. This, along with lowerBound
(inclusive),
upperBound
(exclusive), form partition strides for generated WHERE
clause expressions used to split the column columnName
evenly.connectionProperties
- JDBC database connection arguments, a list of arbitrary string
tag/value. Normally at least a "user" and "password" property
should be included.public Dataset<Row> jdbc(java.lang.String url, java.lang.String table, java.lang.String[] predicates, java.util.Properties connectionProperties)
DataFrame
representing the database table accessible via JDBC URL
url named table using connection properties. The predicates
parameter gives a list
expressions suitable for inclusion in WHERE clauses; each one defines one partition
of the DataFrame
.
Don't create too many partitions in parallel on a large cluster; otherwise Spark might crash your external database systems.
url
- JDBC database url of the form jdbc:subprotocol:subname
table
- Name of the table in the external database.predicates
- Condition in the where clause for each partition.connectionProperties
- JDBC database connection arguments, a list of arbitrary string
tag/value. Normally at least a "user" and "password" property
should be included.public Dataset<Row> json(scala.collection.Seq<java.lang.String> paths)
DataFrame
.
This function goes through the input once to determine the input schema. If you know the schema in advance, use the version that specifies the schema to avoid the extra scan.
You can set the following JSON-specific options to deal with non-standard JSON files:
primitivesAsString
(default false
): infers all primitive values as a string typeprefersDecimal
(default false
): infers all floating-point values as a decimal
type. If the values do not fit in decimal, then it infers them as doubles.allowComments
(default false
): ignores Java/C++ style comment in JSON recordsallowUnquotedFieldNames
(default false
): allows unquoted JSON field namesallowSingleQuotes
(default true
): allows single quotes in addition to double quotes
allowNumericLeadingZeros
(default false
): allows leading zeros in numbers
(e.g. 00012)allowBackslashEscapingAnyCharacter
(default false
): allows accepting quoting of all
character using backslash quoting mechanismmode
(default PERMISSIVE
): allows a mode for dealing with corrupt records
during parsing.PERMISSIVE
: sets other fields to null
when it meets a corrupted record, and puts the
malformed string into a new field configured by columnNameOfCorruptRecord
. When
a schema is set by user, it sets null
for extra fields.DROPMALFORMED
: ignores the whole corrupted records.FAILFAST
: throws an exception when it meets corrupted records.columnNameOfCorruptRecord
(default _corrupt_record
): allows renaming the new field
having malformed string created by PERMISSIVE
mode. This overrides
spark.sql.columnNameOfCorruptRecord
.paths
- (undocumented)public Dataset<Row> json(JavaRDD<java.lang.String> jsonRDD)
JavaRDD[String]
storing JSON objects (one object per record) and
returns the result as a DataFrame
.
Unless the schema is specified using schema
function, this function goes through the
input once to determine the input schema.
jsonRDD
- input RDD with one JSON object per recordpublic Dataset<Row> json(RDD<java.lang.String> jsonRDD)
RDD[String]
storing JSON objects (one object per record) and
returns the result as a DataFrame
.
Unless the schema is specified using schema
function, this function goes through the
input once to determine the input schema.
jsonRDD
- input RDD with one JSON object per recordpublic Dataset<Row> csv(scala.collection.Seq<java.lang.String> paths)
DataFrame
.
This function goes through the input once to determine the input schema. To avoid going
through the entire data once, specify the schema explicitly using schema
.
You can set the following CSV-specific options to deal with CSV files:
sep
(default ,
): sets the single character as a separator for each
field and value.encoding
(default UTF-8
): decodes the CSV files by the given encoding
type.quote
(default "
): sets the single character used for escaping quoted values where
the separator can be part of the value.escape
(default \
): sets the single character used for escaping quotes inside
an already quoted value.comment
(default empty string): sets the single character used for skipping lines
beginning with this character. By default, it is disabled.header
(default false
): uses the first line as names of columns.ignoreLeadingWhiteSpace
(default false
): defines whether or not leading whitespaces
from values being read should be skipped.ignoreTrailingWhiteSpace
(default false
): defines whether or not trailing
whitespaces from values being read should be skipped.nullValue
(default empty string): sets the string representation of a null value.nanValue
(default NaN
): sets the string representation of a non-number" value.positiveInf
(default Inf
): sets the string representation of a positive infinity
value.negativeInf
(default -Inf
): sets the string representation of a negative infinity
value.dateFormat
(default null
): sets the string that indicates a date format. Custom date
formats follow the formats at java.text.SimpleDateFormat
. This applies to both date type
and timestamp type. By default, it is null
which means trying to parse times and date by
java.sql.Timestamp.valueOf()
and java.sql.Date.valueOf()
.maxColumns
(default 20480
): defines a hard limit of how many columns
a record can have.maxCharsPerColumn
(default 1000000
): defines the maximum number of characters allowed
for any given value being read.mode
(default PERMISSIVE
): allows a mode for dealing with corrupt records
during parsing.PERMISSIVE
: sets other fields to null
when it meets a corrupted record. When
a schema is set by user, it sets null
for extra fields.DROPMALFORMED
: ignores the whole corrupted records.FAILFAST
: throws an exception when it meets corrupted records.paths
- (undocumented)public Dataset<Row> parquet(scala.collection.Seq<java.lang.String> paths)
DataFrame
. This function returns an empty
DataFrame
if no paths are passed in.
paths
- (undocumented)public Dataset<Row> orc(java.lang.String path)
DataFrame
.
path
- input pathpublic Dataset<Row> table(java.lang.String tableName)
DataFrame
.
tableName
- (undocumented)public Dataset<java.lang.String> text(scala.collection.Seq<java.lang.String> paths)
Dataset
of String. The underlying schema of the Dataset
contains a single string column named "value".
Each line in the text file is a new row in the resulting Dataset. For example:
// Scala:
spark.read.text("/path/to/spark/README.md")
// Java:
spark.read().text("/path/to/spark/README.md")
paths
- input path