Package org.apache.spark.sql.types
Class StructType
Object
org.apache.spark.sql.types.DataType
org.apache.spark.sql.types.StructType
- All Implemented Interfaces:
Serializable
,scala.collection.generic.FilterMonadic<StructField,
,scala.collection.Traversable<StructField>> scala.collection.generic.GenericTraversableTemplate<StructField,
,scala.collection.GenTraversable> scala.collection.generic.HasNewBuilder<StructField,
,scala.collection.Traversable<StructField>> scala.collection.GenIterable<StructField>
,scala.collection.GenIterableLike<StructField,
,scala.collection.GenIterable<StructField>> scala.collection.GenSeq<StructField>
,scala.collection.GenSeqLike<StructField,
,scala.collection.GenSeq<StructField>> scala.collection.GenTraversable<StructField>
,scala.collection.GenTraversableLike<StructField,
,scala.collection.Traversable<StructField>> scala.collection.GenTraversableOnce<StructField>
,scala.collection.Iterable<StructField>
,scala.collection.IterableLike<StructField,
,scala.collection.Iterable<StructField>> scala.collection.Parallelizable<StructField,
,scala.collection.parallel.ParIterable<StructField>> scala.collection.Seq<StructField>
,scala.collection.SeqLike<StructField,
,scala.collection.Seq<StructField>> scala.collection.Traversable<StructField>
,scala.collection.TraversableLike<StructField,
,scala.collection.Traversable<StructField>> scala.collection.TraversableOnce<StructField>
,scala.Equals
,scala.Function1<Object,
,StructField> scala.PartialFunction<Object,
,StructField> scala.Product
,scala.Serializable
public class StructType
extends DataType
implements scala.collection.Seq<StructField>, scala.Product, scala.Serializable
A
StructType
object can be constructed by
StructType(fields: Seq[StructField])
For a StructType
object, one or multiple StructField
s can be extracted by names.
If multiple StructField
s are extracted, a StructType
object will be returned.
If a provided name does not have a matching field, it will be ignored. For the case
of extracting a single StructField
, a null
will be returned.
Scala Example:
import org.apache.spark.sql._
import org.apache.spark.sql.types._
val struct =
StructType(
StructField("a", IntegerType, true) ::
StructField("b", LongType, false) ::
StructField("c", BooleanType, false) :: Nil)
// Extract a single StructField.
val singleField = struct("b")
// singleField: StructField = StructField(b,LongType,false)
// If this struct does not have a field called "d", it throws an exception.
struct("d")
// java.lang.IllegalArgumentException: d does not exist.
// ...
// Extract multiple StructFields. Field names are provided in a set.
// A StructType object will be returned.
val twoFields = struct(Set("b", "c"))
// twoFields: StructType =
// StructType(StructField(b,LongType,false), StructField(c,BooleanType,false))
// Any names without matching fields will throw an exception.
// For the case shown below, an exception is thrown due to "d".
struct(Set("b", "c", "d"))
// java.lang.IllegalArgumentException: d does not exist.
// ...
A Row
object is used as a value of the StructType
.
Scala Example:
import org.apache.spark.sql._
import org.apache.spark.sql.types._
val innerStruct =
StructType(
StructField("f1", IntegerType, true) ::
StructField("f2", LongType, false) ::
StructField("f3", BooleanType, false) :: Nil)
val struct = StructType(
StructField("a", innerStruct, true) :: Nil)
// Create a Row with the schema defined by struct
val row = Row(Row(1, 2, true))
- Since:
- 1.3.0
- See Also:
-
Nested Class Summary
Nested classes/interfaces inherited from interface scala.collection.TraversableLike
scala.collection.TraversableLike.WithFilter
Nested classes/interfaces inherited from interface scala.collection.TraversableOnce
scala.collection.TraversableOnce.BufferedCanBuildFrom<A extends Object,
CC extends scala.collection.TraversableOnce<Object>>, scala.collection.TraversableOnce.FlattenOps<A extends Object>, scala.collection.TraversableOnce.ForceImplicitAmbiguity, scala.collection.TraversableOnce.MonadOps<A extends Object>, scala.collection.TraversableOnce.OnceCanBuildFrom<A extends Object> -
Constructor Summary
-
Method Summary
Modifier and TypeMethodDescriptionCreates a newStructType
by adding a new nullable field with no metadata where the dataType is specified as a String.Creates a newStructType
by adding a new field with no metadata where the dataType is specified as a String.Creates a newStructType
by adding a new field and specifying metadata where the dataType is specified as a String.Creates a newStructType
by adding a new field and specifying metadata where the dataType is specified as a String.Creates a newStructType
by adding a new nullable field with no metadata.Creates a newStructType
by adding a new field with no metadata.Creates a newStructType
by adding a new field and specifying metadata.Creates a newStructType
by adding a new field and specifying metadata.add
(StructField field) Creates a newStructType
by adding a new field.apply
(int fieldIndex) Extracts theStructField
with the given name.Returns aStructType
containingStructField
s of the given names, preserving the original order of fields.String representation for the type saved in external catalogs.int
The default size of a value of the StructType is the total default sizes of all field types.boolean
int
fieldIndex
(String name) Returns the index of a given field.String[]
Returns all field names in an array.fields()
static scala.Option<StructType>
findMissingFields
(StructType source, StructType target, scala.Function2<String, String, Object> resolver) Returns aStructType
that contains missing fields recursively fromsource
totarget
.static StructType
Creates StructType for a given DDL-formatted string, which is a comma separated list of field definitions, e.g., a INT, b STRING.int
hashCode()
scala.collection.Iterator<StructField>
iterator()
int
length()
String[]
names()
Returns all field names in an array.void
Readable string representation for the type.sql()
toDDL()
Returns a string containing a schema in DDL format.toString()
treeString
(int maxDepth) Methods inherited from class org.apache.spark.sql.types.DataType
equalsIgnoreCaseAndNullability, equalsIgnoreNullability, equalsStructurally, equalsStructurallyByName, fromJson, json, parseTypeWithFallback, prettyJson, typeName
Methods inherited from interface scala.Function1
apply, apply$mcDD$sp, apply$mcDF$sp, apply$mcDI$sp, apply$mcDJ$sp, apply$mcFD$sp, apply$mcFF$sp, apply$mcFI$sp, apply$mcFJ$sp, apply$mcID$sp, apply$mcIF$sp, apply$mcII$sp, apply$mcIJ$sp, apply$mcJD$sp, apply$mcJF$sp, apply$mcJI$sp, apply$mcJJ$sp, apply$mcVD$sp, apply$mcVF$sp, apply$mcVI$sp, apply$mcVJ$sp, apply$mcZD$sp, apply$mcZF$sp, apply$mcZI$sp, apply$mcZJ$sp, compose
Methods inherited from interface scala.collection.generic.GenericTraversableTemplate
flatten, foreach, genericBuilder, head, isEmpty, newBuilder, transpose, unzip, unzip3
Methods inherited from interface scala.collection.GenSeqLike
indexOf, indexOf, indexWhere, isDefinedAt, lastIndexOf, lastIndexOf, lastIndexWhere, prefixLength, startsWith, toSeq
Methods inherited from interface scala.collection.GenTraversableLike
flatMap, foreach, map, size
Methods inherited from interface scala.collection.GenTraversableOnce
sizeHintIfCheap
Methods inherited from interface scala.collection.IterableLike
canEqual, copyToArray, drop, dropRight, exists, find, foldRight, forall, foreach, grouped, head, reduceRight, sameElements, slice, sliding, sliding, take, takeRight, takeWhile, toIterable, toIterator, toStream, zip, zipAll, zipWithIndex
Methods inherited from interface scala.collection.Parallelizable
par
Methods inherited from interface scala.PartialFunction
andThen, applyOrElse, isDefinedAt, lift, orElse, runWith
Methods inherited from interface scala.Product
productArity, productElement, productIterator, productPrefix
Methods inherited from interface scala.collection.Seq
companion, seq
Methods inherited from interface scala.collection.SeqLike
$colon$plus, $plus$colon, combinations, contains, containsSlice, corresponds, diff, distinct, endsWith, indexOfSlice, indexOfSlice, indexWhere, indices, intersect, isEmpty, lastIndexOfSlice, lastIndexOfSlice, lastIndexWhere, lengthCompare, padTo, parCombiner, patch, permutations, reverse, reverseIterator, reverseMap, segmentLength, size, sortBy, sorted, sortWith, startsWith, thisCollection, toCollection, toSeq, union, updated, view, view
Methods inherited from interface scala.collection.TraversableLike
$plus$plus, $plus$plus$colon, $plus$plus$colon, collect, dropWhile, filter, filterImpl, filterNot, flatMap, groupBy, hasDefiniteSize, headOption, init, inits, isTraversableAgain, last, lastOption, map, newBuilder, partition, repr, scan, scanLeft, scanRight, sliceWithKnownBound, sliceWithKnownDelta, span, splitAt, stringPrefix, tail, tails, to, toTraversable, withFilter
Methods inherited from interface scala.collection.TraversableOnce
$colon$bslash, $div$colon, addString, addString, addString, aggregate, collectFirst, copyToArray, copyToArray, copyToBuffer, count, fold, foldLeft, max, maxBy, min, minBy, mkString, mkString, mkString, nonEmpty, product, reduce, reduceLeft, reduceLeftOption, reduceOption, reduceRightOption, reversed, sum, toArray, toBuffer, toIndexedSeq, toList, toMap, toSet, toVector
-
Constructor Details
-
StructType
-
StructType
public StructType()No-arg constructor for kryo.
-
-
Method Details
-
fromDDL
Creates StructType for a given DDL-formatted string, which is a comma separated list of field definitions, e.g., a INT, b STRING.- Parameters:
ddl
- (undocumented)- Returns:
- (undocumented)
- Since:
- 2.2.0
-
findMissingFields
public static scala.Option<StructType> findMissingFields(StructType source, StructType target, scala.Function2<String, String, Object> resolver) Returns aStructType
that contains missing fields recursively fromsource
totarget
. Note that this doesn't support looking into array type and map type recursively.- Parameters:
source
- (undocumented)target
- (undocumented)resolver
- (undocumented)- Returns:
- (undocumented)
-
fields
-
fieldNames
Returns all field names in an array. -
names
Returns all field names in an array. This is an alias offieldNames
.- Returns:
- (undocumented)
- Since:
- 2.4.0
-
equals
- Specified by:
equals
in interfacescala.Equals
- Specified by:
equals
in interfacescala.collection.GenSeqLike<StructField,
scala.collection.GenSeq<StructField>> - Overrides:
equals
in classObject
-
toString
- Specified by:
toString
in interfacescala.Function1<Object,
StructField> - Specified by:
toString
in interfacescala.collection.SeqLike<StructField,
scala.collection.Seq<StructField>> - Specified by:
toString
in interfacescala.collection.TraversableLike<StructField,
scala.collection.Traversable<StructField>> - Overrides:
toString
in classObject
-
hashCode
public int hashCode()- Specified by:
hashCode
in interfacescala.collection.GenSeqLike<StructField,
scala.collection.GenSeq<StructField>> - Overrides:
hashCode
in classObject
-
add
Creates a newStructType
by adding a new field.val struct = (new StructType) .add(StructField("a", IntegerType, true)) .add(StructField("b", LongType, false)) .add(StructField("c", StringType, true))
- Parameters:
field
- (undocumented)- Returns:
- (undocumented)
-
add
Creates a newStructType
by adding a new nullable field with no metadata.val struct = (new StructType) .add("a", IntegerType) .add("b", LongType) .add("c", StringType)
- Parameters:
name
- (undocumented)dataType
- (undocumented)- Returns:
- (undocumented)
-
add
Creates a newStructType
by adding a new field with no metadata.val struct = (new StructType) .add("a", IntegerType, true) .add("b", LongType, false) .add("c", StringType, true)
- Parameters:
name
- (undocumented)dataType
- (undocumented)nullable
- (undocumented)- Returns:
- (undocumented)
-
add
Creates a newStructType
by adding a new field and specifying metadata.val struct = (new StructType) .add("a", IntegerType, true, Metadata.empty) .add("b", LongType, false, Metadata.empty) .add("c", StringType, true, Metadata.empty)
- Parameters:
name
- (undocumented)dataType
- (undocumented)nullable
- (undocumented)metadata
- (undocumented)- Returns:
- (undocumented)
-
add
Creates a newStructType
by adding a new field and specifying metadata.val struct = (new StructType) .add("a", IntegerType, true, "comment1") .add("b", LongType, false, "comment2") .add("c", StringType, true, "comment3")
- Parameters:
name
- (undocumented)dataType
- (undocumented)nullable
- (undocumented)comment
- (undocumented)- Returns:
- (undocumented)
-
add
Creates a newStructType
by adding a new nullable field with no metadata where the dataType is specified as a String.val struct = (new StructType) .add("a", "int") .add("b", "long") .add("c", "string")
- Parameters:
name
- (undocumented)dataType
- (undocumented)- Returns:
- (undocumented)
-
add
Creates a newStructType
by adding a new field with no metadata where the dataType is specified as a String.val struct = (new StructType) .add("a", "int", true) .add("b", "long", false) .add("c", "string", true)
- Parameters:
name
- (undocumented)dataType
- (undocumented)nullable
- (undocumented)- Returns:
- (undocumented)
-
add
Creates a newStructType
by adding a new field and specifying metadata where the dataType is specified as a String.val struct = (new StructType) .add("a", "int", true, Metadata.empty) .add("b", "long", false, Metadata.empty) .add("c", "string", true, Metadata.empty)
- Parameters:
name
- (undocumented)dataType
- (undocumented)nullable
- (undocumented)metadata
- (undocumented)- Returns:
- (undocumented)
-
add
Creates a newStructType
by adding a new field and specifying metadata where the dataType is specified as a String.val struct = (new StructType) .add("a", "int", true, "comment1") .add("b", "long", false, "comment2") .add("c", "string", true, "comment3")
- Parameters:
name
- (undocumented)dataType
- (undocumented)nullable
- (undocumented)comment
- (undocumented)- Returns:
- (undocumented)
-
apply
Extracts theStructField
with the given name.- Parameters:
name
- (undocumented)- Returns:
- (undocumented)
- Throws:
IllegalArgumentException
- if a field with the given name does not exist
-
apply
Returns aStructType
containingStructField
s of the given names, preserving the original order of fields.- Parameters:
names
- (undocumented)- Returns:
- (undocumented)
- Throws:
IllegalArgumentException
- if at least one given field name does not exist
-
fieldIndex
Returns the index of a given field.- Parameters:
name
- (undocumented)- Returns:
- (undocumented)
- Throws:
IllegalArgumentException
- if a field with the given name does not exist
-
treeString
-
treeString
-
printTreeString
public void printTreeString() -
apply
- Specified by:
apply
in interfacescala.collection.GenSeqLike<StructField,
scala.collection.GenSeq<StructField>> - Specified by:
apply
in interfacescala.collection.SeqLike<StructField,
scala.collection.Seq<StructField>>
-
length
public int length()- Specified by:
length
in interfacescala.collection.GenSeqLike<StructField,
scala.collection.GenSeq<StructField>> - Specified by:
length
in interfacescala.collection.SeqLike<StructField,
scala.collection.Seq<StructField>>
-
iterator
- Specified by:
iterator
in interfacescala.collection.GenIterableLike<StructField,
scala.collection.GenIterable<StructField>> - Specified by:
iterator
in interfacescala.collection.IterableLike<StructField,
scala.collection.Iterable<StructField>>
-
defaultSize
public int defaultSize()The default size of a value of the StructType is the total default sizes of all field types.- Specified by:
defaultSize
in classDataType
- Returns:
- (undocumented)
-
simpleString
Description copied from class:DataType
Readable string representation for the type.- Overrides:
simpleString
in classDataType
-
catalogString
Description copied from class:DataType
String representation for the type saved in external catalogs.- Overrides:
catalogString
in classDataType
-
sql
-
toDDL
Returns a string containing a schema in DDL format. For example, the following value:StructType(Seq(StructField("eventId", IntegerType), StructField("s", StringType)))
will be converted toeventId
INT,s
STRING. The returned DDL schema can be used in a table creation.- Returns:
- (undocumented)
- Since:
- 2.4.0
-