Class StructType

Object
org.apache.spark.sql.types.DataType
org.apache.spark.sql.types.StructType
All Implemented Interfaces:
Serializable, scala.collection.generic.FilterMonadic<StructField,scala.collection.Traversable<StructField>>, scala.collection.generic.GenericTraversableTemplate<StructField,scala.collection.GenTraversable>, scala.collection.generic.HasNewBuilder<StructField,scala.collection.Traversable<StructField>>, scala.collection.GenIterable<StructField>, scala.collection.GenIterableLike<StructField,scala.collection.GenIterable<StructField>>, scala.collection.GenSeq<StructField>, scala.collection.GenSeqLike<StructField,scala.collection.GenSeq<StructField>>, scala.collection.GenTraversable<StructField>, scala.collection.GenTraversableLike<StructField,scala.collection.Traversable<StructField>>, scala.collection.GenTraversableOnce<StructField>, scala.collection.Iterable<StructField>, scala.collection.IterableLike<StructField,scala.collection.Iterable<StructField>>, scala.collection.Parallelizable<StructField,scala.collection.parallel.ParIterable<StructField>>, scala.collection.Seq<StructField>, scala.collection.SeqLike<StructField,scala.collection.Seq<StructField>>, scala.collection.Traversable<StructField>, scala.collection.TraversableLike<StructField,scala.collection.Traversable<StructField>>, scala.collection.TraversableOnce<StructField>, scala.Equals, scala.Function1<Object,StructField>, scala.PartialFunction<Object,StructField>, scala.Product, scala.Serializable

public class StructType extends DataType implements scala.collection.Seq<StructField>, scala.Product, scala.Serializable
A StructType object can be constructed by

 StructType(fields: Seq[StructField])
 
For a StructType object, one or multiple StructFields can be extracted by names. If multiple StructFields are extracted, a StructType object will be returned. If a provided name does not have a matching field, it will be ignored. For the case of extracting a single StructField, a null will be returned.

Scala Example:


 import org.apache.spark.sql._
 import org.apache.spark.sql.types._

 val struct =
   StructType(
     StructField("a", IntegerType, true) ::
     StructField("b", LongType, false) ::
     StructField("c", BooleanType, false) :: Nil)

 // Extract a single StructField.
 val singleField = struct("b")
 // singleField: StructField = StructField(b,LongType,false)

 // If this struct does not have a field called "d", it throws an exception.
 struct("d")
 // java.lang.IllegalArgumentException: d does not exist.
 //   ...

 // Extract multiple StructFields. Field names are provided in a set.
 // A StructType object will be returned.
 val twoFields = struct(Set("b", "c"))
 // twoFields: StructType =
 //   StructType(StructField(b,LongType,false), StructField(c,BooleanType,false))

 // Any names without matching fields will throw an exception.
 // For the case shown below, an exception is thrown due to "d".
 struct(Set("b", "c", "d"))
 // java.lang.IllegalArgumentException: d does not exist.
 //    ...
 

A Row object is used as a value of the StructType.

Scala Example:


 import org.apache.spark.sql._
 import org.apache.spark.sql.types._

 val innerStruct =
   StructType(
     StructField("f1", IntegerType, true) ::
     StructField("f2", LongType, false) ::
     StructField("f3", BooleanType, false) :: Nil)

 val struct = StructType(
   StructField("a", innerStruct, true) :: Nil)

 // Create a Row with the schema defined by struct
 val row = Row(Row(1, 2, true))
 

Since:
1.3.0
See Also:
  • Nested Class Summary

    Nested classes/interfaces inherited from interface scala.collection.TraversableLike

    scala.collection.TraversableLike.WithFilter

    Nested classes/interfaces inherited from interface scala.collection.TraversableOnce

    scala.collection.TraversableOnce.BufferedCanBuildFrom<A extends Object,CC extends scala.collection.TraversableOnce<Object>>, scala.collection.TraversableOnce.FlattenOps<A extends Object>, scala.collection.TraversableOnce.ForceImplicitAmbiguity, scala.collection.TraversableOnce.MonadOps<A extends Object>, scala.collection.TraversableOnce.OnceCanBuildFrom<A extends Object>
  • Constructor Summary

    Constructors
    Constructor
    Description
    No-arg constructor for kryo.
     
  • Method Summary

    Modifier and Type
    Method
    Description
    add(String name, String dataType)
    Creates a new StructType by adding a new nullable field with no metadata where the dataType is specified as a String.
    add(String name, String dataType, boolean nullable)
    Creates a new StructType by adding a new field with no metadata where the dataType is specified as a String.
    add(String name, String dataType, boolean nullable, String comment)
    Creates a new StructType by adding a new field and specifying metadata where the dataType is specified as a String.
    add(String name, String dataType, boolean nullable, Metadata metadata)
    Creates a new StructType by adding a new field and specifying metadata where the dataType is specified as a String.
    add(String name, DataType dataType)
    Creates a new StructType by adding a new nullable field with no metadata.
    add(String name, DataType dataType, boolean nullable)
    Creates a new StructType by adding a new field with no metadata.
    add(String name, DataType dataType, boolean nullable, String comment)
    Creates a new StructType by adding a new field and specifying metadata.
    add(String name, DataType dataType, boolean nullable, Metadata metadata)
    Creates a new StructType by adding a new field and specifying metadata.
    Creates a new StructType by adding a new field.
    apply(int fieldIndex)
     
    apply(String name)
    Extracts the StructField with the given name.
    apply(scala.collection.immutable.Set<String> names)
    Returns a StructType containing StructFields of the given names, preserving the original order of fields.
    String representation for the type saved in external catalogs.
    int
    The default size of a value of the StructType is the total default sizes of all field types.
    boolean
    equals(Object that)
     
    int
    Returns the index of a given field.
    Returns all field names in an array.
     
    static scala.Option<StructType>
    findMissingFields(StructType source, StructType target, scala.Function2<String,String,Object> resolver)
    Returns a StructType that contains missing fields recursively from source to target.
    static StructType
    Creates StructType for a given DDL-formatted string, which is a comma separated list of field definitions, e.g., a INT, b STRING.
    int
     
    scala.collection.Iterator<StructField>
     
    int
     
    Returns all field names in an array.
    void
     
    Readable string representation for the type.
    sql()
     
    Returns a string containing a schema in DDL format.
     
     
    treeString(int maxDepth)
     

    Methods inherited from class java.lang.Object

    getClass, notify, notifyAll, wait, wait, wait

    Methods inherited from interface scala.Function1

    apply, apply$mcDD$sp, apply$mcDF$sp, apply$mcDI$sp, apply$mcDJ$sp, apply$mcFD$sp, apply$mcFF$sp, apply$mcFI$sp, apply$mcFJ$sp, apply$mcID$sp, apply$mcIF$sp, apply$mcII$sp, apply$mcIJ$sp, apply$mcJD$sp, apply$mcJF$sp, apply$mcJI$sp, apply$mcJJ$sp, apply$mcVD$sp, apply$mcVF$sp, apply$mcVI$sp, apply$mcVJ$sp, apply$mcZD$sp, apply$mcZF$sp, apply$mcZI$sp, apply$mcZJ$sp, compose

    Methods inherited from interface scala.collection.generic.GenericTraversableTemplate

    flatten, foreach, genericBuilder, head, isEmpty, newBuilder, transpose, unzip, unzip3

    Methods inherited from interface scala.collection.GenSeqLike

    indexOf, indexOf, indexWhere, isDefinedAt, lastIndexOf, lastIndexOf, lastIndexWhere, prefixLength, startsWith, toSeq

    Methods inherited from interface scala.collection.GenTraversableLike

    flatMap, foreach, map, size

    Methods inherited from interface scala.collection.GenTraversableOnce

    sizeHintIfCheap

    Methods inherited from interface scala.collection.IterableLike

    canEqual, copyToArray, drop, dropRight, exists, find, foldRight, forall, foreach, grouped, head, reduceRight, sameElements, slice, sliding, sliding, take, takeRight, takeWhile, toIterable, toIterator, toStream, zip, zipAll, zipWithIndex

    Methods inherited from interface scala.collection.Parallelizable

    par

    Methods inherited from interface scala.PartialFunction

    andThen, applyOrElse, isDefinedAt, lift, orElse, runWith

    Methods inherited from interface scala.Product

    productArity, productElement, productIterator, productPrefix

    Methods inherited from interface scala.collection.Seq

    companion, seq

    Methods inherited from interface scala.collection.SeqLike

    $colon$plus, $plus$colon, combinations, contains, containsSlice, corresponds, diff, distinct, endsWith, indexOfSlice, indexOfSlice, indexWhere, indices, intersect, isEmpty, lastIndexOfSlice, lastIndexOfSlice, lastIndexWhere, lengthCompare, padTo, parCombiner, patch, permutations, reverse, reverseIterator, reverseMap, segmentLength, size, sortBy, sorted, sortWith, startsWith, thisCollection, toCollection, toSeq, union, updated, view, view

    Methods inherited from interface scala.collection.TraversableLike

    $plus$plus, $plus$plus$colon, $plus$plus$colon, collect, dropWhile, filter, filterImpl, filterNot, flatMap, groupBy, hasDefiniteSize, headOption, init, inits, isTraversableAgain, last, lastOption, map, newBuilder, partition, repr, scan, scanLeft, scanRight, sliceWithKnownBound, sliceWithKnownDelta, span, splitAt, stringPrefix, tail, tails, to, toTraversable, withFilter

    Methods inherited from interface scala.collection.TraversableOnce

    $colon$bslash, $div$colon, addString, addString, addString, aggregate, collectFirst, copyToArray, copyToArray, copyToBuffer, count, fold, foldLeft, max, maxBy, min, minBy, mkString, mkString, mkString, nonEmpty, product, reduce, reduceLeft, reduceLeftOption, reduceOption, reduceRightOption, reversed, sum, toArray, toBuffer, toIndexedSeq, toList, toMap, toSet, toVector
  • Constructor Details

    • StructType

      public StructType(StructField[] fields)
    • StructType

      public StructType()
      No-arg constructor for kryo.
  • Method Details

    • fromDDL

      public static StructType fromDDL(String ddl)
      Creates StructType for a given DDL-formatted string, which is a comma separated list of field definitions, e.g., a INT, b STRING.

      Parameters:
      ddl - (undocumented)
      Returns:
      (undocumented)
      Since:
      2.2.0
    • findMissingFields

      public static scala.Option<StructType> findMissingFields(StructType source, StructType target, scala.Function2<String,String,Object> resolver)
      Returns a StructType that contains missing fields recursively from source to target. Note that this doesn't support looking into array type and map type recursively.
      Parameters:
      source - (undocumented)
      target - (undocumented)
      resolver - (undocumented)
      Returns:
      (undocumented)
    • fields

      public StructField[] fields()
    • fieldNames

      public String[] fieldNames()
      Returns all field names in an array.
    • names

      public String[] names()
      Returns all field names in an array. This is an alias of fieldNames.

      Returns:
      (undocumented)
      Since:
      2.4.0
    • equals

      public boolean equals(Object that)
      Specified by:
      equals in interface scala.Equals
      Specified by:
      equals in interface scala.collection.GenSeqLike<StructField,scala.collection.GenSeq<StructField>>
      Overrides:
      equals in class Object
    • toString

      public String toString()
      Specified by:
      toString in interface scala.Function1<Object,StructField>
      Specified by:
      toString in interface scala.collection.SeqLike<StructField,scala.collection.Seq<StructField>>
      Specified by:
      toString in interface scala.collection.TraversableLike<StructField,scala.collection.Traversable<StructField>>
      Overrides:
      toString in class Object
    • hashCode

      public int hashCode()
      Specified by:
      hashCode in interface scala.collection.GenSeqLike<StructField,scala.collection.GenSeq<StructField>>
      Overrides:
      hashCode in class Object
    • add

      public StructType add(StructField field)
      Creates a new StructType by adding a new field.
      
       val struct = (new StructType)
         .add(StructField("a", IntegerType, true))
         .add(StructField("b", LongType, false))
         .add(StructField("c", StringType, true))
      
      Parameters:
      field - (undocumented)
      Returns:
      (undocumented)
    • add

      public StructType add(String name, DataType dataType)
      Creates a new StructType by adding a new nullable field with no metadata.

      val struct = (new StructType) .add("a", IntegerType) .add("b", LongType) .add("c", StringType)

      Parameters:
      name - (undocumented)
      dataType - (undocumented)
      Returns:
      (undocumented)
    • add

      public StructType add(String name, DataType dataType, boolean nullable)
      Creates a new StructType by adding a new field with no metadata.

      val struct = (new StructType) .add("a", IntegerType, true) .add("b", LongType, false) .add("c", StringType, true)

      Parameters:
      name - (undocumented)
      dataType - (undocumented)
      nullable - (undocumented)
      Returns:
      (undocumented)
    • add

      public StructType add(String name, DataType dataType, boolean nullable, Metadata metadata)
      Creates a new StructType by adding a new field and specifying metadata.
      
       val struct = (new StructType)
         .add("a", IntegerType, true, Metadata.empty)
         .add("b", LongType, false, Metadata.empty)
         .add("c", StringType, true, Metadata.empty)
       
      Parameters:
      name - (undocumented)
      dataType - (undocumented)
      nullable - (undocumented)
      metadata - (undocumented)
      Returns:
      (undocumented)
    • add

      public StructType add(String name, DataType dataType, boolean nullable, String comment)
      Creates a new StructType by adding a new field and specifying metadata.
      
       val struct = (new StructType)
         .add("a", IntegerType, true, "comment1")
         .add("b", LongType, false, "comment2")
         .add("c", StringType, true, "comment3")
       
      Parameters:
      name - (undocumented)
      dataType - (undocumented)
      nullable - (undocumented)
      comment - (undocumented)
      Returns:
      (undocumented)
    • add

      public StructType add(String name, String dataType)
      Creates a new StructType by adding a new nullable field with no metadata where the dataType is specified as a String.

      
       val struct = (new StructType)
         .add("a", "int")
         .add("b", "long")
         .add("c", "string")
       
      Parameters:
      name - (undocumented)
      dataType - (undocumented)
      Returns:
      (undocumented)
    • add

      public StructType add(String name, String dataType, boolean nullable)
      Creates a new StructType by adding a new field with no metadata where the dataType is specified as a String.

      
       val struct = (new StructType)
         .add("a", "int", true)
         .add("b", "long", false)
         .add("c", "string", true)
       
      Parameters:
      name - (undocumented)
      dataType - (undocumented)
      nullable - (undocumented)
      Returns:
      (undocumented)
    • add

      public StructType add(String name, String dataType, boolean nullable, Metadata metadata)
      Creates a new StructType by adding a new field and specifying metadata where the dataType is specified as a String.
      
       val struct = (new StructType)
         .add("a", "int", true, Metadata.empty)
         .add("b", "long", false, Metadata.empty)
         .add("c", "string", true, Metadata.empty)
       
      Parameters:
      name - (undocumented)
      dataType - (undocumented)
      nullable - (undocumented)
      metadata - (undocumented)
      Returns:
      (undocumented)
    • add

      public StructType add(String name, String dataType, boolean nullable, String comment)
      Creates a new StructType by adding a new field and specifying metadata where the dataType is specified as a String.
      
       val struct = (new StructType)
         .add("a", "int", true, "comment1")
         .add("b", "long", false, "comment2")
         .add("c", "string", true, "comment3")
       
      Parameters:
      name - (undocumented)
      dataType - (undocumented)
      nullable - (undocumented)
      comment - (undocumented)
      Returns:
      (undocumented)
    • apply

      public StructField apply(String name)
      Extracts the StructField with the given name.

      Parameters:
      name - (undocumented)
      Returns:
      (undocumented)
      Throws:
      IllegalArgumentException - if a field with the given name does not exist
    • apply

      public StructType apply(scala.collection.immutable.Set<String> names)
      Returns a StructType containing StructFields of the given names, preserving the original order of fields.

      Parameters:
      names - (undocumented)
      Returns:
      (undocumented)
      Throws:
      IllegalArgumentException - if at least one given field name does not exist
    • fieldIndex

      public int fieldIndex(String name)
      Returns the index of a given field.

      Parameters:
      name - (undocumented)
      Returns:
      (undocumented)
      Throws:
      IllegalArgumentException - if a field with the given name does not exist
    • treeString

      public String treeString()
    • treeString

      public String treeString(int maxDepth)
    • printTreeString

      public void printTreeString()
    • apply

      public StructField apply(int fieldIndex)
      Specified by:
      apply in interface scala.collection.GenSeqLike<StructField,scala.collection.GenSeq<StructField>>
      Specified by:
      apply in interface scala.collection.SeqLike<StructField,scala.collection.Seq<StructField>>
    • length

      public int length()
      Specified by:
      length in interface scala.collection.GenSeqLike<StructField,scala.collection.GenSeq<StructField>>
      Specified by:
      length in interface scala.collection.SeqLike<StructField,scala.collection.Seq<StructField>>
    • iterator

      public scala.collection.Iterator<StructField> iterator()
      Specified by:
      iterator in interface scala.collection.GenIterableLike<StructField,scala.collection.GenIterable<StructField>>
      Specified by:
      iterator in interface scala.collection.IterableLike<StructField,scala.collection.Iterable<StructField>>
    • defaultSize

      public int defaultSize()
      The default size of a value of the StructType is the total default sizes of all field types.
      Specified by:
      defaultSize in class DataType
      Returns:
      (undocumented)
    • simpleString

      public String simpleString()
      Description copied from class: DataType
      Readable string representation for the type.
      Overrides:
      simpleString in class DataType
    • catalogString

      public String catalogString()
      Description copied from class: DataType
      String representation for the type saved in external catalogs.
      Overrides:
      catalogString in class DataType
    • sql

      public String sql()
      Overrides:
      sql in class DataType
    • toDDL

      public String toDDL()
      Returns a string containing a schema in DDL format. For example, the following value: StructType(Seq(StructField("eventId", IntegerType), StructField("s", StringType))) will be converted to eventId INT, s STRING. The returned DDL schema can be used in a table creation.

      Returns:
      (undocumented)
      Since:
      2.4.0