org.apache.spark.sql

Class SQLImplicits

Object

org.apache.spark.sql.SQLImplicits

All Implemented Interfaces:

LowPrioritySQLImplicits

Direct Known Subclasses:

SparkSession.implicits$, SQLContext.implicits$

public abstract class SQLImplicits
extends Object
implements LowPrioritySQLImplicits

A collection of implicit methods for converting common Scala objects into Datasets.

Since:

1.6.0

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	SQLImplicits.StringToColumn Converts $"col name" into a Column.

Constructor Summary

Constructors

Constructor and Description
SQLImplicits()

Method Summary

Modifier and Type	Method and Description
<T> DatasetHolder<T>	localSeqToDatasetHolder(scala.collection.Seq<T> s, Encoder<T> evidence$8) Creates a Dataset from a local Seq.
Encoder<boolean[]>	newBooleanArrayEncoder()
Encoder<Object>	newBooleanEncoder()
Encoder<scala.collection.Seq<Object>>	newBooleanSeqEncoder() Deprecated. use newSequenceEncoder
Encoder<Boolean>	newBoxedBooleanEncoder()
Encoder<Byte>	newBoxedByteEncoder()
Encoder<Double>	newBoxedDoubleEncoder()
Encoder<Float>	newBoxedFloatEncoder()
Encoder<Integer>	newBoxedIntEncoder()
Encoder<Long>	newBoxedLongEncoder()
Encoder<Short>	newBoxedShortEncoder()
Encoder<byte[]>	newByteArrayEncoder()
Encoder<Object>	newByteEncoder()
Encoder<scala.collection.Seq<Object>>	newByteSeqEncoder() Deprecated. use newSequenceEncoder
Encoder<java.sql.Date>	newDateEncoder()
Encoder<double[]>	newDoubleArrayEncoder()
Encoder<Object>	newDoubleEncoder()
Encoder<scala.collection.Seq<Object>>	newDoubleSeqEncoder() Deprecated. use newSequenceEncoder
Encoder<java.time.Duration>	newDurationEncoder()
Encoder<float[]>	newFloatArrayEncoder()
Encoder<Object>	newFloatEncoder()
Encoder<scala.collection.Seq<Object>>	newFloatSeqEncoder() Deprecated. use newSequenceEncoder
Encoder<java.time.Instant>	newInstantEncoder()
Encoder<int[]>	newIntArrayEncoder()
Encoder<Object>	newIntEncoder()
Encoder<scala.collection.Seq<Object>>	newIntSeqEncoder() Deprecated. use newSequenceEncoder
Encoder<java.math.BigDecimal>	newJavaDecimalEncoder()
<A extends Enum<?>> Encoder<A>	newJavaEnumEncoder(scala.reflect.api.TypeTags.TypeTag<A> evidence$1)
Encoder<java.time.LocalDate>	newLocalDateEncoder()
Encoder<java.time.LocalDateTime>	newLocalDateTimeEncoder()
Encoder<long[]>	newLongArrayEncoder()
Encoder<Object>	newLongEncoder()
Encoder<scala.collection.Seq<Object>>	newLongSeqEncoder() Deprecated. use newSequenceEncoder
<T extends scala.collection.Map<?,?>> Encoder<T>	newMapEncoder(scala.reflect.api.TypeTags.TypeTag<T> evidence$4)
Encoder<java.time.Period>	newPeriodEncoder()
<A extends scala.Product> Encoder<Object>	newProductArrayEncoder(scala.reflect.api.TypeTags.TypeTag<A> evidence$6)
<A extends scala.Product> Encoder<scala.collection.Seq<A>>	newProductSeqEncoder(scala.reflect.api.TypeTags.TypeTag<A> evidence$2) Deprecated. use newSequenceEncoder
Encoder<scala.math.BigDecimal>	newScalaDecimalEncoder()
<T extends scala.collection.Seq<?>> Encoder<T>	newSequenceEncoder(scala.reflect.api.TypeTags.TypeTag<T> evidence$3)
<T extends scala.collection.immutable.Set<?>> Encoder<T>	newSetEncoder(scala.reflect.api.TypeTags.TypeTag<T> evidence$5) Notice that we serialize Set to Catalyst array.
Encoder<short[]>	newShortArrayEncoder()
Encoder<Object>	newShortEncoder()
Encoder<scala.collection.Seq<Object>>	newShortSeqEncoder() Deprecated. use newSequenceEncoder
Encoder<String[]>	newStringArrayEncoder()
Encoder<String>	newStringEncoder()
Encoder<scala.collection.Seq<String>>	newStringSeqEncoder() Deprecated. use newSequenceEncoder
Encoder<java.sql.Timestamp>	newTimeStampEncoder()
<T> DatasetHolder<T>	rddToDatasetHolder(RDD<T> rdd, Encoder<T> evidence$7) Creates a Dataset from an RDD.
ColumnName	symbolToColumn(scala.Symbol s) An implicit conversion that turns a Scala Symbol into a Column.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.sql.LowPrioritySQLImplicits

newProductEncoder

Constructor Detail

SQLImplicits

public SQLImplicits()

Method Detail

localSeqToDatasetHolder

public <T> DatasetHolder<T> localSeqToDatasetHolder(scala.collection.Seq<T> s,
                                                    Encoder<T> evidence$8)

Creates a Dataset from a local Seq.

Parameters:

s - (undocumented)

evidence$8 - (undocumented)

Returns:

(undocumented)

Since:

1.6.0

newBooleanArrayEncoder

public Encoder<boolean[]> newBooleanArrayEncoder()

Since:

1.6.1

newBooleanEncoder

public Encoder<Object> newBooleanEncoder()

Since:

1.6.0

newBooleanSeqEncoder

public Encoder<scala.collection.Seq<Object>> newBooleanSeqEncoder()

Deprecated. use newSequenceEncoder

Returns:

(undocumented)

Since:

1.6.1

newBoxedBooleanEncoder

public Encoder<Boolean> newBoxedBooleanEncoder()

Since:

2.0.0

newBoxedByteEncoder

public Encoder<Byte> newBoxedByteEncoder()

Since:

2.0.0

newBoxedDoubleEncoder

public Encoder<Double> newBoxedDoubleEncoder()

Since:

2.0.0

newBoxedFloatEncoder

public Encoder<Float> newBoxedFloatEncoder()

Since:

2.0.0

newBoxedIntEncoder

public Encoder<Integer> newBoxedIntEncoder()

Since:

2.0.0

newBoxedLongEncoder

public Encoder<Long> newBoxedLongEncoder()

Since:

2.0.0

newBoxedShortEncoder

public Encoder<Short> newBoxedShortEncoder()

Since:

2.0.0

newByteArrayEncoder

public Encoder<byte[]> newByteArrayEncoder()

Since:

1.6.1

newByteEncoder

public Encoder<Object> newByteEncoder()

Since:

1.6.0

newByteSeqEncoder

public Encoder<scala.collection.Seq<Object>> newByteSeqEncoder()

Deprecated. use newSequenceEncoder

Returns:

(undocumented)

Since:

1.6.1

newDateEncoder

public Encoder<java.sql.Date> newDateEncoder()

Since:

2.2.0

newDoubleArrayEncoder

public Encoder<double[]> newDoubleArrayEncoder()

Since:

1.6.1

newDoubleEncoder

public Encoder<Object> newDoubleEncoder()

Since:

1.6.0

newDoubleSeqEncoder

public Encoder<scala.collection.Seq<Object>> newDoubleSeqEncoder()

Deprecated. use newSequenceEncoder

Returns:

(undocumented)

Since:

1.6.1

newDurationEncoder

public Encoder<java.time.Duration> newDurationEncoder()

Since:

3.2.0

newFloatArrayEncoder

public Encoder<float[]> newFloatArrayEncoder()

Since:

1.6.1

newFloatEncoder

public Encoder<Object> newFloatEncoder()

Since:

1.6.0

newFloatSeqEncoder

public Encoder<scala.collection.Seq<Object>> newFloatSeqEncoder()

Deprecated. use newSequenceEncoder

Returns:

(undocumented)

Since:

1.6.1

newInstantEncoder

public Encoder<java.time.Instant> newInstantEncoder()

Since:

3.0.0

newIntArrayEncoder

public Encoder<int[]> newIntArrayEncoder()

Since:

1.6.1

newIntEncoder

public Encoder<Object> newIntEncoder()

Since:

1.6.0

newIntSeqEncoder

public Encoder<scala.collection.Seq<Object>> newIntSeqEncoder()

Deprecated. use newSequenceEncoder

Returns:

(undocumented)

Since:

1.6.1

newJavaDecimalEncoder

public Encoder<java.math.BigDecimal> newJavaDecimalEncoder()

Since:

2.2.0

newJavaEnumEncoder

public <A extends Enum<?>> Encoder<A> newJavaEnumEncoder(scala.reflect.api.TypeTags.TypeTag<A> evidence$1)

Since:

3.2.0

newLocalDateEncoder

public Encoder<java.time.LocalDate> newLocalDateEncoder()

Since:

3.0.0

newLocalDateTimeEncoder

public Encoder<java.time.LocalDateTime> newLocalDateTimeEncoder()

Since:

3.4.0

newLongArrayEncoder

public Encoder<long[]> newLongArrayEncoder()

Since:

1.6.1

newLongEncoder

public Encoder<Object> newLongEncoder()

Since:

1.6.0

newLongSeqEncoder

public Encoder<scala.collection.Seq<Object>> newLongSeqEncoder()

Deprecated. use newSequenceEncoder

Returns:

(undocumented)

Since:

1.6.1

newMapEncoder

public <T extends scala.collection.Map<?,?>> Encoder<T> newMapEncoder(scala.reflect.api.TypeTags.TypeTag<T> evidence$4)

Since:

2.3.0

newPeriodEncoder

public Encoder<java.time.Period> newPeriodEncoder()

Since:

3.2.0

newProductArrayEncoder

public <A extends scala.Product> Encoder<Object> newProductArrayEncoder(scala.reflect.api.TypeTags.TypeTag<A> evidence$6)

Since:

1.6.1

newProductSeqEncoder

public <A extends scala.Product> Encoder<scala.collection.Seq<A>> newProductSeqEncoder(scala.reflect.api.TypeTags.TypeTag<A> evidence$2)

Deprecated. use newSequenceEncoder

Parameters:

evidence$2 - (undocumented)

Returns:

(undocumented)

Since:

1.6.1

newScalaDecimalEncoder

public Encoder<scala.math.BigDecimal> newScalaDecimalEncoder()

Since:

2.2.0

newSequenceEncoder

public <T extends scala.collection.Seq<?>> Encoder<T> newSequenceEncoder(scala.reflect.api.TypeTags.TypeTag<T> evidence$3)

Since:

2.2.0

newSetEncoder

public <T extends scala.collection.immutable.Set<?>> Encoder<T> newSetEncoder(scala.reflect.api.TypeTags.TypeTag<T> evidence$5)

Notice that we serialize Set to Catalyst array. The set property is only kept when manipulating the domain objects. The serialization format doesn't keep the set property. When we have a Catalyst array which contains duplicated elements and convert it to Dataset[Set[T} by using the encoder, the elements will be de-duplicated.

Parameters:

evidence$5 - (undocumented)

Returns:

(undocumented)

Since:

2.3.0

newShortArrayEncoder

public Encoder<short[]> newShortArrayEncoder()

Since:

1.6.1

newShortEncoder

public Encoder<Object> newShortEncoder()

Since:

1.6.0

newShortSeqEncoder

public Encoder<scala.collection.Seq<Object>> newShortSeqEncoder()

Deprecated. use newSequenceEncoder

Returns:

(undocumented)

Since:

1.6.1

newStringArrayEncoder

public Encoder<String[]> newStringArrayEncoder()

Since:

1.6.1

newStringEncoder

public Encoder<String> newStringEncoder()

Since:

1.6.0

newStringSeqEncoder

public Encoder<scala.collection.Seq<String>> newStringSeqEncoder()

Deprecated. use newSequenceEncoder

Returns:

(undocumented)

Since:

1.6.1

newTimeStampEncoder

public Encoder<java.sql.Timestamp> newTimeStampEncoder()

Since:

2.2.0

rddToDatasetHolder

public <T> DatasetHolder<T> rddToDatasetHolder(RDD<T> rdd,
                                               Encoder<T> evidence$7)

Creates a Dataset from an RDD.

Parameters:

rdd - (undocumented)

evidence$7 - (undocumented)

Returns:

(undocumented)

Since:

1.6.0

symbolToColumn

public ColumnName symbolToColumn(scala.Symbol s)

An implicit conversion that turns a Scala Symbol into a Column.

Parameters:

s - (undocumented)

Returns:

(undocumented)

Since:

1.3.0