Packages

abstract class ColumnVector extends AutoCloseable

An interface representing in-memory columnar data in Spark. This interface defines the main APIs to access the data, as well as their batched versions. The batched versions are considered to be faster and preferable whenever possible.

Most of the APIs take the rowId as a parameter. This is the batch local 0-based row id for values in this ColumnVector.

Spark only calls specific get method according to the data type of this ColumnVector, e.g. if it's int type, Spark is guaranteed to only call #getInt(int) or int).

ColumnVector supports all the data types including nested types. To handle nested types, ColumnVector can have children and is a tree structure. Please refer to #getStruct(int), #getArray(int) and #getMap(int) for the details about how to implement nested types.

ColumnVector is expected to be reused during the entire data loading process, to avoid allocating memory again and again.

ColumnVector is meant to maximize CPU efficiency but not to minimize storage footprint. Implementations should prefer computing efficiency over storage efficiency when design the format. Since it is expected to reuse the ColumnVector instance while loading data, the storage footprint is negligible.

Annotations
@Evolving()
Source
ColumnVector.java
Linear Supertypes
AutoCloseable, AnyRef, Any
Known Subclasses
ArrowColumnVector
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ColumnVector
  2. AutoCloseable
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new ColumnVector(type: DataType)
    Attributes
    protected[vectorized]

Abstract Value Members

  1. abstract def close(): Unit

    Cleans up memory for this column vector.

    Cleans up memory for this column vector. The column vector is not usable after this.

    This overwrites AutoCloseable#close to remove the throws clause, as column vector is in-memory and we don't expect any exception to happen during closing.

    Definition Classes
    ColumnVector → AutoCloseable
    Annotations
    @Override()
  2. abstract def getArray(rowId: Int): ColumnarArray

    Returns the array type value for rowId.

    Returns the array type value for rowId. If the slot for rowId is null, it should return null.

    To support array type, implementations must construct an ColumnarArray and return it in this method. ColumnarArray requires a ColumnVector that stores the data of all the elements of all the arrays in this vector, and an offset and length which points to a range in that ColumnVector, and the range represents the array for rowId. Implementations are free to decide where to put the data vector and offsets and lengths. For example, we can use the first child vector as the data vector, and store offsets and lengths in 2 int arrays in this vector.

  3. abstract def getBinary(rowId: Int): Array[Byte]

    Returns the binary type value for rowId.

    Returns the binary type value for rowId. If the slot for rowId is null, it should return null.

  4. abstract def getBoolean(rowId: Int): Boolean

    Returns the boolean type value for rowId.

    Returns the boolean type value for rowId. The return value is undefined and can be anything, if the slot for rowId is null.

  5. abstract def getByte(rowId: Int): Byte

    Returns the byte type value for rowId.

    Returns the byte type value for rowId. The return value is undefined and can be anything, if the slot for rowId is null.

  6. abstract def getChild(ordinal: Int): ColumnVector

    returns

    child ColumnVector at the given ordinal.

  7. abstract def getDecimal(rowId: Int, precision: Int, scale: Int): Decimal

    Returns the decimal type value for rowId.

    Returns the decimal type value for rowId. If the slot for rowId is null, it should return null.

  8. abstract def getDouble(rowId: Int): Double

    Returns the double type value for rowId.

    Returns the double type value for rowId. The return value is undefined and can be anything, if the slot for rowId is null.

  9. abstract def getFloat(rowId: Int): Float

    Returns the float type value for rowId.

    Returns the float type value for rowId. The return value is undefined and can be anything, if the slot for rowId is null.

  10. abstract def getInt(rowId: Int): Int

    Returns the int type value for rowId.

    Returns the int type value for rowId. The return value is undefined and can be anything, if the slot for rowId is null.

  11. abstract def getLong(rowId: Int): Long

    Returns the long type value for rowId.

    Returns the long type value for rowId. The return value is undefined and can be anything, if the slot for rowId is null.

  12. abstract def getMap(ordinal: Int): ColumnarMap

    Returns the map type value for rowId.

    Returns the map type value for rowId. If the slot for rowId is null, it should return null.

    In Spark, map type value is basically a key data array and a value data array. A key from the key array with a index and a value from the value array with the same index contribute to an entry of this map type value.

    To support map type, implementations must construct a ColumnarMap and return it in this method. ColumnarMap requires a ColumnVector that stores the data of all the keys of all the maps in this vector, and another ColumnVector that stores the data of all the values of all the maps in this vector, and a pair of offset and length which specify the range of the key/value array that belongs to the map type value at rowId.

  13. abstract def getShort(rowId: Int): Short

    Returns the short type value for rowId.

    Returns the short type value for rowId. The return value is undefined and can be anything, if the slot for rowId is null.

  14. abstract def getUTF8String(rowId: Int): UTF8String

    Returns the string type value for rowId.

    Returns the string type value for rowId. If the slot for rowId is null, it should return null.

    Note that the returned UTF8String may point to the data of this column vector, please copy it if you want to keep it after this column vector is freed.

  15. abstract def hasNull(): Boolean

    Returns true if this column vector contains any null values.

  16. abstract def isNullAt(rowId: Int): Boolean

    Returns whether the value at rowId is NULL.

  17. abstract def numNulls(): Int

    Returns the number of nulls in this column vector.

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  6. def closeIfFreeable(): Unit

    Cleans up memory for this column vector if it's resources are freeable between batches.

    Cleans up memory for this column vector if it's resources are freeable between batches. The column vector is not usable after this.

    If this is a writable column vector or constant column vector, it is a no-op.

  7. final def dataType(): DataType

    Returns the data type of this column vector.

  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  10. def getBooleans(rowId: Int, count: Int): Array[Boolean]

    Gets boolean type values from [rowId, rowId + count).

    Gets boolean type values from [rowId, rowId + count). The return values for the null slots are undefined and can be anything.

  11. def getBytes(rowId: Int, count: Int): Array[Byte]

    Gets byte type values from [rowId, rowId + count).

    Gets byte type values from [rowId, rowId + count). The return values for the null slots are undefined and can be anything.

  12. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  13. def getDoubles(rowId: Int, count: Int): Array[Double]

    Gets double type values from [rowId, rowId + count).

    Gets double type values from [rowId, rowId + count). The return values for the null slots are undefined and can be anything.

  14. def getFloats(rowId: Int, count: Int): Array[Float]

    Gets float type values from [rowId, rowId + count).

    Gets float type values from [rowId, rowId + count). The return values for the null slots are undefined and can be anything.

  15. def getInterval(rowId: Int): CalendarInterval

    Returns the calendar interval type value for rowId.

    Returns the calendar interval type value for rowId. If the slot for rowId is null, it should return null.

    In Spark, calendar interval type value is basically two integer values representing the number of months and days in this interval, and a long value representing the number of microseconds in this interval. An interval type vector is the same as a struct type vector with 3 fields: months, days and microseconds.

    To support interval type, implementations must implement #getChild(int) and define 3 child vectors: the first child vector is an int type vector, containing all the month values of all the interval values in this vector. The second child vector is an int type vector, containing all the day values of all the interval values in this vector. The third child vector is a long type vector, containing all the microsecond values of all the interval values in this vector. Note that the ArrowColumnVector leverages its built-in IntervalMonthDayNanoVector instead of above-mentioned protocol.

  16. def getInts(rowId: Int, count: Int): Array[Int]

    Gets int type values from [rowId, rowId + count).

    Gets int type values from [rowId, rowId + count). The return values for the null slots are undefined and can be anything.

  17. def getLongs(rowId: Int, count: Int): Array[Long]

    Gets long type values from [rowId, rowId + count).

    Gets long type values from [rowId, rowId + count). The return values for the null slots are undefined and can be anything.

  18. def getShorts(rowId: Int, count: Int): Array[Short]

    Gets short type values from [rowId, rowId + count).

    Gets short type values from [rowId, rowId + count). The return values for the null slots are undefined and can be anything.

  19. final def getStruct(rowId: Int): ColumnarRow

    Returns the struct type value for rowId.

    Returns the struct type value for rowId. If the slot for rowId is null, it should return null.

    To support struct type, implementations must implement #getChild(int) and make this vector a tree structure. The number of child vectors must be same as the number of fields of the struct type, and each child vector is responsible to store the data for its corresponding struct field.

  20. final def getVariant(rowId: Int): VariantVal

    Returns the Variant value for rowId.

    Returns the Variant value for rowId. Similar to #getInterval(int), the implementation must implement #getChild(int) and define 2 child vectors of binary type for the Variant value and metadata.

  21. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  22. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  23. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  24. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  25. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  26. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  27. def toString(): String
    Definition Classes
    AnyRef → Any
  28. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  29. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  30. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from AutoCloseable

Inherited from AnyRef

Inherited from Any

Ungrouped