Packages

c

org.apache.spark.sql.vectorized

ArrowColumnVector

class ArrowColumnVector extends ColumnVector

Developer API

A column vector backed by Apache Arrow.

Annotations
@DeveloperApi()
Source
ArrowColumnVector.java
Linear Supertypes
ColumnVector, AutoCloseable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ArrowColumnVector
  2. ColumnVector
  3. AutoCloseable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new ArrowColumnVector(vector: ValueVector)

Value Members

  1. def close(): Unit

    Cleans up memory for this column vector.

    Cleans up memory for this column vector. The column vector is not usable after this.

    This overwrites AutoCloseable#close to remove the throws clause, as column vector is in-memory and we don't expect any exception to happen during closing.

    Definition Classes
    ArrowColumnVector → ColumnVector → AutoCloseable
    Annotations
    @Override()
  2. final def dataType(): DataType

    Returns the data type of this column vector.

    Returns the data type of this column vector.

    Definition Classes
    ColumnVector
  3. def getArray(rowId: Int): ColumnarArray

    Returns the array type value for rowId.

    Returns the array type value for rowId. If the slot for rowId is null, it should return null.

    To support array type, implementations must construct an ColumnarArray and return it in this method. ColumnarArray requires a ColumnVector that stores the data of all the elements of all the arrays in this vector, and an offset and length which points to a range in that ColumnVector, and the range represents the array for rowId. Implementations are free to decide where to put the data vector and offsets and lengths. For example, we can use the first child vector as the data vector, and store offsets and lengths in 2 int arrays in this vector.

    Definition Classes
    ArrowColumnVector → ColumnVector
    Annotations
    @Override()
  4. def getBinary(rowId: Int): Array[Byte]

    Returns the binary type value for rowId.

    Returns the binary type value for rowId. If the slot for rowId is null, it should return null.

    Definition Classes
    ArrowColumnVector → ColumnVector
    Annotations
    @Override()
  5. def getBoolean(rowId: Int): Boolean

    Returns the boolean type value for rowId.

    Returns the boolean type value for rowId. The return value is undefined and can be anything, if the slot for rowId is null.

    Definition Classes
    ArrowColumnVector → ColumnVector
    Annotations
    @Override()
  6. def getBooleans(rowId: Int, count: Int): Array[Boolean]

    Gets boolean type values from [rowId, rowId + count).

    Gets boolean type values from [rowId, rowId + count). The return values for the null slots are undefined and can be anything.

    Definition Classes
    ColumnVector
  7. def getByte(rowId: Int): Byte

    Returns the byte type value for rowId.

    Returns the byte type value for rowId. The return value is undefined and can be anything, if the slot for rowId is null.

    Definition Classes
    ArrowColumnVector → ColumnVector
    Annotations
    @Override()
  8. def getBytes(rowId: Int, count: Int): Array[Byte]

    Gets byte type values from [rowId, rowId + count).

    Gets byte type values from [rowId, rowId + count). The return values for the null slots are undefined and can be anything.

    Definition Classes
    ColumnVector
  9. def getChild(ordinal: Int): ArrowColumnVector

    returns

    child ColumnVector at the given ordinal.

    Definition Classes
    ArrowColumnVector → ColumnVector
    Annotations
    @Override()
  10. def getDecimal(rowId: Int, precision: Int, scale: Int): Decimal

    Returns the decimal type value for rowId.

    Returns the decimal type value for rowId. If the slot for rowId is null, it should return null.

    Definition Classes
    ArrowColumnVector → ColumnVector
    Annotations
    @Override()
  11. def getDouble(rowId: Int): Double

    Returns the double type value for rowId.

    Returns the double type value for rowId. The return value is undefined and can be anything, if the slot for rowId is null.

    Definition Classes
    ArrowColumnVector → ColumnVector
    Annotations
    @Override()
  12. def getDoubles(rowId: Int, count: Int): Array[Double]

    Gets double type values from [rowId, rowId + count).

    Gets double type values from [rowId, rowId + count). The return values for the null slots are undefined and can be anything.

    Definition Classes
    ColumnVector
  13. def getFloat(rowId: Int): Float

    Returns the float type value for rowId.

    Returns the float type value for rowId. The return value is undefined and can be anything, if the slot for rowId is null.

    Definition Classes
    ArrowColumnVector → ColumnVector
    Annotations
    @Override()
  14. def getFloats(rowId: Int, count: Int): Array[Float]

    Gets float type values from [rowId, rowId + count).

    Gets float type values from [rowId, rowId + count). The return values for the null slots are undefined and can be anything.

    Definition Classes
    ColumnVector
  15. def getInt(rowId: Int): Int

    Returns the int type value for rowId.

    Returns the int type value for rowId. The return value is undefined and can be anything, if the slot for rowId is null.

    Definition Classes
    ArrowColumnVector → ColumnVector
    Annotations
    @Override()
  16. def getInterval(rowId: Int): CalendarInterval

    Returns the calendar interval type value for rowId.

    Returns the calendar interval type value for rowId. If the slot for rowId is null, it should return null.

    In Spark, calendar interval type value is basically two integer values representing the number of months and days in this interval, and a long value representing the number of microseconds in this interval. An interval type vector is the same as a struct type vector with 3 fields: months, days and microseconds.

    To support interval type, implementations must implement #getChild(int) and define 3 child vectors: the first child vector is an int type vector, containing all the month values of all the interval values in this vector. The second child vector is an int type vector, containing all the day values of all the interval values in this vector. The third child vector is a long type vector, containing all the microsecond values of all the interval values in this vector. Note that the ArrowColumnVector leverages its built-in IntervalMonthDayNanoVector instead of above-mentioned protocol.

    Definition Classes
    ArrowColumnVector → ColumnVector
    Annotations
    @Override()
  17. def getInts(rowId: Int, count: Int): Array[Int]

    Gets int type values from [rowId, rowId + count).

    Gets int type values from [rowId, rowId + count). The return values for the null slots are undefined and can be anything.

    Definition Classes
    ColumnVector
  18. def getLong(rowId: Int): Long

    Returns the long type value for rowId.

    Returns the long type value for rowId. The return value is undefined and can be anything, if the slot for rowId is null.

    Definition Classes
    ArrowColumnVector → ColumnVector
    Annotations
    @Override()
  19. def getLongs(rowId: Int, count: Int): Array[Long]

    Gets long type values from [rowId, rowId + count).

    Gets long type values from [rowId, rowId + count). The return values for the null slots are undefined and can be anything.

    Definition Classes
    ColumnVector
  20. def getMap(rowId: Int): ColumnarMap

    Returns the map type value for rowId.

    Returns the map type value for rowId. If the slot for rowId is null, it should return null.

    In Spark, map type value is basically a key data array and a value data array. A key from the key array with a index and a value from the value array with the same index contribute to an entry of this map type value.

    To support map type, implementations must construct a ColumnarMap and return it in this method. ColumnarMap requires a ColumnVector that stores the data of all the keys of all the maps in this vector, and another ColumnVector that stores the data of all the values of all the maps in this vector, and a pair of offset and length which specify the range of the key/value array that belongs to the map type value at rowId.

    Definition Classes
    ArrowColumnVector → ColumnVector
    Annotations
    @Override()
  21. def getShort(rowId: Int): Short

    Returns the short type value for rowId.

    Returns the short type value for rowId. The return value is undefined and can be anything, if the slot for rowId is null.

    Definition Classes
    ArrowColumnVector → ColumnVector
    Annotations
    @Override()
  22. def getShorts(rowId: Int, count: Int): Array[Short]

    Gets short type values from [rowId, rowId + count).

    Gets short type values from [rowId, rowId + count). The return values for the null slots are undefined and can be anything.

    Definition Classes
    ColumnVector
  23. final def getStruct(rowId: Int): ColumnarRow

    Returns the struct type value for rowId.

    Returns the struct type value for rowId. If the slot for rowId is null, it should return null.

    To support struct type, implementations must implement #getChild(int) and make this vector a tree structure. The number of child vectors must be same as the number of fields of the struct type, and each child vector is responsible to store the data for its corresponding struct field.

    Definition Classes
    ColumnVector
  24. def getUTF8String(rowId: Int): UTF8String

    Returns the string type value for rowId.

    Returns the string type value for rowId. If the slot for rowId is null, it should return null.

    Note that the returned UTF8String may point to the data of this column vector, please copy it if you want to keep it after this column vector is freed.

    Definition Classes
    ArrowColumnVector → ColumnVector
    Annotations
    @Override()
  25. def getValueVector(): ValueVector
  26. final def getVariant(rowId: Int): VariantVal

    Returns the Variant value for rowId.

    Returns the Variant value for rowId. Similar to #getInterval(int), the implementation must implement #getChild(int) and define 2 child vectors of binary type for the Variant value and metadata.

    Definition Classes
    ColumnVector
  27. def hasNull(): Boolean

    Returns true if this column vector contains any null values.

    Returns true if this column vector contains any null values.

    Definition Classes
    ArrowColumnVector → ColumnVector
    Annotations
    @Override()
  28. def isNullAt(rowId: Int): Boolean

    Returns whether the value at rowId is NULL.

    Returns whether the value at rowId is NULL.

    Definition Classes
    ArrowColumnVector → ColumnVector
    Annotations
    @Override()
  29. def numNulls(): Int

    Returns the number of nulls in this column vector.

    Returns the number of nulls in this column vector.

    Definition Classes
    ArrowColumnVector → ColumnVector
    Annotations
    @Override()