Packages

class ColumnarBatch extends AutoCloseable

Developer API

This class wraps multiple ColumnVectors as a row-wise table. It provides a row view of this batch so that Spark can access the data row by row. Instance of it is meant to be reused during the entire data loading process. A data source may extend this class with customized logic.

Annotations
@DeveloperApi()
Source
ColumnarBatch.java
Linear Supertypes
AutoCloseable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ColumnarBatch
  2. AutoCloseable
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new ColumnarBatch(columns: Array[ColumnVector], numRows: Int)

    Create a new batch from existing column vectors.

    Create a new batch from existing column vectors.

    columns

    The columns of this batch

    numRows

    The number of rows in this batch

  2. new ColumnarBatch(columns: Array[ColumnVector])

Value Members

  1. def close(): Unit

    Called to close all the columns in this batch.

    Called to close all the columns in this batch. It is not valid to access the data after calling this. This must be called at the end to clean up memory allocations.

    Definition Classes
    ColumnarBatch → AutoCloseable
    Annotations
    @Override()
  2. def column(ordinal: Int): ColumnVector

    Returns the column at ordinal.

  3. def getRow(rowId: Int): InternalRow

    Returns the row in this batch at rowId.

    Returns the row in this batch at rowId. Returned row is reused across calls.

  4. def numCols(): Int

    Returns the number of columns that make up this batch.

  5. def numRows(): Int

    Returns the number of rows for read, including filtered rows.

  6. def rowIterator(): Iterator[InternalRow]

    Returns an iterator over the rows in this batch.

  7. def setNumRows(numRows: Int): Unit

    Sets the number of rows in this batch.