class ColumnarBatch extends AutoCloseable
This class wraps multiple ColumnVectors as a row-wise table. It provides a row view of this batch so that Spark can access the data row by row. Instance of it is meant to be reused during the entire data loading process. A data source may extend this class with customized logic.
- Annotations
- @DeveloperApi()
- Source
- ColumnarBatch.java
- Alphabetic
- By Inheritance
- ColumnarBatch
- AutoCloseable
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new ColumnarBatch(columns: Array[ColumnVector], numRows: Int)
Create a new batch from existing column vectors.
Create a new batch from existing column vectors.
- columns
The columns of this batch
- numRows
The number of rows in this batch
- new ColumnarBatch(columns: Array[ColumnVector])
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- def close(): Unit
Called to close all the columns in this batch.
Called to close all the columns in this batch. It is not valid to access the data after calling this. This must be called at the end to clean up memory allocations.
- Definition Classes
- ColumnarBatch → AutoCloseable
- Annotations
- @Override()
- def closeIfFreeable(): Unit
Called to close all the columns if their resources are freeable between batches.
Called to close all the columns if their resources are freeable between batches. This is used to clean up memory allocated during columnar processing.
- def column(ordinal: Int): ColumnVector
Returns the column at
ordinal. - final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def getRow(rowId: Int): InternalRow
Returns the row in this batch at
rowId.Returns the row in this batch at
rowId. Returned row is reused across calls. - def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- def numCols(): Int
Returns the number of columns that make up this batch.
- def numRows(): Int
Returns the number of rows for read, including filtered rows.
- def rowIterator(): Iterator[InternalRow]
Returns an iterator over the rows in this batch.
- def setNumRows(numRows: Int): Unit
Sets the number of rows in this batch.
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)