Class ArrowUtils

Object
org.apache.spark.sql.util.ArrowUtils

public class ArrowUtils extends Object
  • Constructor Details

    • ArrowUtils

      public ArrowUtils()
  • Method Details

    • rootAllocator

      public static org.apache.arrow.memory.RootAllocator rootAllocator()
    • toArrowType

      public static org.apache.arrow.vector.types.pojo.ArrowType toArrowType(DataType dt, String timeZoneId, boolean largeVarTypes)
      Maps data type from Spark to Arrow. NOTE: timeZoneId required for TimestampTypes
    • fromArrowType

      public static DataType fromArrowType(org.apache.arrow.vector.types.pojo.ArrowType dt)
    • toArrowField

      public static org.apache.arrow.vector.types.pojo.Field toArrowField(String name, DataType dt, boolean nullable, String timeZoneId, boolean largeVarTypes)
      Maps field from Spark to Arrow. NOTE: timeZoneId required for TimestampType
    • fromArrowField

      public static DataType fromArrowField(org.apache.arrow.vector.types.pojo.Field field)
    • toArrowSchema

      public static org.apache.arrow.vector.types.pojo.Schema toArrowSchema(StructType schema, String timeZoneId, boolean errorOnDuplicatedFieldNames, boolean largeVarTypes)
      Maps schema from Spark to Arrow. NOTE: timeZoneId required for TimestampType in StructType
      Parameters:
      schema - (undocumented)
      timeZoneId - (undocumented)
      errorOnDuplicatedFieldNames - (undocumented)
      largeVarTypes - (undocumented)
      Returns:
      (undocumented)
    • fromArrowSchema

      public static StructType fromArrowSchema(org.apache.arrow.vector.types.pojo.Schema schema)