Class AvroUtils.AvroSchemaHelper

Object
org.apache.spark.sql.avro.AvroUtils.AvroSchemaHelper
Enclosing class:
AvroUtils

public static class AvroUtils.AvroSchemaHelper extends Object
Helper class to perform field lookup/matching on Avro schemas.

This will match avroSchema against catalystSchema, attempting to find a matching field in the Avro schema for each field in the Catalyst schema and vice-versa, respecting settings for case sensitivity. The match results can be accessed using the getter methods.

param: avroSchema The schema in which to search for fields. Must be of type RECORD. param: catalystSchema The Catalyst schema to use for matching. param: avroPath The seq of parent field names leading to avroSchema. param: catalystPath The seq of parent field names leading to catalystSchema. param: positionalFieldMatch If true, perform field matching in a positional fashion (structural comparison between schemas, ignoring names); otherwise, perform field matching using field names.

  • Constructor Summary

    Constructors
    Constructor
    Description
    AvroSchemaHelper(org.apache.avro.Schema avroSchema, StructType catalystSchema, scala.collection.immutable.Seq<String> avroPath, scala.collection.immutable.Seq<String> catalystPath, boolean positionalFieldMatch)
     
  • Method Summary

    Modifier and Type
    Method
    Description
    scala.Option<org.apache.avro.Schema.Field>
    getAvroField(String fieldName, int catalystPos)
    Get the Avro field corresponding to the provided Catalyst field name/position, if any.
    scala.collection.immutable.Seq<org.apache.spark.sql.avro.AvroUtils.AvroMatchedField>
    The fields which have matching equivalents in both Avro and Catalyst schemas.
    void
    validateNoExtraCatalystFields(boolean ignoreNullable)
    Validate that there are no Catalyst fields which don't have a matching Avro field, throwing IncompatibleSchemaException if such extra fields are found.
    void
    Validate that there are no Avro fields which don't have a matching Catalyst field, throwing IncompatibleSchemaException if such extra fields are found.

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
  • Constructor Details

    • AvroSchemaHelper

      public AvroSchemaHelper(org.apache.avro.Schema avroSchema, StructType catalystSchema, scala.collection.immutable.Seq<String> avroPath, scala.collection.immutable.Seq<String> catalystPath, boolean positionalFieldMatch)
  • Method Details

    • getAvroField

      public scala.Option<org.apache.avro.Schema.Field> getAvroField(String fieldName, int catalystPos)
      Get the Avro field corresponding to the provided Catalyst field name/position, if any.
    • matchedFields

      public scala.collection.immutable.Seq<org.apache.spark.sql.avro.AvroUtils.AvroMatchedField> matchedFields()
      The fields which have matching equivalents in both Avro and Catalyst schemas.
    • validateNoExtraCatalystFields

      public void validateNoExtraCatalystFields(boolean ignoreNullable)
      Validate that there are no Catalyst fields which don't have a matching Avro field, throwing IncompatibleSchemaException if such extra fields are found. If ignoreNullable is false, consider nullable Catalyst fields to be eligible to be an extra field; otherwise, ignore nullable Catalyst fields when checking for extras.
      Parameters:
      ignoreNullable - (undocumented)
    • validateNoExtraRequiredAvroFields

      public void validateNoExtraRequiredAvroFields()
      Validate that there are no Avro fields which don't have a matching Catalyst field, throwing IncompatibleSchemaException if such extra fields are found. Only required (non-nullable) fields are checked; nullable fields are ignored.