Package org.apache.spark.sql.avro

Class AvroUtils.AvroSchemaHelper

Object

org.apache.spark.sql.avro.AvroUtils.AvroSchemaHelper

Enclosing class:

AvroUtils

public static class AvroUtils.AvroSchemaHelper
extends Object

Helper class to perform field lookup/matching on Avro schemas.

This will match avroSchema against catalystSchema, attempting to find a matching field in the Avro schema for each field in the Catalyst schema and vice-versa, respecting settings for case sensitivity. The match results can be accessed using the getter methods.

param: avroSchema The schema in which to search for fields. Must be of type RECORD. param: catalystSchema The Catalyst schema to use for matching. param: avroPath The seq of parent field names leading to avroSchema. param: catalystPath The seq of parent field names leading to catalystSchema. param: positionalFieldMatch If true, perform field matching in a positional fashion (structural comparison between schemas, ignoring names); otherwise, perform field matching using field names.

Constructor Summary

Constructors

Constructor	Description
AvroSchemaHelper(org.apache.avro.Schema avroSchema, StructType catalystSchema, scala.collection.immutable.Seq<String> avroPath, scala.collection.immutable.Seq<String> catalystPath, boolean positionalFieldMatch)

Method Summary

Modifier and Type	Method	Description
scala.Option<org.apache.avro.Schema.Field>	getAvroField(String fieldName, int catalystPos)	Get the Avro field corresponding to the provided Catalyst field name/position, if any.
scala.collection.immutable.Seq<org.apache.spark.sql.avro.AvroUtils.AvroMatchedField>	matchedFields()	The fields which have matching equivalents in both Avro and Catalyst schemas.
void	validateNoExtraCatalystFields(boolean ignoreNullable)	Validate that there are no Catalyst fields which don't have a matching Avro field, throwing IncompatibleSchemaException if such extra fields are found.
void	validateNoExtraRequiredAvroFields()	Validate that there are no Avro fields which don't have a matching Catalyst field, throwing IncompatibleSchemaException if such extra fields are found.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

AvroSchemaHelper

public AvroSchemaHelper(org.apache.avro.Schema avroSchema,
 StructType catalystSchema,
 scala.collection.immutable.Seq<String> avroPath,
 scala.collection.immutable.Seq<String> catalystPath,
 boolean positionalFieldMatch)

Method Details

getAvroField

public scala.Option<org.apache.avro.Schema.Field> getAvroField(String fieldName,
 int catalystPos)

Get the Avro field corresponding to the provided Catalyst field name/position, if any.

matchedFields

public scala.collection.immutable.Seq<org.apache.spark.sql.avro.AvroUtils.AvroMatchedField> matchedFields()

The fields which have matching equivalents in both Avro and Catalyst schemas.

validateNoExtraCatalystFields

public void validateNoExtraCatalystFields(boolean ignoreNullable)

Validate that there are no Catalyst fields which don't have a matching Avro field, throwing IncompatibleSchemaException if such extra fields are found. If ignoreNullable is false, consider nullable Catalyst fields to be eligible to be an extra field; otherwise, ignore nullable Catalyst fields when checking for extras.

Parameters:

ignoreNullable - (undocumented)

validateNoExtraRequiredAvroFields

public void validateNoExtraRequiredAvroFields()

Validate that there are no Avro fields which don't have a matching Catalyst field, throwing IncompatibleSchemaException if such extra fields are found. Only required (non-nullable) fields are checked; nullable fields are ignored.