pyspark.sql.avro.functions.to_avro#

pyspark.sql.avro.functions.to_avro(data, jsonFormatSchema='')[source]#

Converts a column into binary of avro format.

New in version 3.0.0.

Changed in version 3.5.0: Supports Spark Connect.

Parameters

dataColumn or str: the data column.
jsonFormatSchemastr, optional: user-specified output avro schema in JSON string format.

Notes

Avro is built-in but external data source module since Spark 2.4. Please deploy the application as per the deployment section of “Apache Avro Data Source Guide”.

Examples

>>> from pyspark.sql import Row
>>> from pyspark.sql.avro.functions import to_avro
>>> data = ['SPADES']
>>> df = spark.createDataFrame(data, "string")
>>> df.select(to_avro(df.value).alias("suite")).collect()
[Row(suite=bytearray(b'\x00\x0cSPADES'))]

>>> jsonFormatSchema = '''["null", {"type": "enum", "name": "value",
...     "symbols": ["SPADES", "HEARTS", "DIAMONDS", "CLUBS"]}]'''
>>> df.select(to_avro(df.value, jsonFormatSchema).alias("suite")).collect()
[Row(suite=bytearray(b'\x02\x00'))]