Migration Guide: PySpark (Python on Spark)

Note that this migration guide describes the items specific to PySpark. Many items of SQL migration can be applied when migrating PySpark to higher versions. Please refer Migration Guide: SQL, Datasets and DataFrame.

Upgrading from PySpark 2.4 to 3.0

Upgrading from PySpark 2.3 to 2.4

Upgrading from PySpark 2.3.0 to 2.3.1 and above

Upgrading from PySpark 2.2 to 2.3

Upgrading from PySpark 1.4 to 1.5

Upgrading from PySpark 1.0-1.2 to 1.3

Python DataTypes No Longer Singletons

When using DataTypes in Python you will need to construct them (i.e. StringType()) instead of referencing a singleton.