Upgrading from PySpark 2.2 to 2.3¶
In PySpark, now we need Pandas 0.19.2 or upper if you want to use Pandas related functionalities, such as
createDataFramefrom Pandas DataFrame, etc.
In PySpark, the behavior of timestamp values for Pandas related functionalities was changed to respect session timezone. If you want to use the old behavior, you need to set a configuration
spark.sql.execution.pandas.respectSessionTimeZoneto False. See SPARK-22395 for details.
fillnaalso accepts boolean and replaces nulls with booleans. In prior Spark versions, PySpark just ignores it and returns the original Dataset/DataFrame.
df.replacedoes not allow to omit value when
to_replaceis not a dictionary. Previously, value could be omitted in the other cases and had None by default, which is counterintuitive and error-prone.