Spark Release 3.1.3

Spark 3.1.3 is a maintenance release containing stability fixes. This release is based on the branch-3.1 maintenance branch of Spark. We strongly recommend all 3.1.3 users to upgrade to this stable release.

Notable changes

  • [SPARK-38075]: Hive script transform with order by and limit will return fake rows
  • [SPARK-37784]: CodeGenerator.addBufferedState() does not properly handle UDTs
  • [SPARK-37079]: Fix DataFrameWriterV2.partitionedBy to send the arguments to JVM properly
  • [SPARK-37049]: executorIdleTimeout is not working for pending pods on K8s
  • [SPARK-36783]: ScanOperation should not push Filter through nondeterministic Project
  • [SPARK-36782]: Deadlock between map-output-dispatcher and dispatcher-BlockManagerMaster upon migrating shuffle blocks
  • [SPARK-36704]: InaccessibleObjectException in Java 9+ on startup
  • [SPARK-36740]: Various NaN issues with array operations.
  • [SPARK-36532]: Deadlock in CoarseGrainedExecutorBackend.onDisconnected
  • [SPARK-36489]: Aggregate functions over no grouping keys, on tables with a single bucket, return multiple rows
  • [SPARK-36339]: aggsBuffer should collect AggregateExpression in the map range
  • [SPARK-35391]: Memory leak in ExecutorAllocationListener breaks dynamic allocation under high load

Dependency Changes

While being a maintence release we did still upgrade some dependencies in this release they are:

You can consult JIRA for the detailed changes.

We would like to acknowledge all community members for contributing patches to this release.

Spark News Archive