Spark Release 3.1.3
Spark 3.1.3 is a maintenance release containing stability fixes. This release is based on the branch-3.1 maintenance branch of Spark. We strongly recommend all 3.1.3 users to upgrade to this stable release.
- [SPARK-38075]: Hive script transform with order by and limit will return fake rows
- [SPARK-37784]: CodeGenerator.addBufferedState() does not properly handle UDTs
- [SPARK-37079]: Fix DataFrameWriterV2.partitionedBy to send the arguments to JVM properly
- [SPARK-37049]: executorIdleTimeout is not working for pending pods on K8s
- [SPARK-36783]: ScanOperation should not push Filter through nondeterministic Project
- [SPARK-36782]: Deadlock between map-output-dispatcher and dispatcher-BlockManagerMaster upon migrating shuffle blocks
- [SPARK-36704]: InaccessibleObjectException in Java 9+ on startup
- [SPARK-36740]: Various NaN issues with array operations.
- [SPARK-36532]: Deadlock in CoarseGrainedExecutorBackend.onDisconnected
- [SPARK-36489]: Aggregate functions over no grouping keys, on tables with a single bucket, return multiple rows
- [SPARK-36339]: aggsBuffer should collect AggregateExpression in the map range
- [SPARK-35391]: Memory leak in ExecutorAllocationListener breaks dynamic allocation under high load
While being a maintence release we did still upgrade some dependencies in this release they are:
You can consult JIRA for the detailed changes.
We would like to acknowledge all community members for contributing patches to this release.
Spark News Archive