Spark Release 3.2.4

Spark 3.2.4 is a maintenance release containing stability fixes. This release is based on the branch-3.2 maintenance branch of Spark. We strongly recommend all 3.2 users to upgrade to this stable release.

Notable changes

  • [SPARK-38173]: Quoted column cannot be recognized correctly when quotedRegexColumnNames is true
  • [SPARK-39399]: proxy-user not working for Spark on k8s in cluster deploy mode
  • [SPARK-40817]: Remote spark.jars URIs ignored for Spark on Kubernetes in cluster mode
  • [SPARK-40819]: Parquet INT64 (TIMESTAMP(NANOS,true)) now throwing Illegal Parquet type instead of automatically converting to LongType
  • [SPARK-41162]: Anti-join must not be pushed below aggregation with ambiguous predicates
  • [SPARK-41254]: YarnAllocator.rpIdToYarnResource map is not properly updated
  • [SPARK-41360]: Avoid BlockManager re-registration if the executor has been lost
  • [SPARK-41376]: Executor netty direct memory check should respect
  • [SPARK-41388]: getReusablePVCs should ignore recently created PVCs in the previous batch
  • [SPARK-41415]: SASL Request Retries
  • [SPARK-41554]: Decimal.changePrecision produces ArrayIndexOutOfBoundsException
  • [SPARK-41732]: Session window: analysis rule “SessionWindowing” does not apply tree-pattern based pruning
  • [SPARK-41952]: Upgrade Parquet to fix off-heap memory leaks in Zstd codec
  • [SPARK-41989]: PYARROW_IGNORE_TIMEZONE warning can break application logging setup
  • [SPARK-42071]: Register scala.math.Ordering$Reverse to KyroSerializer
  • [SPARK-42090]: Introduce sasl retry count in RetryingBlockTransferor
  • [SPARK-42157]: spark.scheduler.mode=FAIR should provide FAIR scheduler
  • [SPARK-42168]: CoGroup with window function returns incorrect result when partition keys differ in order
  • [SPARK-42259]: ResolveGroupingAnalytics should take care of Python UDAF
  • [SPARK-42462]: Prevent from publishing OCI manifests
  • [SPARK-42478]: Make a serializable jobTrackerId instead of a non-serializable JobID in FileWriterFactory
  • [SPARK-42596]: OMP_NUM_THREADS not set to number of executor cores by default
  • [SPARK-42697]: /api/v1/applications return 0 for duration
  • [SPARK-42747]: Fix incorrect internal status of LoR and AFT
  • [SPARK-42785]: When spark submit without –deploy-mode, will face NPE in Kubernetes Case
  • [SPARK-42906]: Replace a starting digit with x in resource name prefix
  • [SPARK-42967]: Fix SparkListenerTaskStart.stageAttemptId when a task is started after the stage is cancelled
  • [SPARK-43004]: vendor==vendor typo in ResourceRequest.equals()

Dependency Changes

While being a maintenance release we did still upgrade some dependencies in this release they are:

You can consult JIRA for the detailed changes.

We would like to acknowledge all community members for contributing patches to this release.

Spark News Archive