pyspark.BarrierTaskContext.barrier

BarrierTaskContext.barrier()[source]

Sets a global barrier and waits until all tasks in this stage hit this barrier. Similar to MPI_Barrier function in MPI, this function blocks until all tasks in the same stage have reached this routine.

New in version 2.4.0.

Warning

In a barrier stage, each task much have the same number of barrier() calls, in all possible code branches. Otherwise, you may get the job hanging or a SparkException after timeout.

Notes

This API is experimental