pyspark.resource.ResourceProfileBuilder

class pyspark.resource.ResourceProfileBuilder[source]

Resource profile Builder to build a resource profile to associate with an RDD. A ResourceProfile allows the user to specify executor and task requirements for an RDD that will get applied during a stage. This allows the user to change the resource requirements between stages.

New in version 3.1.0.

Notes

This API is evolving.

Methods

clearExecutorResourceRequests()

clearTaskResourceRequests()

require(resourceRequest)

Attributes

build

executorResources

taskResources