pyspark.sql.DataFrameStatFunctions.sampleBy¶
-
DataFrameStatFunctions.
sampleBy
(col: str, fractions: Dict[Any, float], seed: Optional[int] = None) → pyspark.sql.dataframe.DataFrame[source]¶ Returns a stratified sample without replacement based on the fraction given on each stratum.
New in version 1.5.0.
Changed in version 3.4.0: Supports Spark Connect.
- Parameters
- Returns
- a new
DataFrame
that represents the stratified sample
- a new
Examples
>>> from pyspark.sql.functions import col >>> dataset = spark.range(0, 100).select((col("id") % 3).alias("key")) >>> sampled = dataset.sampleBy("key", fractions={0: 0.1, 1: 0.2}, seed=0) >>> sampled.groupBy("key").count().orderBy("key").show() +---+-----+ |key|count| +---+-----+ | 0| 3| | 1| 6| +---+-----+ >>> dataset.sampleBy(col("key"), fractions={2: 1.0}, seed=0).count() 33