pyspark.pandas.DataFrame.corr¶
-
DataFrame.
corr
(method: str = 'pearson', min_periods: Optional[int] = None) → pyspark.pandas.frame.DataFrame[source]¶ Compute pairwise correlation of columns, excluding NA/null values.
New in version 3.3.0.
- Parameters
- method{‘pearson’, ‘spearman’, ‘kendall’}
pearson : standard correlation coefficient
spearman : Spearman rank correlation
kendall : Kendall Tau correlation coefficient
Changed in version 3.4.0: support ‘kendall’ for method parameter
- min_periodsint, optional
Minimum number of observations required per pair of columns to have a valid result.
New in version 3.4.0.
- Returns
- DataFrame
See also
Notes
Pearson, Kendall and Spearman correlation are currently computed using pairwise complete observations.
The complexity of Kendall correlation is O(#row * #row), if the dataset is too large, sampling ahead of correlation computation is recommended.
Examples
>>> df = ps.DataFrame([(.2, .3), (.0, .6), (.6, .0), (.2, .1)], ... columns=['dogs', 'cats']) >>> df.corr('pearson') dogs cats dogs 1.000000 -0.851064 cats -0.851064 1.000000
>>> df.corr('spearman') dogs cats dogs 1.000000 -0.948683 cats -0.948683 1.000000
>>> df.corr('kendall') dogs cats dogs 1.000000 -0.912871 cats -0.912871 1.000000