public class KolmogorovSmirnovTest
extends java.lang.Object
Implementation note: We seek to implement the KS test with a minimal number of distributed passes. We sort the RDD, and then perform the following operations on a per-partition basis: calculate an empirical cumulative distribution value for each observation, and a theoretical cumulative distribution value. We know the latter to be correct, while the former will be off by a constant (how large the constant is depends on how many values precede it in other partitions). However, given that this constant simply shifts the empirical CDF upwards, but doesn't change its shape, and furthermore, that constant is the same within a given partition, we can pick 2 values in each partition that can potentially resolve to the largest global distance. Namely, we pick the minimum distance and the maximum distance. Additionally, we keep track of how many elements are in each partition. Once these three values have been returned for every partition, we can collect and operate locally. Locally, we can now adjust each distance by the appropriate constant (the cumulative sum of number of elements in the prior partitions divided by thedata set size). Finally, we take the maximum absolute value, and this is the statistic.
Modifier and Type | Class and Description |
---|---|
static class |
KolmogorovSmirnovTest.NullHypothesis$ |
Constructor and Description |
---|
KolmogorovSmirnovTest() |
Modifier and Type | Method and Description |
---|---|
protected static void |
initializeLogIfNecessary(boolean isInterpreter) |
protected static boolean |
isTraceEnabled() |
protected static org.slf4j.Logger |
log() |
protected static void |
logDebug(scala.Function0<java.lang.String> msg) |
protected static void |
logDebug(scala.Function0<java.lang.String> msg,
java.lang.Throwable throwable) |
protected static void |
logError(scala.Function0<java.lang.String> msg) |
protected static void |
logError(scala.Function0<java.lang.String> msg,
java.lang.Throwable throwable) |
protected static void |
logInfo(scala.Function0<java.lang.String> msg) |
protected static void |
logInfo(scala.Function0<java.lang.String> msg,
java.lang.Throwable throwable) |
protected static java.lang.String |
logName() |
protected static void |
logTrace(scala.Function0<java.lang.String> msg) |
protected static void |
logTrace(scala.Function0<java.lang.String> msg,
java.lang.Throwable throwable) |
protected static void |
logWarning(scala.Function0<java.lang.String> msg) |
protected static void |
logWarning(scala.Function0<java.lang.String> msg,
java.lang.Throwable throwable) |
static KolmogorovSmirnovTestResult |
testOneSample(RDD<java.lang.Object> data,
scala.Function1<java.lang.Object,java.lang.Object> cdf) |
static KolmogorovSmirnovTestResult |
testOneSample(RDD<java.lang.Object> data,
org.apache.commons.math3.distribution.RealDistribution distObj) |
static KolmogorovSmirnovTestResult |
testOneSample(RDD<java.lang.Object> data,
java.lang.String distName,
double... params)
A convenience function that allows running the KS test for 1 set of sample data against
a named distribution
|
static KolmogorovSmirnovTestResult |
testOneSample(RDD<java.lang.Object> data,
java.lang.String distName,
scala.collection.Seq<java.lang.Object> params) |
public static KolmogorovSmirnovTestResult testOneSample(RDD<java.lang.Object> data, java.lang.String distName, double... params)
data
- the sample data that we wish to evaluatedistName
- the name of the theoretical distributionparams
- Variable length parameter for distribution's parametersKolmogorovSmirnovTestResult
summarizing the
test results (p-value, statistic, and null hypothesis)public static KolmogorovSmirnovTestResult testOneSample(RDD<java.lang.Object> data, scala.Function1<java.lang.Object,java.lang.Object> cdf)
public static KolmogorovSmirnovTestResult testOneSample(RDD<java.lang.Object> data, org.apache.commons.math3.distribution.RealDistribution distObj)
public static KolmogorovSmirnovTestResult testOneSample(RDD<java.lang.Object> data, java.lang.String distName, scala.collection.Seq<java.lang.Object> params)
protected static java.lang.String logName()
protected static org.slf4j.Logger log()
protected static void logInfo(scala.Function0<java.lang.String> msg)
protected static void logDebug(scala.Function0<java.lang.String> msg)
protected static void logTrace(scala.Function0<java.lang.String> msg)
protected static void logWarning(scala.Function0<java.lang.String> msg)
protected static void logError(scala.Function0<java.lang.String> msg)
protected static void logInfo(scala.Function0<java.lang.String> msg, java.lang.Throwable throwable)
protected static void logDebug(scala.Function0<java.lang.String> msg, java.lang.Throwable throwable)
protected static void logTrace(scala.Function0<java.lang.String> msg, java.lang.Throwable throwable)
protected static void logWarning(scala.Function0<java.lang.String> msg, java.lang.Throwable throwable)
protected static void logError(scala.Function0<java.lang.String> msg, java.lang.Throwable throwable)
protected static boolean isTraceEnabled()
protected static void initializeLogIfNecessary(boolean isInterpreter)