class StreamingTest extends Logging with Serializable
Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs. The Boolean identifies which sample each observation comes from, and the Double is the numeric value of the observation.
To address novelty affects, the peacePeriod
specifies a set number of initial
org.apache.spark.rdd.RDD batches of the DStream
to be dropped from significance testing.
The windowSize
sets the number of batches each significance test is to be performed over. The
window is sliding with a stride length of 1 batch. Setting windowSize to 0 will perform
cumulative processing, using all batches seen so far.
Different tests may be used for assessing statistical significance depending on assumptions
satisfied by data. For more details, see StreamingTestMethod
. The testMethod
specifies
which test will be used.
Use a builder pattern to construct a streaming test in an application, for example:
val model = new StreamingTest() .setPeacePeriod(10) .setWindowSize(0) .setTestMethod("welch") .registerStream(DStream)
- Annotations
- @Since( "1.6.0" )
- Source
- StreamingTest.scala
- Alphabetic
- By Inheritance
- StreamingTest
- Serializable
- Serializable
- Logging
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
-
new
StreamingTest()
- Annotations
- @Since( "1.6.0" )
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
def
initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
def
isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
def
log: Logger
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logName: String
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
def
registerStream(data: JavaDStream[BinarySample]): JavaDStream[StreamingTestResult]
Register a
JavaDStream
of values for significance testing.Register a
JavaDStream
of values for significance testing.- data
stream of BinarySample(isExperiment,value) pairs where the isExperiment denotes group (true = experiment, false = control) and the value is the numerical metric to test for significance
- returns
stream of significance testing results
- Annotations
- @Since( "1.6.0" )
-
def
registerStream(data: DStream[BinarySample]): DStream[StreamingTestResult]
Register a
DStream
of values for significance testing.Register a
DStream
of values for significance testing.- data
stream of BinarySample(key,value) pairs where the key denotes group membership (true = experiment, false = control) and the value is the numerical metric to test for significance
- returns
stream of significance testing results
- Annotations
- @Since( "1.6.0" )
-
def
setPeacePeriod(peacePeriod: Int): StreamingTest.this.type
Set the number of initial batches to ignore.
Set the number of initial batches to ignore. Default: 0.
- Annotations
- @Since( "1.6.0" )
-
def
setTestMethod(method: String): StreamingTest.this.type
Set the statistical method used for significance testing.
Set the statistical method used for significance testing. Default: "welch"
- Annotations
- @Since( "1.6.0" )
-
def
setWindowSize(windowSize: Int): StreamingTest.this.type
Set the number of batches to compute significance tests over.
Set the number of batches to compute significance tests over. Default: 0. A value of 0 will use all batches seen so far.
- Annotations
- @Since( "1.6.0" )
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()