org.apache.spark.streaming.scheduler.rate

Interface RateEstimator

All Superinterfaces:

java.io.Serializable

public interface RateEstimator
extends scala.Serializable

A component that estimates the rate at which an InputDStream should ingest records, based on updates at every batch completion.

Please see org.apache.spark.streaming.scheduler.RateController for more details.

Method Summary

Modifier and Type	Method and Description
scala.Option<Object>	compute(long time, long elements, long processingDelay, long schedulingDelay) Computes the number of records the stream attached to this RateEstimator should ingest per second, given an update on the size and completion times of the latest batch.

Method Detail

compute

scala.Option<Object> compute(long time,
                             long elements,
                             long processingDelay,
                             long schedulingDelay)

Computes the number of records the stream attached to this RateEstimator should ingest per second, given an update on the size and completion times of the latest batch.

Parameters:

time - The timestamp of the current batch interval that just finished

elements - The number of records that were processed in this batch

processingDelay - The time in ms that took for the job to complete

schedulingDelay - The time in ms that the job spent in the scheduling queue

Returns:

(undocumented)