Package org.apache.spark.streaming.scheduler.rate

Interface RateEstimator

All Superinterfaces:
Serializable
public interface RateEstimator extends Serializable
A component that estimates the rate at which an InputDStream should ingest records, based on updates at every batch completion.

Please see org.apache.spark.streaming.scheduler.RateController for more details.

  • Method Summary

    Modifier and Type
    Method
    Description
    scala.Option<Object>
    compute(long time, long elements, long processingDelay, long schedulingDelay)
    Computes the number of records the stream attached to this RateEstimator should ingest per second, given an update on the size and completion times of the latest batch.

  • Method Details

    • compute

      scala.Option<Object> compute(long time, long elements, long processingDelay, long schedulingDelay)
      Computes the number of records the stream attached to this RateEstimator should ingest per second, given an update on the size and completion times of the latest batch.

      Parameters:
      time - The timestamp of the current batch interval that just finished
      elements - The number of records that were processed in this batch
      processingDelay - The time in ms that took for the job to complete
      schedulingDelay - The time in ms that the job spent in the scheduling queue
      Returns:
      (undocumented)