Package org.apache.spark.streaming.receiver

Interface BlockGeneratorListener

public interface BlockGeneratorListener
Listener object for BlockGenerator events

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    onAddData(Object data, Object metadata)
    Called after a data item is added into the BlockGenerator.
    void
    onError(String message, Throwable throwable)
    Called when an error has occurred in the BlockGenerator.
    void
    onGenerateBlock(StreamBlockId blockId)
    Called when a new block of data is generated by the block generator.
    void
    onPushBlock(StreamBlockId blockId, scala.collection.mutable.ArrayBuffer<?> arrayBuffer)
    Called when a new block is ready to be pushed.

  • Method Details

    • onAddData

      void onAddData(Object data, Object metadata)
      Called after a data item is added into the BlockGenerator. The data addition and this callback are synchronized with the block generation and its associated callback, so block generation waits for the active data addition+callback to complete. This is useful for updating metadata on successful buffering of a data item, specifically that metadata that will be useful when a block is generated. Any long blocking operation in this callback will hurt the throughput.
      Parameters:
      data - (undocumented)
      metadata - (undocumented)

    • onError

      void onError(String message, Throwable throwable)
      Called when an error has occurred in the BlockGenerator. Can be called form many places so better to not do any long block operation in this callback.
      Parameters:
      message - (undocumented)
      throwable - (undocumented)

    • onGenerateBlock

      void onGenerateBlock(StreamBlockId blockId)
      Called when a new block of data is generated by the block generator. The block generation and this callback are synchronized with the data addition and its associated callback, so the data addition waits for the block generation+callback to complete. This is useful for updating metadata when a block has been generated, specifically metadata that will be useful when the block has been successfully stored. Any long blocking operation in this callback will hurt the throughput.
      Parameters:
      blockId - (undocumented)

    • onPushBlock

      void onPushBlock(StreamBlockId blockId, scala.collection.mutable.ArrayBuffer<?> arrayBuffer)
      Called when a new block is ready to be pushed. Callers are supposed to store the block into Spark in this method. Internally this is called from a single thread, that is not synchronized with any other callbacks. Hence it is okay to do long blocking operation in this callback.
      Parameters:
      blockId - (undocumented)
      arrayBuffer - (undocumented)