Packages

abstract class Receiver[T] extends Serializable

:: DeveloperApi :: Abstract class of a receiver that can be run on worker nodes to receive external data. A custom receiver can be defined by defining the functions onStart() and onStop(). onStart() should define the setup steps necessary to start receiving data, and onStop() should define the cleanup steps necessary to stop receiving data. Exceptions while receiving can be handled either by restarting the receiver with restart(...) or stopped completely by stop(...).

A custom receiver in Scala would look like this.

class MyReceiver(storageLevel: StorageLevel) extends NetworkReceiver[String](storageLevel) {
    def onStart() {
        // Setup stuff (start threads, open sockets, etc.) to start receiving data.
        // Must start new thread to receive data, as onStart() must be non-blocking.

        // Call store(...) in those threads to store received data into Spark's memory.

        // Call stop(...), restart(...) or reportError(...) on any thread based on how
        // different errors need to be handled.

        // See corresponding method documentation for more details
    }

    def onStop() {
        // Cleanup stuff (stop threads, close sockets, etc.) to stop receiving data.
    }
}

A custom receiver in Java would look like this.

class MyReceiver extends Receiver<String> {
    public MyReceiver(StorageLevel storageLevel) {
        super(storageLevel);
    }

    public void onStart() {
         // Setup stuff (start threads, open sockets, etc.) to start receiving data.
         // Must start new thread to receive data, as onStart() must be non-blocking.

         // Call store(...) in those threads to store received data into Spark's memory.

         // Call stop(...), restart(...) or reportError(...) on any thread based on how
         // different errors need to be handled.

         // See corresponding method documentation for more details
    }

    public void onStop() {
         // Cleanup stuff (stop threads, close sockets, etc.) to stop receiving data.
    }
}
Annotations
@DeveloperApi()
Source
Receiver.scala
Linear Supertypes
Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Receiver
  2. Serializable
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new Receiver(storageLevel: StorageLevel)

Abstract Value Members

  1. abstract def onStart(): Unit

    This method is called by the system when the receiver is started.

    This method is called by the system when the receiver is started. This function must initialize all resources (threads, buffers, etc.) necessary for receiving data. This function must be non-blocking, so receiving the data must occur on a different thread. Received data can be stored with Spark by calling store(data).

    If there are errors in threads started here, then following options can be done (i) reportError(...) can be called to report the error to the driver. The receiving of data will continue uninterrupted. (ii) stop(...) can be called to stop receiving data. This will call onStop() to clear up all resources allocated (threads, buffers, etc.) during onStart(). (iii) restart(...) can be called to restart the receiver. This will call onStop() immediately, and then onStart() after a delay.

  2. abstract def onStop(): Unit

    This method is called by the system when the receiver is stopped.

    This method is called by the system when the receiver is stopped. All resources (threads, buffers, etc.) set up in onStart() must be cleaned up in this method.

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @IntrinsicCandidate()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @IntrinsicCandidate()
  9. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @IntrinsicCandidate()
  10. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  11. def isStarted(): Boolean

    Check if the receiver has started or not.

  12. def isStopped(): Boolean

    Check if receiver has been marked for stopping.

    Check if receiver has been marked for stopping. Use this to identify when the receiving of data should be stopped.

  13. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  14. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @IntrinsicCandidate()
  15. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @IntrinsicCandidate()
  16. def preferredLocation: Option[String]

    Override this to specify a preferred location (hostname).

  17. def reportError(message: String, throwable: Throwable): Unit

    Report exceptions in receiving data.

  18. def restart(message: String, error: Throwable, millisecond: Int): Unit

    Restart the receiver.

    Restart the receiver. This method schedules the restart and returns immediately. The stopping and subsequent starting of the receiver (by calling onStop() and onStart()) is performed asynchronously in a background thread.

  19. def restart(message: String, error: Throwable): Unit

    Restart the receiver.

    Restart the receiver. This method schedules the restart and returns immediately. The stopping and subsequent starting of the receiver (by calling onStop() and onStart()) is performed asynchronously in a background thread. The delay between the stopping and the starting is defined by the Spark configuration spark.streaming.receiverRestartDelay. The message and exception will be reported to the driver.

  20. def restart(message: String): Unit

    Restart the receiver.

    Restart the receiver. This method schedules the restart and returns immediately. The stopping and subsequent starting of the receiver (by calling onStop() and onStart()) is performed asynchronously in a background thread. The delay between the stopping and the starting is defined by the Spark configuration spark.streaming.receiverRestartDelay. The message will be reported to the driver.

  21. def stop(message: String, error: Throwable): Unit

    Stop the receiver completely due to an exception

  22. def stop(message: String): Unit

    Stop the receiver completely.

  23. val storageLevel: StorageLevel
  24. def store(bytes: ByteBuffer, metadata: Any): Unit

    Store the bytes of received data as a data block into Spark's memory.

    Store the bytes of received data as a data block into Spark's memory. The metadata will be associated with this block of data for being used in the corresponding InputDStream.

  25. def store(bytes: ByteBuffer): Unit

    Store the bytes of received data as a data block into Spark's memory.

    Store the bytes of received data as a data block into Spark's memory. Note that the data in the ByteBuffer must be serialized using the same serializer that Spark is configured to use.

  26. def store(dataIterator: Iterator[T], metadata: Any): Unit

    Store an iterator of received data as a data block into Spark's memory.

    Store an iterator of received data as a data block into Spark's memory. The metadata will be associated with this block of data for being used in the corresponding InputDStream.

  27. def store(dataIterator: Iterator[T]): Unit

    Store an iterator of received data as a data block into Spark's memory.

  28. def store(dataIterator: Iterator[T], metadata: Any): Unit

    Store an iterator of received data as a data block into Spark's memory.

    Store an iterator of received data as a data block into Spark's memory. The metadata will be associated with this block of data for being used in the corresponding InputDStream.

  29. def store(dataIterator: Iterator[T]): Unit

    Store an iterator of received data as a data block into Spark's memory.

  30. def store(dataBuffer: ArrayBuffer[T], metadata: Any): Unit

    Store an ArrayBuffer of received data as a data block into Spark's memory.

    Store an ArrayBuffer of received data as a data block into Spark's memory. The metadata will be associated with this block of data for being used in the corresponding InputDStream.

  31. def store(dataBuffer: ArrayBuffer[T]): Unit

    Store an ArrayBuffer of received data as a data block into Spark's memory.

  32. def store(dataItem: T): Unit

    Store a single item of received data to Spark's memory.

    Store a single item of received data to Spark's memory. These single items will be aggregated together into data blocks before being pushed into Spark's memory.

  33. def streamId: Int

    Get the unique identifier the receiver input stream that this receiver is associated with.

  34. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  35. def toString(): String
    Definition Classes
    AnyRef → Any
  36. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  37. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  38. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped