Package

org.apache.spark.sql

streaming

Permalink

package streaming

Visibility
  1. Public
  2. All

Type Members

  1. final class DataStreamReader extends Logging

    Permalink

    Interface used to load a streaming Dataset from external storage systems (e.g.

    Interface used to load a streaming Dataset from external storage systems (e.g. file systems, key-value stores, etc). Use SparkSession.readStream to access this.

    Annotations
    @Experimental() @Evolving()
    Since

    2.0.0

  2. final class DataStreamWriter[T] extends AnyRef

    Permalink

    :: Experimental :: Interface used to write a streaming Dataset to external storage systems (e.g.

    :: Experimental :: Interface used to write a streaming Dataset to external storage systems (e.g. file systems, key-value stores, etc). Use Dataset.writeStream to access this.

    Annotations
    @Experimental() @Evolving()
    Since

    2.0.0

  3. class OutputMode extends AnyRef

    Permalink
  4. case class ProcessingTime(intervalMs: Long) extends Trigger with Product with Serializable

    Permalink

    :: Experimental :: A trigger that runs a query periodically based on the processing time.

    :: Experimental :: A trigger that runs a query periodically based on the processing time. If interval is 0, the query will run as fast as possible.

    Scala Example:

    df.write.trigger(ProcessingTime("10 seconds"))
    
    import scala.concurrent.duration._
    df.write.trigger(ProcessingTime(10.seconds))

    Java Example:

    df.write.trigger(ProcessingTime.create("10 seconds"))
    
    import java.util.concurrent.TimeUnit
    df.write.trigger(ProcessingTime.create(10, TimeUnit.SECONDS))
    Annotations
    @Experimental() @Evolving()
    Since

    2.0.0

  5. class SinkProgress extends Serializable

    Permalink

    :: Experimental :: Information about progress made for a sink in the execution of a StreamingQuery during a trigger.

    :: Experimental :: Information about progress made for a sink in the execution of a StreamingQuery during a trigger. See StreamingQueryProgress for more information.

    Annotations
    @Experimental() @Evolving()
    Since

    2.1.0

  6. class SourceProgress extends Serializable

    Permalink

    :: Experimental :: Information about progress made for a source in the execution of a StreamingQuery during a trigger.

    :: Experimental :: Information about progress made for a source in the execution of a StreamingQuery during a trigger. See StreamingQueryProgress for more information.

    Annotations
    @Experimental() @Evolving()
    Since

    2.1.0

  7. class StateOperatorProgress extends Serializable

    Permalink

    :: Experimental :: Information about updates made to stateful operators in a StreamingQuery during a trigger.

    :: Experimental :: Information about updates made to stateful operators in a StreamingQuery during a trigger.

    Annotations
    @Experimental() @Evolving()
  8. trait StreamingQuery extends AnyRef

    Permalink

    :: Experimental :: A handle to a query that is executing continuously in the background as new data arrives.

    :: Experimental :: A handle to a query that is executing continuously in the background as new data arrives. All these methods are thread-safe.

    Annotations
    @Experimental() @Evolving()
    Since

    2.0.0

  9. class StreamingQueryException extends Exception

    Permalink

    :: Experimental :: Exception that stopped a StreamingQuery.

    :: Experimental :: Exception that stopped a StreamingQuery. Use cause get the actual exception that caused the failure.

    Annotations
    @Experimental() @Evolving()
    Since

    2.0.0

  10. abstract class StreamingQueryListener extends AnyRef

    Permalink

    :: Experimental :: Interface for listening to events related to StreamingQueries.

    :: Experimental :: Interface for listening to events related to StreamingQueries.

    Annotations
    @Experimental() @Evolving()
    Since

    2.0.0

    Note

    The methods are not thread-safe as they may be called from different threads.

  11. class StreamingQueryManager extends AnyRef

    Permalink

    :: Experimental :: A class to manage all the StreamingQuery active on a SparkSession.

    :: Experimental :: A class to manage all the StreamingQuery active on a SparkSession.

    Annotations
    @Experimental() @Evolving()
    Since

    2.0.0

  12. class StreamingQueryProgress extends Serializable

    Permalink

    :: Experimental :: Information about progress made in the execution of a StreamingQuery during a trigger.

    :: Experimental :: Information about progress made in the execution of a StreamingQuery during a trigger. Each event relates to processing done for a single trigger of the streaming query. Events are emitted even when no new data is available to be processed.

    Annotations
    @Experimental() @Evolving()
    Since

    2.1.0

  13. class StreamingQueryStatus extends Serializable

    Permalink

    :: Experimental :: Reports information about the instantaneous status of a streaming query.

    :: Experimental :: Reports information about the instantaneous status of a streaming query.

    Annotations
    @Experimental() @Evolving()
    Since

    2.1.0

  14. sealed trait Trigger extends AnyRef

    Permalink

    :: Experimental :: Used to indicate how often results should be produced by a StreamingQuery.

    :: Experimental :: Used to indicate how often results should be produced by a StreamingQuery.

    Annotations
    @Experimental() @Evolving()
    Since

    2.0.0

Value Members

  1. object ProcessingTime extends Serializable

    Permalink

    :: Experimental :: Used to create ProcessingTime triggers for StreamingQuerys.

    :: Experimental :: Used to create ProcessingTime triggers for StreamingQuerys.

    Annotations
    @Experimental() @Evolving()
    Since

    2.0.0

  2. object StreamingQueryListener

    Permalink

    :: Experimental :: Companion object of StreamingQueryListener that defines the listener events.

    :: Experimental :: Companion object of StreamingQueryListener that defines the listener events.

    Annotations
    @Experimental() @Evolving()
    Since

    2.0.0

Ungrouped