Packages

class Builder extends AnyRef

Builder for KinesisInputDStream instances.

Source
KinesisInputDStream.scala
Since

2.2.0

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Builder
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new Builder()

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def build(): KinesisInputDStream[Array[Byte]]

    Create a new instance of KinesisInputDStream with configured parameters and using the default message handler, which returns Array[Byte].

    Create a new instance of KinesisInputDStream with configured parameters and using the default message handler, which returns Array[Byte].

    returns

    Instance of KinesisInputDStream constructed with configured parameters

  6. def buildWithMessageHandler[T](handler: (Record) ⇒ T)(implicit arg0: ClassTag[T]): KinesisInputDStream[T]

    Create a new instance of KinesisInputDStream with configured parameters and the provided message handler.

    Create a new instance of KinesisInputDStream with configured parameters and the provided message handler.

    handler

    Function converting Record instances read by the KCL to DStream type T

    returns

    Instance of KinesisInputDStream constructed with configured parameters

  7. def checkpointAppName(appName: String): Builder

    Sets the KCL application name to use when checkpointing state to DynamoDB.

    Sets the KCL application name to use when checkpointing state to DynamoDB. This is a required parameter.

    appName

    Value to use for the KCL app name (used when creating the DynamoDB checkpoint table and when writing metrics to CloudWatch)

    returns

    Reference to this KinesisInputDStream.Builder

  8. def checkpointInterval(interval: Duration): Builder

    Sets how often the KCL application state is checkpointed to DynamoDB.

    Sets how often the KCL application state is checkpointed to DynamoDB. Defaults to the Spark Streaming batch interval if no custom value is specified.

    interval

    Duration specifying how often the KCL state should be checkpointed to DynamoDB.

    returns

    Reference to this KinesisInputDStream.Builder

  9. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  10. def cloudWatchCredentials(credentials: SparkAWSCredentials): Builder

    Sets the SparkAWSCredentials to use for authenticating to the AWS CloudWatch endpoint.

    Sets the SparkAWSCredentials to use for authenticating to the AWS CloudWatch endpoint. Will use the same credentials used for AWS Kinesis if no custom value is set.

    credentials

    SparkAWSCredentials to use for CloudWatch authentication

    returns

    Reference to this KinesisInputDStream.Builder

  11. def dynamoDBCredentials(credentials: SparkAWSCredentials): Builder

    Sets the SparkAWSCredentials to use for authenticating to the AWS DynamoDB endpoint.

    Sets the SparkAWSCredentials to use for authenticating to the AWS DynamoDB endpoint. Will use the same credentials used for AWS Kinesis if no custom value is set.

    credentials

    SparkAWSCredentials to use for DynamoDB authentication

    returns

    Reference to this KinesisInputDStream.Builder

  12. def endpointUrl(url: String): Builder

    Sets the AWS Kinesis endpoint URL.

    Sets the AWS Kinesis endpoint URL. Defaults to "https://kinesis.us-east-1.amazonaws.com" if no custom value is specified

    url

    Kinesis endpoint URL to use

    returns

    Reference to this KinesisInputDStream.Builder

  13. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  14. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  15. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  16. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  17. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  18. def initialPosition(initialPosition: KinesisInitialPosition): Builder

    Sets the initial position data is read from in the Kinesis stream.

    Sets the initial position data is read from in the Kinesis stream. Defaults to KinesisInitialPositions.Latest if no custom value is specified.

    initialPosition

    KinesisInitialPosition value specifying where Spark Streaming will start reading records in the Kinesis stream from

    returns

    Reference to this KinesisInputDStream.Builder

  19. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  20. def kinesisCredentials(credentials: SparkAWSCredentials): Builder

    Sets the SparkAWSCredentials to use for authenticating to the AWS Kinesis endpoint.

    Sets the SparkAWSCredentials to use for authenticating to the AWS Kinesis endpoint. Defaults to DefaultCredentialsProvider if no custom value is specified.

    credentials

    SparkAWSCredentials to use for Kinesis authentication

    returns

    Reference to this KinesisInputDStream.Builder

  21. def metricsEnabledDimensions(metricsEnabledDimensions: Set[String]): Builder

    Sets the enabled CloudWatch metrics dimensions.

    Sets the enabled CloudWatch metrics dimensions. Defaults to KinesisClientLibConfiguration.DEFAULT_METRICS_ENABLED_DIMENSIONS if no custom value is specified.

    metricsEnabledDimensions

    Set[String] to specify which CloudWatch metrics dimensions should be enabled

    returns

    Reference to this KinesisInputDStream.Builder

    See also

    https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-kcl.html#metric-levels

  22. def metricsLevel(metricsLevel: MetricsLevel): Builder

    Sets the CloudWatch metrics level.

    Sets the CloudWatch metrics level. Defaults to KinesisClientLibConfiguration.DEFAULT_METRICS_LEVEL if no custom value is specified.

    metricsLevel

    MetricsLevel to specify the CloudWatch metrics level

    returns

    Reference to this KinesisInputDStream.Builder

    See also

    https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-kcl.html#metric-levels

  23. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  24. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  25. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  26. def regionName(regionName: String): Builder

    Sets the AWS region to construct clients for.

    Sets the AWS region to construct clients for. Defaults to "us-east-1" if no custom value is specified.

    regionName

    Name of AWS region to use (e.g. "us-west-2")

    returns

    Reference to this KinesisInputDStream.Builder

  27. def storageLevel(storageLevel: StorageLevel): Builder

    Sets the storage level of the blocks for the DStream created.

    Sets the storage level of the blocks for the DStream created. Defaults to StorageLevel.MEMORY_AND_DISK_2 if no custom value is specified.

    storageLevel

    StorageLevel to use for the DStream data blocks

    returns

    Reference to this KinesisInputDStream.Builder

  28. def streamName(streamName: String): Builder

    Sets the name of the Kinesis stream that the DStream will read from.

    Sets the name of the Kinesis stream that the DStream will read from. This is a required parameter.

    streamName

    Name of Kinesis stream that the DStream will read from

    returns

    Reference to this KinesisInputDStream.Builder

  29. def streamingContext(jssc: JavaStreamingContext): Builder

    Sets the StreamingContext that will be used to construct the Kinesis DStream.

    Sets the StreamingContext that will be used to construct the Kinesis DStream. This is a required parameter.

    jssc

    JavaStreamingContext used to construct Kinesis DStreams

    returns

    Reference to this KinesisInputDStream.Builder

  30. def streamingContext(ssc: StreamingContext): Builder

    Sets the StreamingContext that will be used to construct the Kinesis DStream.

    Sets the StreamingContext that will be used to construct the Kinesis DStream. This is a required parameter.

    ssc

    StreamingContext used to construct Kinesis DStreams

    returns

    Reference to this KinesisInputDStream.Builder

  31. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  32. def toString(): String
    Definition Classes
    AnyRef → Any
  33. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  34. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  35. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Deprecated Value Members

  1. def initialPositionInStream(initialPosition: InitialPositionInStream): Builder

    Sets the initial position data is read from in the Kinesis stream.

    Sets the initial position data is read from in the Kinesis stream. Defaults to InitialPositionInStream.LATEST if no custom value is specified. This function would be removed when we deprecate the KinesisUtils.

    initialPosition

    InitialPositionInStream value specifying where Spark Streaming will start reading records in the Kinesis stream from

    returns

    Reference to this KinesisInputDStream.Builder

    Annotations
    @deprecated
    Deprecated

    (Since version 2.3.0) use initialPosition(initialPosition: KinesisInitialPosition)

Inherited from AnyRef

Inherited from Any

Ungrouped