Packages

class Builder extends AnyRef

Builder for KinesisInputDStream instances.

Source
KinesisInputDStream.scala
Since

2.2.0

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Builder
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new Builder()

Value Members

  1. def build(): KinesisInputDStream[Array[Byte]]

    Create a new instance of KinesisInputDStream with configured parameters and using the default message handler, which returns Array[Byte].

    Create a new instance of KinesisInputDStream with configured parameters and using the default message handler, which returns Array[Byte].

    returns

    Instance of KinesisInputDStream constructed with configured parameters

  2. def buildWithMessageHandler[T](handler: (Record) ⇒ T)(implicit arg0: ClassTag[T]): KinesisInputDStream[T]

    Create a new instance of KinesisInputDStream with configured parameters and the provided message handler.

    Create a new instance of KinesisInputDStream with configured parameters and the provided message handler.

    handler

    Function converting Record instances read by the KCL to DStream type T

    returns

    Instance of KinesisInputDStream constructed with configured parameters

  3. def checkpointAppName(appName: String): Builder

    Sets the KCL application name to use when checkpointing state to DynamoDB.

    Sets the KCL application name to use when checkpointing state to DynamoDB. This is a required parameter.

    appName

    Value to use for the KCL app name (used when creating the DynamoDB checkpoint table and when writing metrics to CloudWatch)

    returns

    Reference to this KinesisInputDStream.Builder

  4. def checkpointInterval(interval: Duration): Builder

    Sets how often the KCL application state is checkpointed to DynamoDB.

    Sets how often the KCL application state is checkpointed to DynamoDB. Defaults to the Spark Streaming batch interval if no custom value is specified.

    interval

    Duration specifying how often the KCL state should be checkpointed to DynamoDB.

    returns

    Reference to this KinesisInputDStream.Builder

  5. def cloudWatchCredentials(credentials: SparkAWSCredentials): Builder

    Sets the SparkAWSCredentials to use for authenticating to the AWS CloudWatch endpoint.

    Sets the SparkAWSCredentials to use for authenticating to the AWS CloudWatch endpoint. Will use the same credentials used for AWS Kinesis if no custom value is set.

    credentials

    SparkAWSCredentials to use for CloudWatch authentication

    returns

    Reference to this KinesisInputDStream.Builder

  6. def dynamoDBCredentials(credentials: SparkAWSCredentials): Builder

    Sets the SparkAWSCredentials to use for authenticating to the AWS DynamoDB endpoint.

    Sets the SparkAWSCredentials to use for authenticating to the AWS DynamoDB endpoint. Will use the same credentials used for AWS Kinesis if no custom value is set.

    credentials

    SparkAWSCredentials to use for DynamoDB authentication

    returns

    Reference to this KinesisInputDStream.Builder

  7. def endpointUrl(url: String): Builder

    Sets the AWS Kinesis endpoint URL.

    Sets the AWS Kinesis endpoint URL. Defaults to "https://kinesis.us-east-1.amazonaws.com" if no custom value is specified

    url

    Kinesis endpoint URL to use

    returns

    Reference to this KinesisInputDStream.Builder

  8. def initialPosition(initialPosition: KinesisInitialPosition): Builder

    Sets the initial position data is read from in the Kinesis stream.

    Sets the initial position data is read from in the Kinesis stream. Defaults to KinesisInitialPositions.Latest if no custom value is specified.

    initialPosition

    KinesisInitialPosition value specifying where Spark Streaming will start reading records in the Kinesis stream from

    returns

    Reference to this KinesisInputDStream.Builder

  9. def kinesisCredentials(credentials: SparkAWSCredentials): Builder

    Sets the SparkAWSCredentials to use for authenticating to the AWS Kinesis endpoint.

    Sets the SparkAWSCredentials to use for authenticating to the AWS Kinesis endpoint. Defaults to DefaultCredentialsProvider if no custom value is specified.

    credentials

    SparkAWSCredentials to use for Kinesis authentication

    returns

    Reference to this KinesisInputDStream.Builder

  10. def metricsEnabledDimensions(metricsEnabledDimensions: Set[String]): Builder

    Sets the enabled CloudWatch metrics dimensions.

    Sets the enabled CloudWatch metrics dimensions. Defaults to KinesisClientLibConfiguration.DEFAULT_METRICS_ENABLED_DIMENSIONS if no custom value is specified.

    metricsEnabledDimensions

    Set[String] to specify which CloudWatch metrics dimensions should be enabled

    returns

    Reference to this KinesisInputDStream.Builder

    See also

    https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-kcl.html#metric-levels

  11. def metricsLevel(metricsLevel: MetricsLevel): Builder

    Sets the CloudWatch metrics level.

    Sets the CloudWatch metrics level. Defaults to KinesisClientLibConfiguration.DEFAULT_METRICS_LEVEL if no custom value is specified.

    metricsLevel

    MetricsLevel to specify the CloudWatch metrics level

    returns

    Reference to this KinesisInputDStream.Builder

    See also

    https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-kcl.html#metric-levels

  12. def regionName(regionName: String): Builder

    Sets the AWS region to construct clients for.

    Sets the AWS region to construct clients for. Defaults to "us-east-1" if no custom value is specified.

    regionName

    Name of AWS region to use (e.g. "us-west-2")

    returns

    Reference to this KinesisInputDStream.Builder

  13. def storageLevel(storageLevel: StorageLevel): Builder

    Sets the storage level of the blocks for the DStream created.

    Sets the storage level of the blocks for the DStream created. Defaults to StorageLevel.MEMORY_AND_DISK_2 if no custom value is specified.

    storageLevel

    StorageLevel to use for the DStream data blocks

    returns

    Reference to this KinesisInputDStream.Builder

  14. def streamName(streamName: String): Builder

    Sets the name of the Kinesis stream that the DStream will read from.

    Sets the name of the Kinesis stream that the DStream will read from. This is a required parameter.

    streamName

    Name of Kinesis stream that the DStream will read from

    returns

    Reference to this KinesisInputDStream.Builder

  15. def streamingContext(jssc: JavaStreamingContext): Builder

    Sets the StreamingContext that will be used to construct the Kinesis DStream.

    Sets the StreamingContext that will be used to construct the Kinesis DStream. This is a required parameter.

    jssc

    JavaStreamingContext used to construct Kinesis DStreams

    returns

    Reference to this KinesisInputDStream.Builder

  16. def streamingContext(ssc: StreamingContext): Builder

    Sets the StreamingContext that will be used to construct the Kinesis DStream.

    Sets the StreamingContext that will be used to construct the Kinesis DStream. This is a required parameter.

    ssc

    StreamingContext used to construct Kinesis DStreams

    returns

    Reference to this KinesisInputDStream.Builder

Deprecated Value Members

  1. def initialPositionInStream(initialPosition: InitialPositionInStream): Builder

    Sets the initial position data is read from in the Kinesis stream.

    Sets the initial position data is read from in the Kinesis stream. Defaults to InitialPositionInStream.LATEST if no custom value is specified. This function would be removed when we deprecate the KinesisUtils.

    initialPosition

    InitialPositionInStream value specifying where Spark Streaming will start reading records in the Kinesis stream from

    returns

    Reference to this KinesisInputDStream.Builder

    Annotations
    @deprecated
    Deprecated

    (Since version 2.3.0) use initialPosition(initialPosition: KinesisInitialPosition)