org.apache.spark.examples.streaming
Class KinesisWordCountASL

Object
  org.apache.spark.examples.streaming.KinesisWordCountASL

All Implemented Interfaces:

Logging

public class KinesisWordCountASL
extends Object
implements Logging

Consumes messages from a Amazon Kinesis streams and does wordcount.

This example spins up 1 Kinesis Receiver per shard for the given stream. It then starts pulling from the last checkpointed sequence number of the given stream.

Usage: KinesisWordCountASL is the name of the consumer app, used to track the read data in DynamoDB name of the Kinesis stream (ie. mySparkStream) endpoint of the Kinesis service (e.g. https://kinesis.us-east-1.amazonaws.com)

Example: # export AWS keys if necessary $ export AWS_ACCESS_KEY_ID= $ export AWS_SECRET_KEY=

# run the example $ SPARK_HOME/bin/run-example streaming.KinesisWordCountASL myAppName mySparkStream \ https://kinesis.us-east-1.amazonaws.com

There is a companion helper class called KinesisWordProducerASL which puts dummy data onto the Kinesis stream.

This code uses the DefaultAWSCredentialsProviderChain to find credentials in the following order: Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY Java System Properties - aws.accessKeyId and aws.secretKey Credential profiles file - default location (~/.aws/credentials) shared by all AWS SDKs Instance profile credentials - delivered through the Amazon EC2 metadata service For more information, see http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/credentials.html

See http://spark.apache.org/docs/latest/streaming-kinesis-integration.html for more details on the Kinesis Spark Streaming integration.

Constructor Summary

KinesisWordCountASL()

Method Summary

static void

main(String[] args)

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

KinesisWordCountASL

public KinesisWordCountASL()

Method Detail

main

public static void main(String[] args)