Hi,
Is it possible to setup streams from multiple Kinesis streams and process
them in a single job? From what I have read, this should be possible,
however, the Kinesis layer errors out whenever I try to receive from more
than a single Kinesis Stream.
Here is the code. Currently, I am focused on just getting receivers setup
and working for the two Kinesis Streams, as such, this code just attempts to
print out the contents of both streams:
implicit val formats = Serialization.formats(NoTypeHints)
val conf = new SparkConf().setMaster("local[*]").setAppName("test")
val ssc = new StreamingContext(conf, Seconds(1))
val rawStream = KinesisUtils.createStream(ssc, "erich-test",
"kinesis.us-east-1.amazonaws.com", Duration(1000),
InitialPositionInStream.TRIM_HORIZON, StorageLevel.MEMORY_ONLY)
rawStream.map(msg => new String(msg)).print
val loaderStream = KinesisUtils.createStream(
ssc,
"dev-loader",
"kinesis.us-east-1.amazonaws.com",
Duration(1000),
InitialPositionInStream.TRIM_HORIZON,
StorageLevel.MEMORY_ONLY)
val loader = loaderStream.map(msg => new String(msg)).print
ssc.start()
Thanks,
-Erich
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Multiple-Kinesis-Streams-in-a-single-Streaming-job-tp22889.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
