Hi TD, This is a common pattern that is emerging today. Kafka --> SS --> Kafka.
Spark Streaming comes with a built in consumer to read from Kafka. It will be great to have an easy way for users to write back to Kafka without having to code a customer producer using the Kafka Producert APIs. Are there any plans to commit the code in the above github repo? If so, do you have a rough estimate of when. Thanks, Anand On Fri, Nov 7, 2014 at 1:25 PM, Tathagata Das <tathagata.das1...@gmail.com> wrote: > I am not aware of any obvious existing pattern that does exactly this. > Generally this sort of computation (subset, denormalization) things are so > generic sounding terms but actually have very specific requirements that it > hard to refer to a design pattern without more requirement info. > > If you want to feed back to kafka, you can take a look at this pull request > > https://github.com/apache/spark/pull/2994 > > On Thu, Nov 6, 2014 at 4:15 PM, bdev <buntu...@gmail.com> wrote: > >> We are looking at consuming the kafka stream using Spark Streaming and >> transform into various subsets like applying some transformation or >> de-normalizing some fields, etc. and feed it back into Kafka as a >> different >> topic for downstream consumers. >> >> Wanted to know if there are any existing patterns for achieving this. >> >> Thanks! >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Any-patterns-for-multiplexing-the-streaming-data-tp18303.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >
