Curious, if you have 1:1 mapping between Stream1:topic1 and
Stream2:topic2 then why not run different instances of the app for
each and pass as arguments to each instance the input source and
output topic?
On Thu, Sep 18, 2014 at 8:07 AM, Padmanabhan, Mahesh (contractor)
<mahesh.padmanab...@twc-contractor.com> wrote:
> Hi all,
>
> I am using Spark 1.0 streaming to ingest a a high volume stream of data
> (approx. 1mm lines every few seconds) transform it into two outputs and send
> those outputs to two separate Apache Kafka topics. I have two blocks of
> output code like this:
>
> Stream1 =
> ….
>
> Stream2 =
>
> …
>
> Stream1.foreachRDD {
> // emit to Kafka topic 1
> }
> Stream2.foreachRDD {
> // emit to Kafka topic 2
> }
>
> …
>
> My problem is that I never see debug statements in the Stream2.foreachRDD
> code nor is the topic 2 consumer seeing anything. I am wondering if this is
> because these statements are serialized and Spark never gets out of Stream1
> output code before it is ready to process the next Stream1 batch thereby
> never entering the Stream2 output code.
>
> Do I have to parallelize these two output code blocks or am I missing
> something more fundamental?
>
> Thanks,
> Mahesh
>
> ________________________________
> This E-mail and any of its attachments may contain Time Warner Cable
> proprietary information, which is privileged, confidential, or subject to
> copyright belonging to Time Warner Cable. This E-mail is intended solely for
> the use of the individual or entity to which it is addressed. If you are not
> the intended recipient of this E-mail, you are hereby notified that any
> dissemination, distribution, copying, or action taken in relation to the
> contents of and attachments to this E-mail is strictly prohibited and may be
> unlawful. If you have received this E-mail in error, please notify the
> sender immediately and permanently delete the original and any copy of this
> E-mail and any printout.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
