ct: Re: Spark Streaming + Kafka + scala job message read issue
Hi Bryan,
Yes we are using only 1 thread per topic as we have only one Kafka server with
1 partition.
What kind of logs will tell us what offset spark stream is reading from Kafka
or is it resetting something without reading?
Regards
Meghanathan (WT01 - NEP)
Sent: 27 December 2015 11:08
To: Bryan
Cc: Vivek Meghanathan (WT01 - NEP) ;
duc.was.h...@gmail.com; user@spark.apache.org
Subject: Re: Spark Streaming + Kafka + scala job message read issue
Hi Bryan,
Yes we are using only 1 thread per topic as we have only one Kafka
.@gmail.com>
Cc: duc.was.h...@gmail.com<mailto:duc.was.h...@gmail.com>;
vivek.meghanat...@wipro.com<mailto:vivek.meghanat...@wipro.com>;
user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: Spark Streaming + Kafka + scala job message read issue
Hi Brian,PhuDuc,
All 8 jobs
...@gmail.com
Cc: duc.was.h...@gmail.com; vivek.meghanat...@wipro.com; user@spark.apache.org
Subject: Re: Spark Streaming + Kafka + scala job message read issue
Hi Brian,PhuDuc,
All 8 jobs are consuming 8 different IN topics. 8 different Scala jobs running
each topic map mentioned below has only 1 thread
; for
Windows 10 phone
From: PhuDuc Nguyen<mailto:duc.was.h...@gmail.com>
Sent: Friday, December 25, 2015 3:35 PM
To: vivek.meghanat...@wipro.com<mailto:vivek.meghanat...@wipro.com>
Cc: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: Spark Streaming + Kafka + scala
Agreed. I did not see that they were using the same group name.
Sent from Outlook Mail for Windows 10 phone
From: PhuDuc Nguyen
Sent: Friday, December 25, 2015 3:35 PM
To: vivek.meghanat...@wipro.com
Cc: user@spark.apache.org
Subject: Re: Spark Streaming + Kafka + scala job message read issue
>
>
>
>
>
> Regards,
> Vivek M
>
> *From:* Bryan [mailto:bryan.jeff...@gmail.com]
> *Sent:* 24 December 2015 17:20
> *To:* Vivek Meghanathan (WT01 - NEP) ;
> user@spark.apache.org
> *Subject:* RE: Spark Streaming + Kafka + scala job message read issue
>
>
>
...@wipro.com
Sent: Friday, December 25, 2015 2:18 PM
To: bryan.jeff...@gmail.com; user@spark.apache.org
Subject: Re: Spark Streaming + Kafka + scala job message read issue
Any help is highly appreciated, i am completely stuck here..
From: Vivek Meghanathan (WT01 - NEP)
Sent: Thursday, December 24
Any help is highly appreciated, i am completely stuck here..
From: Vivek Meghanathan (WT01 - NEP)
Sent: Thursday, December 24, 2015 7:50 PM
To: Bryan; user@spark.apache.org
Subject: RE: Spark Streaming + Kafka + scala job message read issue
We are using the
anathan (WT01 - NEP) ;
user@spark.apache.org
Subject: RE: Spark Streaming + Kafka + scala job message read issue
Are you using a direct stream consumer, or the older receiver based consumer?
If the latter, do the number of partitions you’ve specified for your topic
match the number of partitions
while missing data from other partitions.
Regards,
Bryan Jeffrey
Sent from Outlook Mail for Windows 10 phone
From: vivek.meghanat...@wipro.com
Sent: Thursday, December 24, 2015 5:22 AM
To: user@spark.apache.org
Subject: Spark Streaming + Kafka + scala job message read issue
Hi All,
We are
Hi All,
We are using Bitnami Kafka 0.8.2 + spark 1.5.2 in Google cloud platform. Our
spark streaming job(consumer) not receiving all the messages sent to the
specific topic. It receives 1 out of ~50 messages(added log in the job stream
and identified). We are not seeing any errors in the kaf
12 matches
Mail list logo
