Hi,
This is first time I am trying structured streaming with Kafka. I have
simple code to read from Kafka and display it on the console. Message is in
JSON format. However, when I run my code nothin after below line gets
printed.
17/07/21 13:43:41 INFO AppInfoParser: Kafka commitId : a7a17cdec9eaa
Hi,
I am reading data from Kafka into spark. It runs fine for sometime but
then hangs forever with following output. I don't see and errors in logs.
How do I debug this?
2015-12-01 06:04:30,697 [dag-scheduler-event-loop] INFO (Logging.scala:59)
- Adding task set 19.0 with 4 tasks
2015-12-01 06:0
Hi,
Has anyone used Protobuff with spark-cassandra connector? I am using
protobuff-3.0-beta with spark-1.4 and cassandra-connector-2.10. I keep
getting "Unable to find proto buffer class" in my code. I checked version
of protobuff jar and it is loaded with 3.0-beta in classpath. Protobuff is
comin
r messages through
> the GSLB to that endpoint, rather than talking to Kafka directly.
>
> -Todd
>
>
>
> On Tue, Nov 3, 2015 at 10:15 AM, Cassa L wrote:
>
> > Hi,
> > Has anyone used load balancers between publishers and Kafka brokers? I
> > want to d
Hi,
Has anyone used load balancers between publishers and Kafka brokers? I
want to do active-passive setup of Kafka in two datacenters. My question
is can I add GSLB layer between these two Kafka clusters to configure
automatic fail over while publishing data?
Thanks,
LCassa
a/src/main/scala/org/apache/spark/streaming/kafka/KafkaRDD.scala
>
> On Fri, Aug 28, 2015 at 11:32 AM, Cassa L wrote:
>
> > Hi I am using below Spark jars with Direct Stream API.
> > spark-streaming-kafka_2.10
> >
> > When I look at its pom.xml, Kafka librarie
o its not supported.
>
> Thanks,
> Harsha
>
>
> On August 28, 2015 at 11:00:30 AM, Cassa L (lcas...@gmail.com) wrote:
>
> Hi,
> I was going through SSL setup of Kafka.
> https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka
> However, I am also using Spa
Hi,
I was going through SSL setup of Kafka.
https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka
However, I am also using Spark-Kafka streaming to read data from Kafka. Is
there a way to activate SSL for spark streaming API or not possible at all?
Thanks,
LCassa
it.
>
> Thanks,
> Harsha
>
>
> On August 18, 2015 at 10:17:15 AM, Cassa L (lcas...@gmail.com) wrote:
>
> Hi,
> Has anyone used patch provided in below JIRA to setup SSL for Kafka?
> https://issues.apache.org/jira/browse/KAFKA-1690
>
> I see there are many pathc
Hi,
Has anyone used patch provided in below JIRA to setup SSL for Kafka?
https://issues.apache.org/jira/browse/KAFKA-1690
I see there are many pathces. which one is stable? If anyone has
instructions on how o use it, it will help.
LCassa.
s to 1.4 seconds.
> >
> >
> > On 8/6/15, 4:35 PM, "Cassa L" wrote:
> >
> > >Hi Grant,
> > >Yes, I saw exception in Spark and Kafka. In Kafka server logs I get this
> > >exception:
> > >kafka.common.OffsetOutOfRangeException: Request
h...@hortonworks.com> wrote:
>
> > retention.ms is actually millisecond, you want a value much larger then
> > 1440, which translates to 1.4 seconds.
> >
> >
> > On 8/6/15, 4:35 PM, "Cassa L" wrote:
> >
> > >Hi Grant,
> > >Yes, I saw e
e/SPARK-8474
>
> What versions of Spark and Kafka are you using? Can you include more of the
> spark log? Any errors shown in the Kafka log?
>
> Thanks,
> Grant
>
> On Thu, Aug 6, 2015 at 1:17 PM, Cassa L wrote:
>
> > Hi,
> > Has anyone tried streaming API of
Hi,
Has anyone tried streaming API of Spark with Kafka? I am experimenting new
Spark API to read from Kafka.
KafkaUtils.createDirectStream(...)
Every now and then, I get following error "spark
kafka.common.OffsetOutOfRangeException" and my spark script stops working.
I have simple topic with just
Hi,
I want to know what usually, are the JAVA_HEAP settings recommended for
kafka servers in production.
Thanks
LCassa
Thanks. that helps.
LCassa
On Wed, Apr 16, 2014 at 3:39 PM, Joel Koshy wrote:
> Does this help:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIchoosethenumberofpartitionsforatopic
> ?
>
>
> On Wed, Apr 16, 2014 at 03:34:28PM -0700, Cassa L w
Hi,
I am sizing Kafka setup for the first time. One question I have is, how
to determine number of partitions my topic will need. Are there any
considerations to be taken in order to determine this number. I am planning
to have a three node Kafka cluster.
LCassa.
.server":name="PurgatorySize",type="FetchRequestPurgatory"
>
> Thanks,
>
> Jun
>
> On Mon, Mar 24, 2014 at 12:38 PM, Cassa L wrote:
>
> > Hi,
> > We have been doing some evaluation testing against Kafka.We have 48GB
> RAM
> > on
Hi,
We have been doing some evaluation testing against Kafka.We have 48GB RAM
on each broker. I created 3 broker cluster and one zookeeper and sent
10,000Messages/seconds to this cluster, continuously. Payload is very
small, less than Kilobyte. Consumer was readind 5,000Messages per second.
Wha
face with them directly (that's the purview of the
> network operations team).
>
> -Todd
>
> On 3/12/14 1:47 PM, "Cassa L" wrote:
>
> >Hi Todd,
> > I am not very clear about you mentioned - "Both the producer and
> >consumer
> >clien
ad
> balancer for this because it's the most robust solution. Only the metadata
> request itself will go through the load balancer. Once the producer has
> the metadata for the topic and partition, it reconnects directly to the
> leader for the partition for producing.
>
>
Hi,
Is there documentation on how connection management, connection pooling
works in Kafka for multiple brokers? Do I need to take care of my own
connection management for cluster? e.g. while publishing messages, if one
broker stops responding, I need to switch to the other. Does Kafka client
take
er"
>
> Details of the producer configs can be found:
>
> http://kafka.apache.org/documentation.html#producerconfigs
>
>
> On Wed, Jan 22, 2014 at 4:51 PM, Cassa L wrote:
>
> > Hi all,
> > I am a new user of Kafka. I tried Java example of it and it work
Hi all,
I am a new user of Kafka. I tried Java example of it and it worked great.
Now when I tried to write to a Producer (Java) and send ByteArray to it as
a data, I get ClassCastException as below
java.lang.ClassCastException: java.lang.String cannot be cast to [B
at kafka.serializer.Defa
24 matches
Mail list logo
