Re: Improving Kafka State Store performance

Hi Bill, I will repeat my tests with Rocks DB enabled and I will revert to you with details. I might take 1-2 days to get back to you with details since I am traveling. But I will try my level best to get it tonight. On Mon, Sep 18, 2017 at 5:30 PM, Bill Bejeck wrote: > I'm following up from y

Re: Kafka Streams application Unable to Horizontally scale and the application on other instances refusing to start.

Thank you Bill, Apologies for late reply . On Mon, Sep 18, 2017 at 5:45 PM, Bill Bejeck wrote: > Hi, > > I just sent you a follow-up message on the other thread we have going > regarding state store performance. > > I guess we can consider this thread closed and we'll continue working on > the S

Re: Kafka SSL error

Another point .. on adding the chain.p12 (PKCS#12 format) .. order of the certs in the keystore is as shown below Alias : 1 Cert[1] -> server cert Cert[2] -> Intermediate cert Cert[3] -> Root cert mentioning that, since one of the articles i read mentioned that the certs imported should be in fol

Re: Kafka SSL error

Hi Jakub, Thanks for the detailed note... here is the update -> I was able to convert the host.cert.pem to PKCS#12 & import the cert into the kafka.server.keystore.jks (also into kafka.server.truststore.jks) wrt the host.root.pem & host.intermed.pem certs - i'm assuming i need to convert them to

is a topic compressed?

I have a producer configured to snappy compress data sent to a cluster. Is there some way to verify that the data indeed is being compressed? If I peek at the .log files on the broker, I can read some plain text amongst binary. Similarly, tcpdump shows plain text readable data. I do not know if thi

Re: Metrics: committed offset, client version

KIP-188 is expected to be in the upcoming 1.0.0 release. It will add client-side JMX metrics that show the client version number. https://cwiki.apache.org/confluence/display/KAFKA/KIP-188+-+Add+new+metrics+to+support+health+checks

Re: Kafka Internals Video/Blog

This recent meetup had a presentation of the internals of the Kafka Controller. https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/242656767/ The video is not yet available, but hopefully will be soon. -J

Re: Kafka SSL error

ad 1) The problem is that the signed certificate (host.cert.pem) which the CA provides is only the public key. You have to combine it with the private key which you created when requesting the signed certificate. The private key is never sent to the CA so they cannot provide it back. You or whoever

Can't reset consumer group offsets in Kafka 0.11.0.1

Hi there, The Kafka servers in question were 0.11.0.0 this morning. I upgraded to 0.11.0.1 after discovering this issue but I don’t see any change. Scala 2.11. A client using the new-style consumer groups is not consuming any new messages because its current committed offset is out of range. I sh

Can't reset consumer group offsets in Kafka 0.11.0.1

Hi there, The Kafka servers in question were 0.11.0.0 this morning. I upgraded to 0.11.0.1 after discovering this issue but I don’t see any change. Scala 2.11. A client using the new-style consumer groups is not consuming any new messages because its current committed offset is out of range. I sh

Re: Kafka SSL error

Hello - thanks for the response Here is the update on the issue. I'm using certs signed/provided by org-wide CA (geotrust, not a self-signed cert) The Signed(by the CA - geotrust) cert provided has 3 certificates - host.chain.pem (certificate chain - contains the Root, Intermediate, Signed Server

Re: Kafka Internals Video/Blog

Check out the Kafka wiki: https://cwiki.apache.org/confluence/display/KAFKA/Index It contains many design and discussion pages -- it's sometime a little hidden, but I am sure you can find them. -Matthias On 9/20/17 10:18 AM, M. Manna wrote: > Raghav, > > I would say Kafka documentation on Kafk

Re: Data loss while upgrading confluent 3.0.0 kafka cluster to confluent 3.2.2

One clarification below: On Wed, Sep 20, 2017 at 3:50 PM, Ismael Juma wrote: > Comments inline. > > On Wed, Sep 20, 2017 at 11:56 AM, Yogesh Sangvikar < > yogesh.sangvi...@gmail.com> wrote: > >> 2. At which point in the sequence below was the code for the brokers >> updated to 0.10.2? >> >> [Com

Re: Kafka Internals Video/Blog

Raghav, I would say Kafka documentation on Kafka.apache.org and confluent.io are the best ones that I found for myself. They both talk about some code for you to start with. I do recommend that you fork out Kafka from GitHub if you wanna go as far as debugging the code to see a flow of execution-

Re: Kafka Internals Video/Blog

Kafka has been evolving rapidly. Is there area(s) which you're particularly interested ? You can find all the relevant KIPs under: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals Normally KIPs contain fair amount of technical details. >From each KIP, you can find J

Re: Kafka Internals Video/Blog

Thanks Manna. I did watch that earlier. I am looking for something more detailed in terms of implementation and design details. I want to dabble with the code but given the complexity of the code, a good starter guide would be helpful. Thanks. On Wed, Sep 20, 2017 at 9:53 AM, M. Manna wrote:

Re: Kafka Internals Video/Blog

There's a video where Jay Kreps talks about how Kafka works - YouTube has it as the top 5 under "How Kafka Works". On 20 Sep 2017 5:49 pm, "Raghav" wrote: > Hi > > Just wondering if there is any video/blog that goes over Kafka Internal and > under the hood design and implementation details. I a

Kafka Internals Video/Blog

Hi Just wondering if there is any video/blog that goes over Kafka Internal and under the hood design and implementation details. I am a newbie and I would like to dabble with the code and understand design of it. Just wondering if there is any video, blog etc that goes over it ? Thanks. -- Ragh

Re: Spark Executor - jaas.conf with useTicketCache=true

Hi, Apologies, but bumping this email. Is there any way we can use use ticketcache with Kafka? Thanks, Hugo On Tue, Sep 19, 2017 at 2:45 PM, Hugo Reinwald wrote: > Hi All, > > I am connecting to a secured kafka cluster from spark. My jaas.conf looks > like below - > KafkaClient { > com.sun.secu

Re: Data loss while upgrading confluent 3.0.0 kafka cluster to confluent 3.2.2

Comments inline. On Wed, Sep 20, 2017 at 11:56 AM, Yogesh Sangvikar < yogesh.sangvi...@gmail.com> wrote: > 2. At which point in the sequence below was the code for the brokers > updated to 0.10.2? > > [Comment: On the kafka servers, we have confluent-3.0.0 and confluent-3.2.2 > packages deployed

Re: Data loss while upgrading confluent 3.0.0 kafka cluster to confluent 3.2.2

Hi Ismael, A few questions: 1. Please share the code for the test script. [Comment: We are publishing events using kafka-rest *POST: /topics/* API. And, using Jmeter script to call the API to publish events continuously for 2 hrs. The "key" value for the event is constant so that, we can ch

Re: 0.10.2.2 bug fix release

Hi Mikhail, Is there a reason why you can't upgrade to 0.11.0.1 instead? Thanks, Ismael On Wed, Sep 20, 2017 at 8:19 AM, Mikhail Lesyk wrote: > As I see zero issues to be fixed here: > https://issues.apache.org/jira/projects/KAFKA/versions/12340570 > and considering 0.11.0.1 was released recen

Metrics: committed offset, client version

Hi! I am wondering if there are broker/client metrics for: - client version (to keep track of clients that needs an upgrade) - committed offsets (to detect situations when commits fail systematically with everything else being ok) Thank you, Stanislav.

Re: Question about Kafka

I'm running this command: "/usr/local/kafka/bin/kafka-producer-perf-test.sh --topic words --num-records 2 --record-size 100 --throughput 1 --producer-props acks=-1 bootstrap.servers=:9092,:9092,1:9092 buffer.memory=67108864 compression.type=none batch.size=8196" a

Re: Kafka SSL error

Hi, Looking at your commands it looks as if you generated a self signed key for server, self signed key for client and then imported the CA keys public keys into the truststores. I don’t think this will work because now you have two different self signed keys in the keystores and the presumably th

Re: Question about Kafka

Producer configuration? On Wed, Sep 20, 2017, 2:50 PM MAHA ALSAYASNEH < maha.alsayas...@univ-grenoble-alpes.fr> wrote: > Hello, > > Any suggestion regarding this msg: > " org.apache.kafka.common.errors.TimeoutException: Expiring 61 record(s) > for due to 30001 ms has passed since batch creat

0.10.2.2 bug fix release

As I see zero issues to be fixed here: https://issues.apache.org/jira/projects/KAFKA/versions/12340570 and considering 0.11.0.1 was released recently, Are there any estimates regarding release 0.10.2.2 be out? It has very important bug fixes.