Kerberos Integration Issue

Hi, I have implement SASL Kerberos on my HDP cluster kafka. Kerberos is working fine. When I am starting producer console I am repetedly getting below messages: [data-ee-dev1@ bin]$ ./kafka-console-consumer.sh --zookeeper --topic kafka-kerberos-test1 --security-protocol PL

Root disk latancy

Hi We are faced with latency on root disks of our Kafka instances. For example, at one time we had average ios the latency of about 190 ms with 34 ios in the disk queue during few seconds. There were only write ios. Looks like kafka process was hanging. It decreased its CPU utilization to 20 times

Process messages from StateStore

Hi! I have an application which uses an input and output topic, and every message from the input topic should have a corresponding message (with the same key) in the output topic. To detect lost messages (=no output after a certain amount of time, ~10days) I tried to use a KTable - KTable left jo

NPE in low level Kafka topology

Hi, I've upgraded our 0.11 based stream application to the trunk version, and I get an intermittent NPE. It's is quite a big topology, and I haven't succeeded in reproducing it on a simpler topology. It builds the topology, starts Kafka Streams, runs for about 20s., and then it terminates It seems

Multiple consumers subscribing to a topic

Hi I am looking for the below setup in one of the Applications I am working on: Application generates events and publishes the events to a Kafka topic. The topic is subscribed by multiple consumers. Assuming there are 10 events triggered in the application and published to the topic. How do we

Re: NPE in low level Kafka topology

Can you show the related code from OneToManyGroupedProcessor ? Thanks On Mon, Jun 18, 2018 at 4:29 AM, Frank Lyaruu wrote: > Hi, I've upgraded our 0.11 based stream application to the trunk version, > and I get an intermittent NPE. It's is quite a big topology, and I haven't > succeeded in repr

Avro binary format compression benefit

Hi I would like to know if anyone has seen any benefit in specifying a compression.type for Kafka Producer sending avro binary (not JSON) encoded messages to Kafka cluster. I can appreciate the compression benefits for JSON but is there really any benefit in compressing binary avro data? Kafka

Error topic

Hi two questions - Does it make sense to have an error topic for “retryable” errors if the retries parameter is already set to like 10 (default)? For non retryable errors, is the approach to try to send to the topic, catch the specific exception and send to another topic (just send the whole me

Re: NPE in low level Kafka topology

Yes, here it is: https://gist.github.com/flyaruu/84b65d52a01987946b07d21bafe0d5f1 It ran completely fine for the last year (and still does), it just does not seem to enjoy the upgrade of Kafka Streams. regards, Frank On Mon, Jun 18, 2018 at 4:49 PM Ted Yu wrote: > Can you show the related cod

Quota management hack queries

Hi Couple of questions on Quota management Is my understanding correct that quota is enforced by delaying the ack to the Producer? 1. What is the impact then if the Producer has set ack to none – Is Quota still enforceable? 2. If the Producer has not set the clientId, would the quota b

Re: Multiple consumers subscribing to a topic

Hi Nitin, 1) A Kafka Consumer uses a poll loop to pull messages from the topics it is subscribed to. You can see examples of how this can be implemented in Java here: https://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0-9-consumer-client/ 2,3) Consumers normally c

Migration Path from 0.8

Hello, First time writing in the mailing list! At this moment we are running 3 Kafka (0.8) broker cluster. We want to try to upgrade our cluster to Kafka 1.0.  Is there any recommendation for migrating from 0.8 ? Would it be possible to move directly from 0.8 to 1.0 or we should go through all

Aw: Migration Path from 0.8

Hi Martin, even though with Kafka I as a fellow user never found it problematic to just give upgrades a shot no matter the version increase in a test environment, since you are still on 0.8 you should thoroughly study the details at this link and depending on whether you care about one or more

Aw: Migration Path from 0.8

Sven, Thanks for the quick reply! If that is the case I’ll try to move straight to 1.0 in our staging environment with one broker and see how it goes! Thanks again! Best, Martín On Mon, Jun 18, 2018 at 5:55 PM Sven Ludwig < Sven Ludwig ( Sven Ludwig ) > wrote: > > > > Hi Martin, > > e

Re: Migration Path from 0.8

From: Martín Fernández Sent: Monday, June 18, 2018 4:38 PM To: users@kafka.apache.org Subject: Migration Path from 0.8 Hello, First time writing in the mailing list! At this moment we are running 3 Kafka (0.8) broker cluster. We want to try to upgrade our clu

Re: NPE in low level Kafka topology

Hello Frank, Your OneToManyGroupedProcessor.java looks fine to me. Is it consistently re-producible? What if you restart from fresh using the trunk version? Guozhang On Mon, Jun 18, 2018 at 9:03 AM, Frank Lyaruu wrote: > Yes, here it is: > > https://gist.github.com/flyaruu/84b65d52a01987946b