Re: KeeperException

2019-05-14 Thread Kamal Chandraprakash
Those are not errors. See the logs are logged in INFO mode. https://stackoverflow.com/a/48067058/3209010 On Wed, May 15, 2019 at 10:44 AM Gagan Sabharwal wrote: > Hi team, > > Any pointers on the same ? > > Regards > Gagan > > On Mon, May 13, 2019 at 11:24 AM Gagan Sabharwal > wrote: > > > Hi

Re: KeeperException

2019-05-14 Thread Gagan Sabharwal
Hi team, Any pointers on the same ? Regards Gagan On Mon, May 13, 2019 at 11:24 AM Gagan Sabharwal wrote: > Hi team, > > I have just started to learn kafka following > https://kafka.apache.org/quickstart > > When I have started my zookeeper server and kafka server, I got a few > exceptions in

optimal message size in Kafka

2019-05-14 Thread Raghav
Hi Is there any study that shows why smaller size messages are optimal for Kafka, and as size increases to 1MB and more, the throughput decreases ? What are the design choices in Kafka that leads to this behavior. Can any developer or committer share any insight and point to the relevant piece o

Re: Kafka Connect - HDFS or FileStream

2019-05-14 Thread Vinay Jain
Redirecting to a file will not work, we would not be able to create a different filenames after some time or filesize. HDFS connect sink already has those options , also i could use some minor transformations while using the connect On Tue, May 14, 2019 at 7:28 AM Hans Jespersen wrote: > Can you

Logging and monitoring for compliance reasons the client id, topic, partition(s), offsets, and consumer group (or other relevant info) for client reads

2019-05-14 Thread Wade Chandler
We are using Apache Kafka as a long term event store for a Fully Eventually Consistent Event Sourced Architecture. We also have various compliance requirements. We certainly have encrypted volumes and connections (at rest and transit); we are setup for TLS authentication; do not have authorizati

Re: Evergrowing __consumer_offsets topic

2019-05-14 Thread Michael Jaschob
Claudia, This sounds like an issue I and other users have reported recently: https://issues.apache.org/jira/browse/KAFKA-8335. Looks like a fix was merged yesterday. Michael On Tue, May 14, 2019 at 4:44 AM Claudia Wegmann wrote: > Dear kafka users, > > I run a kafka cluster (version 2.1.1) wit

Evergrowing __consumer_offsets topic

2019-05-14 Thread Claudia Wegmann
Dear kafka users, I run a kafka cluster (version 2.1.1) with 6 brokers to process ~100 messages per second with a number of kafka streams apps. There are currently 53 topics with 30 partitions each. I have exactly once processing enabled. My problem is that the __consumer_offsets topic is growi

Re: Customers getting duplicate emails

2019-05-14 Thread ASHOK MACHERLA
Dear John To overcome this issue what parameters I have to add in consumer side?? Please tell me if settings required I'll put it in that .. Thanks.. Sent from Outlook

Consumer fails with SchemaException (kafka 0.9.0.0)

2019-05-14 Thread Ilya Karpov
Hi Kafka users! I have a legacy kafka cluster that hosts a topic of click stream events. Topic is consumed by 10 instances of an app with the same consumer group. Sometimes I see exception is logs of those consumers like this: org.apache.kafka.common.protocol.types.SchemaException: Error readin