Potential message incompatibility between versions 0.11 and 2.2.1 of Kafka?

2020-04-10 Thread Malcolm McFarland
Hey folks, We're using Samza 0.14.1 for a project, with Kafka 0.11.0.2 as a client. We're also using Amazon's MSK for our clusters, which is currently set to Kafka broker version 2.2.1. Occasionally, when starting Samza containers, we'll see the beginnings of an attempt to read Samza's checkpoint

Re: [kafka-clients] Re: [VOTE] 2.5.0 RC3

2020-04-10 Thread Israel Ekpo
+1 (non-binding) Used the following environment in my validation of the release artifacts: Ubuntu 18.04, OpenJDK 11, Scala 2.13.1, Gradle 5.6.2 Verified GPG Signatures for all release artifacts Verified md5 sha1 sha512 checksums for each artifact Checked Scala and Java Docs Ran Unit and Integrati

Re: Kafka Streams endless rebalancing

2020-04-10 Thread John Roesler
Hey Alex, Huh. Unprefixed configs apply to all consumers, but in this case, it's irrelevant because only the "main" consumer participates in group management (so the config only applies to the main consumer). So you actually have max.poll.interval.ms set to Integer.MAX_VALUE, which amounts to 25

Re: Kafka Streams endless rebalancing

2020-04-10 Thread Alex Craig
Thanks John, I double-checked my configs and I've actually got the max.poll.interval.ms set to the max (not prefixed with anything so presumably that’s the “main” consumer). So I think that means the problem isn’t due to a single batch of messages not getting processed/committed within the polling

Re: Global state store: Lazy loading

2020-04-10 Thread Matthias J. Sax
Global state store are designed to load data upfront from a topics. I don't think you can "bend" it easily. You can keep the topic empty before starting the application, and put data into that topic on demand, and the store would pick it up afterwards. Not sure if this would be feasible for your u

Re: [kafka-clients] Re: [VOTE] 2.5.0 RC3

2020-04-10 Thread Manikumar
Hi David, +1 (binding) - Verified signatures, artifacts, Release notes - Built from sources, ran tests - Ran core/streams quick start for Scala 2.13 binary, ran few manual tests - Verified docs As discussed offline, we need to add upgrade instructions to 2.5 docs. Thanks for driving the releas

Re: How to set custom properties and message expiration for Kafka topic message.

2020-04-10 Thread Nicolas Carlot
Ok, then you may be looking for custom headers: https://cwiki.apache.org/confluence/display/KAFKA/KIP-82+-+Add+Record+Headers Le ven. 10 avr. 2020 à 16:08, Naveen Kumar M a écrit : > Hello Nicolas, > > Thanks a lot for the response. "Custom properties" are nothing but > additional message prope

Re: [kafka-clients] Re: [VOTE] 2.5.0 RC3

2020-04-10 Thread Bill Bejeck
Hi David, +1 (non-binding) Verified signatures, built jars from source, ran unit and integration tests, all passed. Thanks for running the release. -Bill On Wed, Apr 8, 2020 at 10:10 AM David Arthur wrote: > Passing Jenkins build on 2.5 branch: > https://builds.apache.org/job/kafka-2.5-jdk8/9

Re: How to set custom properties and message expiration for Kafka topic message.

2020-04-10 Thread Naveen Kumar M
Hello Nicolas, Thanks a lot for the response. "Custom properties" are nothing but additional message properties based on our requirement and which are not part of message payload. For example trackingId which I need to add this additional property while producing message and read the same in consu

Re: How to set custom properties and message expiration for Kafka topic message.

2020-04-10 Thread Nicolas Carlot
Hello, Message expiration is based on topic configuration, not on your producer configuration. Look at kafka server configuration. retention.ms is one of the properties that drive the message deletion I don't know what is "customer properties", if you're talking about "custom properties", what do

Re: How to set custom properties and message expiration for Kafka topic message.

2020-04-10 Thread Naveen Kumar M
Hello Team, Could you please help me on this? Thanks and regards, Naveen On Tue, Apr 7, 2020, 2:27 PM Naveen Kumar M wrote: > Hello Friends, > > Hope you are all doing good! > > I am bit new to Kafka messaging system. > > We are using Kafka as messaging broker. > > Could someone let me know ho

Re: how to start DEBUG level

2020-04-10 Thread Jacek Szewczyk
Please change log4j.logger.kafka=DEBUG, kafkaAppender And it will work. Jacek > On Apr 10, 2020, at 07:51, deadwind4 wrote: > > Hi > > > I wanna change kafka log to DEBUG level > > > I change option log4j.rootLogger=DEBUG, stdout, kafkaAppender in > log4j.properties file > > > but in