Re: KafkaProducer - Oversized batches

2019-12-08 Thread Tomoyuki Saito
Hi Guozhang, Thank you for your suggestion! I'll create a JIRA within a few days and consider submitting a PR. Thanks, Tomoyuki On 2019/12/09 00:37:49, Guozhang Wang wrote: > Hello Tomoyuki,> > > It seems that issue in 6494 is indeed valid, and I'd personally suggest we> > do option 3) to fix t

Re: Static Membership AND Invalid IP addresses forConsumers

2019-12-08 Thread Guozhang Wang
Hello David, The host information of the group membership is inferred from the socket channel's ip address, I'm not certain how EC2 spot instance's application.server config is used, but if it is not dynamically reflected on the socket channel's inet address then it would be the case you observed.

Re: KafkaProducer - Oversized batches

2019-12-08 Thread Guozhang Wang
Hello Tomoyuki, It seems that issue in 6494 is indeed valid, and I'd personally suggest we do option 3) to fix the flush() behavior. Please feel free to create a JIRA (and also submit your PR if you are interested in contributing :). Guozhang On Sat, Dec 7, 2019 at 7:59 AM Tomoyuki Saito wrote

Re: Kafka transactions not working in 2.3.1

2019-12-08 Thread Soman Ullah
Hello Jonathan, I believe I did not set *log.message.format.version* to the current version before I upgraded to 2.3. However, I did set inter.broker.protocol.version though. Would reverting back to 0.10.1 and upgrading again with *log.message.format.version *work? Thanks, Soman On Sun, Dec 8, 20

Re: How are custom keys compared during joins

2019-12-08 Thread Guozhang Wang
Hello Sachin, Thanks for the detailed description. Your find is right that for stream-table join the table-side updates would not trigger a join since stream records are not "materialized" or buffered during the processing. The community has requested similar semantics to improve as table-table j

Re: Kafka consumer group keeps moving to PreparingRebalance and stops consuming

2019-12-08 Thread Guozhang Wang
Hello Avshalom, I think the first question to answer is where are the new consumers coming from. From your description they seem to be not expected (i.e. you did not intentionally start up new instances), so looking at those VMs that suddenly start new consumers would be my first shot. Guozhang

Re: What timestamp is used by streams when doing windowed joins

2019-12-08 Thread John Roesler
Ah, I didn’t remember that the docs defined the terms that way. Those definitions make sense to me. Yes, if your topics are configured with LogAppendTime, then when we poll records from the topic, the timestamp that comes back attached to the record would be the log append time. If you’re usi

Re: Kafka consumer group keeps moving to PreparingRebalance and stops consuming

2019-12-08 Thread Jamie
Hi Avshalom,  Have you tried increasing the session timeout? What's the current session timeout? Regarding the max.poll.interval.ms - this is the maximum time between calls to poll of the consumer, are there any possible scenarios where the processing of one lot of messages from the consumer (ma

Re: Reducing streams startup bandwidth usage

2019-12-08 Thread Alessandro Tagliapietra
It seems that even with caching enabled, after a while the sent bytes stil go up [image: Screen Shot 2019-12-08 at 12.52.31 PM.png] you can see the deploy when I've enabled caching but it looks like it's still a temporary solution. -- Alessandro Tagliapietra On Sat, Dec 7, 2019 at 10:08 AM Ale

Re: Kafka transactions not working in 2.3.1

2019-12-08 Thread Jonathan Santilli
Hard to tell without logs and more context, I mean, Is a big jump from 0.10.1 to 2.3.1. Did you follow all the instructions/comments? as in the upgrade documentation https://kafka.apache.org/documentation/#upgrade_2_3_0? Cheers! -- Jonathan On Sun, Dec 8, 2019 at 3:18 PM Soman Ullah wrote: >

Re: Kafka transactions not working in 2.3.1

2019-12-08 Thread Soman Ullah
Hello Jonathan, I've installed the same version on all the brokers. The brokers were restarted as well. Any suggestions on how I can fix this? Thanks, Soman On Sat, Dec 7, 2019 at 9:17 AM Jonathan Santilli wrote: > Hello, > > have you ensured you have installed the same version in all brokers?

Re: Kafka consumer group keeps moving to PreparingRebalance and stops consuming

2019-12-08 Thread Avshalom Manevich
Hi Boyang, Thanks for your reply. We looked into this direction, but since we didn't change max.poll.interval from its default value, we're not sure if it's the case. On Fri, 6 Dec 2019 at 17:42, Boyang Chen wrote: > Hey Avshalom, > > the consumer instance is initiated per stream thread. You w