Hey Chad, I don't so it wasn't ACLs. So just to give you an update. As I said before, we decided to restart brokers. It took a while until they came back in normal shape, but as soon as they did I took a loook at the kafka-streams applications and it rebalancing exception has disappeared.
Thanks Chad for your help. It looked like manual reassignment of 2 partitions somehow confused producers.. Have a good day guys, Paweł On Thu, Jul 4, 2019 at 4:54 PM Chad Preisler <chad.preis...@gmail.com> wrote: > I assume Kafka brokers are on a separate server from the stream apps. > > Are you using ACLs? Did they change recently? Maybe an internal topic can’t > be written. > > Is one of the brokers out of disk space? > > Any local state on the stream side? Maybe clean that up? > > Is the replication factor on the consumer group correct? > > I did have an issue with timeouts where I needed to increase > request.timeout.ms on the stream app. If you try that increase > delivery.timeout.ms so that it is larger than request.timeout.ms and make > transaction.timeout.ms larger than delivery.timout.ms. > > Good luck. > > Chad > > On Thu, Jul 4, 2019 at 9:28 AM Paweł Gontarz <pgont...@powerspace.com> > wrote: > > > Thanks Chad, > > > > Unfortunately that's not our case > > > > On Thu, Jul 4, 2019 at 4:19 PM Chad Preisler <chad.preis...@gmail.com> > > wrote: > > > > > I had a similar situation. For us one of our firewall appliances was > > > blocking traffic to the brokers. > > > > > > On Thu, Jul 4, 2019 at 7:43 AM Paweł Gontarz <pgont...@powerspace.com> > > > wrote: > > > > > > > Hey all, > > > > > > > > I have seen already in archive an email concerning this, but as a > > > solution > > > > it has been said to upgrade kafka version to 2.1. In my case, kafka > is > > > > already up to date. > > > > > > > > NOTE: Issue is on since this morning. > > > > Specifying the problem, I'm running two kafka-streams stateful > > > > applications. From the very beginning of the app lifecycle, instances > > > > struggle to reassign correctly partitions between them which > eventually > > > > leads them to > > > > > > > > org.apache.kafka.streams.errors.StreamsException: stream-thread > > > > > [pws-budget-streams-client-mapper-StreamThread-13] Failed to > > rebalance. > > > > > > > > > > > > Due to > > > > > > > > Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout > > > expired > > > > > while initializing transactional state in 60000ms. > > > > > > > > > > > > In the same time I'm observing a big lag on 2 partitions of the topic > > > which > > > > my streams are consuming. > > > > The issue had started just this morning, whereas applications are for > > > > already 1 month running without issues. > > > > > > > > One thing I did before it, was the reassignment of this two > partitions > > to > > > > different nodes. Why? To fight over CPU consumption on one of our > > brokers > > > > (it wasn't balanced evenly). > > > > > > > > I have no clue if it has anything to do with problems on > kafka-streams, > > > > though. > > > > > > > > Anyone encountered similar problems? > > > > > > > > Cheers, > > > > Paweł > > > > > > > > > >
