I assume Kafka brokers are on a separate server from the stream apps. Are you using ACLs? Did they change recently? Maybe an internal topic can’t be written.
Is one of the brokers out of disk space? Any local state on the stream side? Maybe clean that up? Is the replication factor on the consumer group correct? I did have an issue with timeouts where I needed to increase request.timeout.ms on the stream app. If you try that increase delivery.timeout.ms so that it is larger than request.timeout.ms and make transaction.timeout.ms larger than delivery.timout.ms. Good luck. Chad On Thu, Jul 4, 2019 at 9:28 AM Paweł Gontarz <pgont...@powerspace.com> wrote: > Thanks Chad, > > Unfortunately that's not our case > > On Thu, Jul 4, 2019 at 4:19 PM Chad Preisler <chad.preis...@gmail.com> > wrote: > > > I had a similar situation. For us one of our firewall appliances was > > blocking traffic to the brokers. > > > > On Thu, Jul 4, 2019 at 7:43 AM Paweł Gontarz <pgont...@powerspace.com> > > wrote: > > > > > Hey all, > > > > > > I have seen already in archive an email concerning this, but as a > > solution > > > it has been said to upgrade kafka version to 2.1. In my case, kafka is > > > already up to date. > > > > > > NOTE: Issue is on since this morning. > > > Specifying the problem, I'm running two kafka-streams stateful > > > applications. From the very beginning of the app lifecycle, instances > > > struggle to reassign correctly partitions between them which eventually > > > leads them to > > > > > > org.apache.kafka.streams.errors.StreamsException: stream-thread > > > > [pws-budget-streams-client-mapper-StreamThread-13] Failed to > rebalance. > > > > > > > > > Due to > > > > > > Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout > > expired > > > > while initializing transactional state in 60000ms. > > > > > > > > > In the same time I'm observing a big lag on 2 partitions of the topic > > which > > > my streams are consuming. > > > The issue had started just this morning, whereas applications are for > > > already 1 month running without issues. > > > > > > One thing I did before it, was the reassignment of this two partitions > to > > > different nodes. Why? To fight over CPU consumption on one of our > brokers > > > (it wasn't balanced evenly). > > > > > > I have no clue if it has anything to do with problems on kafka-streams, > > > though. > > > > > > Anyone encountered similar problems? > > > > > > Cheers, > > > Paweł > > > > > >
