Partitioning per team

2020-10-26 Thread Jan Bols
For a kafka-streams application, we keep data per team. Data from 2 teams never meet but within a team, data is highly integrated. A team has team members but also has several types of equipment. A team has a lifespan of about 1-3 days after which the team is removed and all data relating to that t

Topic partition reassignment fails because of non-monotonic offsets (Kafka 2.6)

2020-10-26 Thread Joe Ammann
We did an upgrade from Kafka 2.2 to 2.6, followed by a migration (through reassign-partitions) from old to new brokers. As described in https://stackoverflow.com/questions/64514851/apache-kafka-kafka-common-offsetsoutoforderexception-when-reassigning-consume, all but 1 partition (__consumer_offset