Hi John, Zipping up some logs from our running Kafka cluster is going to be a bit difficult. What I can do is try to reproduce this off-line and capture the logs from there.
We also had a look in the PartitionAssignor source code (for 1.1.1) and indeed this behaviour is a bit weird as from the source code I'd expect equally divided partitions. Anyway, hopefully I'll be able to reproduce this issue with some simple unit-test like code. I'll post the results when I have more info. Greets, Bart On Mon, Oct 8, 2018 at 7:36 PM John Roesler <j...@confluent.io> wrote: > Hi Bart, > > This sounds a bit surprising. Is there any chance you can zip up some logs > so we can see the assignment protocol on the nodes? > > Thanks, > -John > > On Mon, Oct 8, 2018 at 4:32 AM Bart Vercammen <b...@cloutrix.com> wrote: > > > Hi, > > > > I recently moved some KafkaStreams applications from v0.10.2.1 to v1.1.1 > > and now I notice a weird behaviour in the partition assignment. > > When starting 4 instances of my Kafka Streams application (on v1.1.1) I > see > > that 17 of the 20 partitions (of a source topic) are assigned to 1 > instance > > of the application while the other 3 instances only get 1 partition > > assigned. (previously (on v0.10.2.1) the all got 5 partitions.) > > > > Is this expected behaviour, as I read that quite some improvements were > > done in the partition assignment strategy for Kafka Streams applications? > > If yes, how can I make it so that the partitions are equally devided > again > > across all running applications? It's a bit weird in my opinion as this > > makes scaling the application very hard. > > > > Also, when initially starting with 1 instance of the application, and > > gradually scaling up, the new instances only get 1 partition assigned ... > > > > All my Streams applications use default configuration (more or less), > > running 1 stream-thread. > > > > Any suggestions / enlightenments on this? > > Greets, > > Bart > > > -- Mvg, Bart Vercammen clouTrix BVBA +32 486 69 17 68 i...@cloutrix.com
