In the new producer we are changing the default behavior back to pure random partitioning and let users to customize their own partitioning schemes if they want. For now reducing topic.metadata.refresh.interval.ms should help because the stickiness only persists until a metadata refresh.
Guozhang On Mon, Jun 9, 2014 at 4:54 PM, Prakash Gowri Shankor < prakash.shan...@gmail.com> wrote: > Is there a way to modify this duration ? This is not adhering to the > "random" behavior that the documentation talks about. > > > On Mon, Jun 9, 2014 at 4:41 PM, Kane Kane <kane.ist...@gmail.com> wrote: > > > Last time I've checked it, producer sticks to partition for 10 minutes. > > > > On Mon, Jun 9, 2014 at 4:13 PM, Prakash Gowri Shankor > > <prakash.shan...@gmail.com> wrote: > > > Hi, > > > > > > This is with 0.8.1.1 and I ran the command line console consumer. > > > I have one broker, one producer and several consumers. I have one > topic, > > > many partitions m, many consumers n, m=n , one consumer group defined > for > > > all the consumers > > > > > > From using Kafka Monitor, I see that each partition is assigned to one > > > consumer now. However, it seems that there is no parallelism in data > > > consumption. What I see happening is that one consumer gets messages > from > > > time t0 to t1 from partition P1. Then another consumer gets messages > from > > > t1 to t2 from partition P2 and so on. > > > > > > *Why is there no parallel consumption happening ?* It looks to me that > > the > > > producer's data goes into P1 from t0 to t1 and then from t1 to t2 into > > P2. > > > I thought that if I dont specify a partitioning key, the producer's > data > > > will get partitioned randomly. It's just that the randomness seems to > be > > > "delayed". Why is this so ? > > > > > > I tried setting topic.metadata.refresh.interval.ms=100 in the > > > producer.properties. > > > > > > That did not seem to change this strange partitioning behavior. > > > > > > Please help. > > > > > > Thanks > > > -- -- Guozhang
