>
>
> Thanks for bringing this up - it is definitely an important point to
> discuss. The underlying issue of KAFKA-1017 was uncovered to some degree by
> the fact that in our deployment we did not significantly increase the total
> number of partitions over 0.7 - i.e., in 0.7 we had say four partitions per
> broker, now we are using (say) eight partitions across the cluster. So with
> random partitioning every producer would end up connecting to nearly every
> broker (unlike 0.7 in which we would connect to only one broker within each
> reconnect interval). In a production-scale deployment that causes the high
> number of connections that KAFKA-1017 addresses.
>
> You are right that the fix of sticking to one partition over the metadata
> refresh interval goes against true consumer parallelism, but this would be
> the case only if there are few producers. If you have a sizable number of
> producers on average all partitions would get uniform volumes of data.
>
> One tweak to KAFKA-1017 that I think is reasonable would be instead of
> sticking to a random partition, stick to a random broker and send to random
> partitions within that broker. This would make the behavior closer to 0.7
> wrt number of connections and random partitioning provided the number of
> partitions per broker is high enough, which is why I mentioned the
> partition count (in our usage) in 0.7 vs 0.8 above. Thoughts?
>
> Joel
>
>
> On Friday, September 13, 2013, Joe Stein wrote:
> >
> > First, let me apologize for not realizing/noticing this until today. One
> > reason I left my last company was not being paid to work on Kafka nor
> being
> able to afford any time for a while to work on it. Now in my new gig (just
> wrapped up my first week, woo hoo) while I am still not "paid to work on
> Kafka" I can afford some more time for it now and maybe in 6 months I will
> be able to hire folks to work on Kafka (with more and more time for myself
> to work on it too) while we also work on client projects (especially Kafka
> based ones).
>
> So, I understand about the changes that were made to fix open file handles
> and make the random pinning be timed based (with a very large default
> time). Got all that.
>
> But, doesn't this completely negate what has been communicated to the
> community for a very long time and the expectation they have? I think it
> does.
>
> The expected functionality for random partitioning is that "This can be
> done in a round-robin fashion simply to balance load" and that the
> "producer" does it for you.
>
> Isn't a primary use case for partitions to paralyze consumers? If so then
> the expectation would be that all consumers would be getting in parallel
> equally in a "round robin fashion" the data that was produced for the
> topic... simply to balance load...with the producer handling it and with
> the client application not having to-do anything. This randomness occurring
> every 10 minutes can't balance load.
>
> If users are going to work around this anyways (as I would honestly do too)
> doing a pseudo semantic random key and essentially forcing real randomness
> to simply balance load to my consumers running in parallel would we still
> end up hitting the KAFKA-1017 problem anyways? If not then why can't we
> just give users the functionality and put back the 3 lines of code 1)
> if(key == null) 2) random.nextInt(numPartitions) 3) else ... If we would
> bump into KAFKA-1017 by working around it then we have not really solved
> the root cause problem and removing expected functionality for a corner
> case that might have other work arounds and/or code changes to solve it
> another way or am I still not getting something?
>
> Also, I was looking at testRandomPartitioner in AsyncProducerTest and I
> don't see how this would ever fail, the assertion is always for partitionId
> == 0 and it should be checking that data is going to different partitions
> for a topic, right?
>
> Let me know, I think this is an important discussion and even if it ends up
> as telling the community to only use one partition that is all you need and
> partitions become our super columns (Apache Cassandra joke, its funny) then
> we manage and support it and that is just how it is but if partitions are a
> good thing and having multiple consumers scale in parrelel for a single
> topic also good then we have to manage and support that.
>
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
>
