How about creating a new class called RandomRefreshPartioner and copy the DefaultPartitioner code to it and then revert the DefaultPartitioner code. I appreciate this is a one time burden for folks using the existing 0.8-beta1 bumping into KAFKA-1017 in production having to switch to the RandomRefreshPartioner and when folks deploy to production will have to consider this property change.
I make this suggestion keeping in mind the new folks that on board with Kafka and when everyone is in development and testing mode for the first time their experience would be as expected from how it would work in production this way. In dev/test when first using Kafka they won't have so many producers for partitions but would look to parallelize their consumers IMHO. The random broker change sounds like maybe a bigger change now this late in the release cycle if we can accommodate folks trying Kafka for the first time and through their development and testing along with full blown production deploys. /******************************************* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop ********************************************/ On Sep 14, 2013, at 8:17 AM, Joel Koshy <jjkosh...@gmail.com> wrote: >> >> >> Thanks for bringing this up - it is definitely an important point to >> discuss. The underlying issue of KAFKA-1017 was uncovered to some degree by >> the fact that in our deployment we did not significantly increase the total >> number of partitions over 0.7 - i.e., in 0.7 we had say four partitions per >> broker, now we are using (say) eight partitions across the cluster. So with >> random partitioning every producer would end up connecting to nearly every >> broker (unlike 0.7 in which we would connect to only one broker within each >> reconnect interval). In a production-scale deployment that causes the high >> number of connections that KAFKA-1017 addresses. >> >> You are right that the fix of sticking to one partition over the metadata >> refresh interval goes against true consumer parallelism, but this would be >> the case only if there are few producers. If you have a sizable number of >> producers on average all partitions would get uniform volumes of data. >> >> One tweak to KAFKA-1017 that I think is reasonable would be instead of >> sticking to a random partition, stick to a random broker and send to random >> partitions within that broker. This would make the behavior closer to 0.7 >> wrt number of connections and random partitioning provided the number of >> partitions per broker is high enough, which is why I mentioned the >> partition count (in our usage) in 0.7 vs 0.8 above. Thoughts? >> >> Joel >> >> >> On Friday, September 13, 2013, Joe Stein wrote: >>> >>> First, let me apologize for not realizing/noticing this until today. One >>> reason I left my last company was not being paid to work on Kafka nor >> being >> able to afford any time for a while to work on it. Now in my new gig (just >> wrapped up my first week, woo hoo) while I am still not "paid to work on >> Kafka" I can afford some more time for it now and maybe in 6 months I will >> be able to hire folks to work on Kafka (with more and more time for myself >> to work on it too) while we also work on client projects (especially Kafka >> based ones). >> >> So, I understand about the changes that were made to fix open file handles >> and make the random pinning be timed based (with a very large default >> time). Got all that. >> >> But, doesn't this completely negate what has been communicated to the >> community for a very long time and the expectation they have? I think it >> does. >> >> The expected functionality for random partitioning is that "This can be >> done in a round-robin fashion simply to balance load" and that the >> "producer" does it for you. >> >> Isn't a primary use case for partitions to paralyze consumers? If so then >> the expectation would be that all consumers would be getting in parallel >> equally in a "round robin fashion" the data that was produced for the >> topic... simply to balance load...with the producer handling it and with >> the client application not having to-do anything. This randomness occurring >> every 10 minutes can't balance load. >> >> If users are going to work around this anyways (as I would honestly do too) >> doing a pseudo semantic random key and essentially forcing real randomness >> to simply balance load to my consumers running in parallel would we still >> end up hitting the KAFKA-1017 problem anyways? If not then why can't we >> just give users the functionality and put back the 3 lines of code 1) >> if(key == null) 2) random.nextInt(numPartitions) 3) else ... If we would >> bump into KAFKA-1017 by working around it then we have not really solved >> the root cause problem and removing expected functionality for a corner >> case that might have other work arounds and/or code changes to solve it >> another way or am I still not getting something? >> >> Also, I was looking at testRandomPartitioner in AsyncProducerTest and I >> don't see how this would ever fail, the assertion is always for partitionId >> == 0 and it should be checking that data is going to different partitions >> for a topic, right? >> >> Let me know, I think this is an important discussion and even if it ends up >> as telling the community to only use one partition that is all you need and >> partitions become our super columns (Apache Cassandra joke, its funny) then >> we manage and support it and that is just how it is but if partitions are a >> good thing and having multiple consumers scale in parrelel for a single >> topic also good then we have to manage and support that. >> >> /******************************************* >> Joe Stein >> Founder, Principal Consultant >> Big Data Open Source Security LLC >> http://www.stealth.ly >> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> >> ********************************************/ >>
