Thank you! We are new to Kafka, so this makes complete sense - the metadata refresh age.
Yes, incrementally assigning 1 partitioner key - we are tracking it in a relational DB along with offset, etc, for the consumers. We haven¹t yet implemented the dynamic consumer side but there are several approaches from the brute/force quick/dirty (e.g. Poll DB at interval for updates), to using another Kafka topic for new partition event notifications (of course, we favor the latter, but timelines will likely mean that the first option is what initially gets implemented). On 8/7/15, 10:56 AM, "Grant Henke" <ghe...@cloudera.com> wrote: >Interesting use case. I would be interested to hear more. Are you >assigning >1 partition per key incrementally? How does your consumer know which >partition has which key? > >I don't think there is a way to manually invalidate the cached metadata in >the public producer api (I could be wrong), but the longest you should >have >to wait is whatever is configured for metadata.max.age.ms. > >metadata.max.age.ms is the period of time in milliseconds after which we >force a refresh of metadata even if we haven't seen any partition >leadership changes to proactively discover any new brokers or partitions. >http://kafka.apache.org/documentation.html#newproducerconfigs > > > > >On Fri, Aug 7, 2015 at 12:34 PM, Gelinas, Chiara <cgeli...@illumina.com> >wrote: > >> Hi All, >> >> We are looking to dynamically create partitions when we see a new piece >>of >> data that requires logical partitioning (so partitioning from a logical >> perspective rather than partitioning solely for load-based reasons). I >> noticed that when I create a partition via AdminUtils.addPartition, and >> then send the message (within milliseconds since it¹s all happening on >>one >> thread execution), I get the following error: >> >> Invalid partition given with record: 3 is not in the range [0...3]. >> >> Basically, the Producer can¹t see the new partition. When I set a >> breakpoint just before the send (which essentially sleeps the thread), >>then >> all is well, and it pushes the message to the new partition with no >>issues. >> >> I am running just one zookeeper, one kafka (no replicas) this is all >> local on my dev environment. >> >> Is this normal behavior or is there possibly some issue with how we are >> using addPartition? Also, once we have replicas in a more realistic >> production environment, should we expect this lag to increase? >> >> The only workaround I can envision for this is to have the thread check >> the partition count via AdminUtils and only move on when the partition >> count comes back as expected. >> >> Thanks, >> Chiara >> >> >> > > >-- >Grant Henke >Software Engineer | Cloudera >gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
