(cont'd) meant to say mod%partition count of System.currentTimeMillis(). Having said that - is there any disadvantage to true random distribution of traffic for a topic?
On Tue, Jan 24, 2017 at 11:17 AM, Jon Yeargers <jon.yearg...@cedexis.com> wrote: > It may be picking a random partition but it sticks with it indefinitely > despite there being a significant disparity in traffic. I need to break it > up in some different fashion. Maybe just a hash of > System.currentTimeMillis()? > > > > On Tue, Jan 24, 2017 at 10:52 AM, Avi Flax <avi.f...@parkassist.com> > wrote: > >> >> > On Jan 24, 2017, at 11:18, Jon Yeargers <jon.yearg...@cedexis.com> >> wrote: >> > >> > If I don't specify a key when I call send a value to kafka (something >> akin >> > to 'kafkaProducer.send(new ProducerRecord<>(TOPIC_PRODUCE, >> jsonView))') how >> > is it keyed? >> >> IIRC, in this case the key is null; i.e. there is no key. >> >> > I am producing to a topic from an external feed. It appears to be >> heavily >> > biased towards certain values and as a result I have 2-3 partitions that >> > are lagging heavily where the rest are staying current. >> >> Hmm, according to the docs this shouldn’t matter: >> >> > If the key is null, then a random broker partition is picked. >> >> https://kafka.apache.org/documentation/#impl_producer >> >> You might want to double-check your code and confirm that it is indeed >> sending no keys… i.e. maybe it’s actually using an empty string as a key, >> or something like that. >> >> > Since I don't use >> > the keys in my consumers Im wondering if I could randomize these values >> > somehow to better distribute the load. >> >> As per the above docs, this _should_ already be the case, based on what >> you’ve described. >> >> That said, if you continue to have trouble, then you can introduce your >> own implementation of kafka.producer.Partitioner, and again as per the docs: >> >> > A custom partitioning strategy can also be plugged in using the >> partitioner.class config parameter. >> >> Also, it so happens that I have implemented a custom random partitioning >> strategy through an alternate approach by using the overloaded >> ProducerRecord constructor that accepts a partition ID. You can easily get >> the set of partition IDs from the Producer with the partitionsFor method. >> >> HTH! >> Avi >> >> ———— >> Software Architect @ Park Assist » http://tech.parkassist.com/ > > >
