Hi Vijay,
PyKafka support custom partitioners so that you can control how the
producer distributes messages. Looking at it now, I realize the
documentation around this is a bit lacking, so I'll add that. In the
meantime, here's an example of how to use the hashing partitioner which is
included with PyKafka.
client = pykafka.KafkaClient()
# Producer with random_partitioner
prod = client.topics['foo'].get_producer()
prod.produce(['foo', 'bar', 'baz'])
# Producer with hashing_partitioner
prod =
client.topics['foo'].get_producer(partitioner=pykafka.partitioners.hashing_partitioner)
prod.produce([('p_key1', 'foo'), ('p_key2', 'foo'), ('p_key1', 'baz')])
Cheers,
Keith
On Wed, Jun 10, 2015 at 6:35 AM Vijay Patil <vijay2110.t...@gmail.com>
wrote:
> Thanks,
> a question, does it support keyed Producer?
> I mean how it works when I have multiple partitions? Will it be able to
> identify partition based of the key which I pass?
>
> On 9 June 2015 at 00:54, Keith Bourgoin <keith.bourg...@gmail.com> wrote:
>
> > Hi Kafka folks,
> >
> > I'm happy to announce the 1.0 release of PyKafka
> > <https://github.com/parsely/pykafka>, a new Python driver for Kafka!
> >
> > This is an upgrade of an older library, named samsa, which was only
> > compatible with Kafka 0.7.x. We've spent the last few months upgrading it
> > to work with Kafka 0.8.2.1 and revamping the codebase to be cleaner and
> > more efficient.
> >
> > The biggest difference between PyKafka and other Python drivers is the
> > inclusion of a balanced consumer implementation. We implemented the same
> > balancing algorithm as the Java driver and PyKafka handles consumer
> > registration via Zookeeper like the Java driver.
> >
> > We at Parse.ly have been using this driver in production for a few weeks
> > and are confident in its stability. Even so, we're looking to add more
> > features over the next couple months and are always trying to find ways
> to
> > improve performance.
> >
> > You can find the project on PyPI <https://pypi.python.org/pypi/pykafka/
> >,
> > Github <https://github.com/parsely/pykafka> and our mailing list
> > <https://groups.google.com/forum/#%21forum/pykafka-user>. We also wrote
> a
> > blog
> > post
> > <
> >
> http://blog.parsely.com/post/2006/announcing-pykafka-python-support-for-kafka-0-8-x/
> > >
> > with a bit more info.
> >
> > Cheers,
> >
> > Keith.
> >
>
