I’m not the maintainer of either pykafka or librdkafka, so I can’t realllLyyyy comment much on the benefit, but you may be right. However, librdkafka is well maintained and solid, so using it as the backing for a Python client gets you the benefit of not having to reinvent features yourself in Python.
New kafka-python stuff looks cool, I’m excited to try it out for producing soon. On Mon, Jan 11, 2016 at 12:37 PM, Dana Powers <dana.pow...@gmail.com> wrote: > Agree - kafka-python was in hibernation waiting for 0.9.0.0 Kafka release, > so a few issues lingered longer than I would have liked. Most of my > comments relate to latest master, which we are hoping to release after a > bit more testing and polish. > > Re librdkafka -- to be honest, I'm skeptical that C protocol bindings are > going to improve python performance much. In my experience, the devil is in > the client logic details, not the wire protocol parsing. Adding a C > compilation step also adds installation and operational overhead (install > gcc or manage linux wheels). So we have avoided adding that to kafka-python > without significant evidence showing performance benefits that can't be > duplicated in pure python. > > -Dana > On Jan 11, 2016 9:02 AM, "Sam Pegler" <sam.peg...@infectiousmedia.com> > wrote: > > > kafka-python (https://github.com/dpkp/kafka-python) has also just merged > > performance improvements to the consumer in > > https://github.com/dpkp/kafka-python/issues/290 which should see a > pretty > > decent boost in throughput. We were somewhat put off by the poor > > performance in earlier versions, I imagine many people would have been in > > the same position so it's worth revisiting. > > > > Sam Pegler > > > > WEBOPS ENGINEER T. +44(0) 07 562 867 486 [image: Infectious Media]3-7 > > Herbal Hill / London / EC1R 5EJwww.infectiousmedia.com [image: > Infectious > > Media] <http://www.infectiousmedia.com/>[image: Facebook] > > <http://www.facebook.com/infectiousmedia>[image: Twitter] > > <https://twitter.com/infectiousmedia>[image: LinkedIn] > > <http://www.linkedin.com/company/infectious-media-ltd>[image: Youtube] > > <http://www.youtube.com/user/InfectiousMediaLtd> This email and any > > attachments are confidential and may also be privileged. If youare not > the > > intended recipient, please notify the sender immediately, and do > > notdisclose > > the contents to another person, use it for any purpose, or store, or > > copythe > > information in any medium. Please also destroy and delete the message > > fromyour > > computer. > > > > On 11 January 2016 at 16:28, Andrew Otto <o...@wikimedia.org> wrote: > > > > > pykafka’s balanced consumer is very useful. pykafka also has Python > > > bindings to the librdkafka C library that you can optionally enable, > > which > > > might get you some speed boosts. > > > > > > python-kafka (oh, I just saw this 0.9x version, hm!) was better at > > > producing than pykafka for us, so we am currently using pykafka for > > > consumption, and python-kafka for production. python-kafka allows you > to > > > produce to multiple topics using the same client instance. (pykafka > may > > > support this soon: https://github.com/Parsely/pykafka/issues/354) > > > > > > > > > > > > On Sat, Jan 9, 2016 at 10:04 AM, Dana Powers <dana.pow...@gmail.com> > > > wrote: > > > > > > > pykafka uses a custom zookeeper implementation for consumer groups. > > > > kafka-python uses the 0.9.0.0 server apis to accomplish the same. > > > > > > > > -Dana > > > > On Jan 8, 2016 18:32, "chengxin Cai" <ia...@outlook.com> wrote: > > > > > > > > > Hi > > > > > > > > > > I heard that Pykakfa can create a balanced consumer. > > > > > > > > > > And there should be no other big difference. > > > > > > > > > > > > > > > Best Regards > > > > > > > > > > > 在 2016年1月9日，08:58，Dana Powers <dana.pow...@rd.io> 写道： > > > > > > > > > > > > Hi Doug, > > > > > > > > > > > > The differences are fairly subtle. kafka-python is a > > community-backed > > > > > > project that aims to be consistent w/ the official java client; > > > pykafka > > > > > is > > > > > > sponsored by parse.ly and aims to provide a pythonic interface. > > > > > whichever > > > > > > you go with, I would love to hear your specific feedback on > > > > kafka-python. > > > > > > > > > > > > -Dana (kafka-python maintainer) > > > > > > > > > > > >> On Fri, Jan 8, 2016 at 4:32 PM, Doug Tomm <dct...@gmail.com> > > wrote: > > > > > >> > > > > > >> we're using kafka-python, weighing pykafka, and wondering if > > there's > > > > > >> another that is bettor to use. does confluent endorse or > > recommend > > > a > > > > > >> particular python package (psorry for the alliteration)? > > > > > >> > > > > > >> doug > > > > > >> > > > > > >> > > > > > > > > > > > > > > >
