Agree - kafka-python was in hibernation waiting for 0.9.0.0 Kafka release, so a few issues lingered longer than I would have liked. Most of my comments relate to latest master, which we are hoping to release after a bit more testing and polish.
Re librdkafka -- to be honest, I'm skeptical that C protocol bindings are going to improve python performance much. In my experience, the devil is in the client logic details, not the wire protocol parsing. Adding a C compilation step also adds installation and operational overhead (install gcc or manage linux wheels). So we have avoided adding that to kafka-python without significant evidence showing performance benefits that can't be duplicated in pure python. -Dana On Jan 11, 2016 9:02 AM, "Sam Pegler" <sam.peg...@infectiousmedia.com> wrote: > kafka-python (https://github.com/dpkp/kafka-python) has also just merged > performance improvements to the consumer in > https://github.com/dpkp/kafka-python/issues/290 which should see a pretty > decent boost in throughput. We were somewhat put off by the poor > performance in earlier versions, I imagine many people would have been in > the same position so it's worth revisiting. > > Sam Pegler > > WEBOPS ENGINEER T. +44(0) 07 562 867 486 [image: Infectious Media]3-7 > Herbal Hill / London / EC1R 5EJwww.infectiousmedia.com [image: Infectious > Media] <http://www.infectiousmedia.com/>[image: Facebook] > <http://www.facebook.com/infectiousmedia>[image: Twitter] > <https://twitter.com/infectiousmedia>[image: LinkedIn] > <http://www.linkedin.com/company/infectious-media-ltd>[image: Youtube] > <http://www.youtube.com/user/InfectiousMediaLtd> This email and any > attachments are confidential and may also be privileged. If youare not the > intended recipient, please notify the sender immediately, and do > notdisclose > the contents to another person, use it for any purpose, or store, or > copythe > information in any medium. Please also destroy and delete the message > fromyour > computer. > > On 11 January 2016 at 16:28, Andrew Otto <o...@wikimedia.org> wrote: > > > pykafka’s balanced consumer is very useful. pykafka also has Python > > bindings to the librdkafka C library that you can optionally enable, > which > > might get you some speed boosts. > > > > python-kafka (oh, I just saw this 0.9x version, hm!) was better at > > producing than pykafka for us, so we am currently using pykafka for > > consumption, and python-kafka for production. python-kafka allows you to > > produce to multiple topics using the same client instance. (pykafka may > > support this soon: https://github.com/Parsely/pykafka/issues/354) > > > > > > > > On Sat, Jan 9, 2016 at 10:04 AM, Dana Powers <dana.pow...@gmail.com> > > wrote: > > > > > pykafka uses a custom zookeeper implementation for consumer groups. > > > kafka-python uses the 0.9.0.0 server apis to accomplish the same. > > > > > > -Dana > > > On Jan 8, 2016 18:32, "chengxin Cai" <ia...@outlook.com> wrote: > > > > > > > Hi > > > > > > > > I heard that Pykakfa can create a balanced consumer. > > > > > > > > And there should be no other big difference. > > > > > > > > > > > > Best Regards > > > > > > > > > 在 2016年1月9日，08:58，Dana Powers <dana.pow...@rd.io> 写道： > > > > > > > > > > Hi Doug, > > > > > > > > > > The differences are fairly subtle. kafka-python is a > community-backed > > > > > project that aims to be consistent w/ the official java client; > > pykafka > > > > is > > > > > sponsored by parse.ly and aims to provide a pythonic interface. > > > > whichever > > > > > you go with, I would love to hear your specific feedback on > > > kafka-python. > > > > > > > > > > -Dana (kafka-python maintainer) > > > > > > > > > >> On Fri, Jan 8, 2016 at 4:32 PM, Doug Tomm <dct...@gmail.com> > wrote: > > > > >> > > > > >> we're using kafka-python, weighing pykafka, and wondering if > there's > > > > >> another that is bettor to use. does confluent endorse or > recommend > > a > > > > >> particular python package (psorry for the alliteration)? > > > > >> > > > > >> doug > > > > >> > > > > >> > > > > > > > > > >
