In 0.8, the producer no longer depends on ZK. It only takes a list of brokers. At LinkedIn, we have a 0.8 C producer implementation and plan to open source it soon.
Thanks, Jun On Fri, Aug 30, 2013 at 9:00 AM, Travis Brady <travis.br...@gmail.com>wrote: > I think this points out the need for a single canonical cross-platform C > client lib with support for Zookeeper that could easily wrapped for use in > other languages. > > It would make it much easier for people using Python, Ruby, Node, Lua, > Haskell, Go, OCaml, etc to have such a library that matches the features of > the Scala/Java client. > Currently writing a client amounts to re-implementing the entire Scala > client from scratch, which is why so many clients settle for simple > producer/consumer support and skip Zk altogether. > > > > > On Thu, Aug 29, 2013 at 10:44 PM, Jun Rao <jun...@gmail.com> wrote: > > > I assume this for Kafka 0.7. One option is to use a VIP in front of the > > brokers for load balancing. > > > > Thanks, > > > > Jun > > > > > > On Thu, Aug 29, 2013 at 1:39 PM, Mark <static.void....@gmail.com> wrote: > > > > > We are thinking about using Kafka to collect events from our Rails > > > application and I was hoping to get some input from the Kafka > community. > > > > > > Currently the only gems available are: > > > > > > https://github.com/acrosa/kafka-rb > > > https://github.com/bpot/poseidon (Can't use since we are only running > > > 1.8.7) > > > > > > Now neither of these integrate with Zookeeper so we are missing quite a > > > few features: > > > > > > - Auto discovery of brokers > > > - Auto discover of partitions for consumers > > > - … fill in the rest here, new to Kafka so don't know everything that > is > > > missing > > > > > > I was wondering what my best options are going forward with Kakfa? I > > think > > > we have the following choices > > > > > > A) Instead of writing directly to Kafka from our application we can > write > > > our events/messages to some other source (Syslog, File, ?) and then > have > > a > > > separate Java process that reads these sources and writes to Kafka. > This > > is > > > a little annoying since we now have to worry about every machine also > > > running the above separate process to write to Kafka. > > > > > > B) Work around the above limitations. Auto-discover of brokers is > > terrible > > > since we don't foresee us adding/removing brokers that frequently. The > > lack > > > of auto-discover of partitions is definitely a loss since we now have > to > > > know which broker/partition to read from at all times. Of course we can > > > just write to Kafka using the above Gems and have our consumers written > > in > > > another language. > > > > > > C)? > > > > > > Any thoughts/opinions? > > > > > > Thanks > > > > > > - M > > > > > > > > >
