I think this points out the need for a single canonical cross-platform C client lib with support for Zookeeper that could easily wrapped for use in other languages.
It would make it much easier for people using Python, Ruby, Node, Lua, Haskell, Go, OCaml, etc to have such a library that matches the features of the Scala/Java client. Currently writing a client amounts to re-implementing the entire Scala client from scratch, which is why so many clients settle for simple producer/consumer support and skip Zk altogether. On Thu, Aug 29, 2013 at 10:44 PM, Jun Rao <jun...@gmail.com> wrote: > I assume this for Kafka 0.7. One option is to use a VIP in front of the > brokers for load balancing. > > Thanks, > > Jun > > > On Thu, Aug 29, 2013 at 1:39 PM, Mark <static.void....@gmail.com> wrote: > > > We are thinking about using Kafka to collect events from our Rails > > application and I was hoping to get some input from the Kafka community. > > > > Currently the only gems available are: > > > > https://github.com/acrosa/kafka-rb > > https://github.com/bpot/poseidon (Can't use since we are only running > > 1.8.7) > > > > Now neither of these integrate with Zookeeper so we are missing quite a > > few features: > > > > - Auto discovery of brokers > > - Auto discover of partitions for consumers > > - … fill in the rest here, new to Kafka so don't know everything that is > > missing > > > > I was wondering what my best options are going forward with Kakfa? I > think > > we have the following choices > > > > A) Instead of writing directly to Kafka from our application we can write > > our events/messages to some other source (Syslog, File, ?) and then have > a > > separate Java process that reads these sources and writes to Kafka. This > is > > a little annoying since we now have to worry about every machine also > > running the above separate process to write to Kafka. > > > > B) Work around the above limitations. Auto-discover of brokers is > terrible > > since we don't foresee us adding/removing brokers that frequently. The > lack > > of auto-discover of partitions is definitely a loss since we now have to > > know which broker/partition to read from at all times. Of course we can > > just write to Kafka using the above Gems and have our consumers written > in > > another language. > > > > C)? > > > > Any thoughts/opinions? > > > > Thanks > > > > - M > > > > >
