Hi -- I am preparing to make a case for using Kafka instead of Rabbit MQ as a broker-based messaging provider. The context is similar to that of the Kafka papers and user stories: the producers publish monitoring data and logs, and a suite of subscribers consume this data (some store it, others perform computations on the event stream). The requirements are typical of this context: low-latency, high-throughput, ability to deal with bursts and operate in/across multiple data centers, etc.
I am familiar with the performance comparison between Kafka, Rabbit MQ and Active MQ from the NetDB 2011 paper<http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf>. However in the two years that passed since then the number of production Kafka installations increased, and people are using it in different ways than those imagined by Kafka's designers. In light of these experiences one can use more data points and color when contrasting to Rabbit MQ (which by the way also evolved since 2011). (And FWIW I know I am not the first one to walk this path; see for example last year's OSCON session on the State of MQ<http://lanyrd.com/2012/oscon/swrcz/>.) I would appreciate it if you could share measurements, results, or even anecdotal evidence along these lines. How have you avoided the "let's use Rabbit MQ because everybody else does it" route when solving problems for which Kafka is a better fit? Thanks, -Dragos
