One more useful link... http://hg.rabbitmq.com/rabbitmq-server/file/bc2fda987fe8/src/rabbit_queue_index.erl#l32 On Jun 8, 2013 9:20 PM, "Alexis Richardson" <alexis.richard...@gmail.com> wrote:
> A few more details for those following this: > > On Sat, Jun 8, 2013 at 9:09 PM, Alexis Richardson > <alexis.richard...@gmail.com> wrote: > > Jonathan > > > > I am aware of the difference between sequential writes and other kinds > > of writes ;p) > > > > AFAIK the Kafka docs describe a sort of platonic alternative system, > > eg "normally people do this.. Kafka does that..". This is a good way > > to explain design decisions. However, I think you may be assuming > > that Rabbit is a lot like the generalised other system. But it is not > > - eg Rabbit does not do lots of random IO. I'm led to understand that > > Rabbit's msg store is closer to log structured storage (a la > > Log-Structured Merge Trees) in some ways. > ... > >> > >> That would be awesome if you can confirm what Rabbit is using as a > >> persistent data structure. > > See extensive comments in here: > > > http://hg.rabbitmq.com/rabbitmq-server/file/bc2fda987fe8/src/rabbit_msg_store.erl > > > >> More importantly, whether it is BTree or > >> something else, is the disk i/o random or linear? > .. > >> This is only speaking of the use case of high throughput with persisting > >> large amounts of data to disk where there is 4 orders of magnitude more > >> than 10x difference. It all comes down to random vs sequential > >> writes/reads to disk as I mentioned above. > > It's not a btree with random writes, hence my puzzlement earlier. > > * there are mostly linear writes in a file > * multiple files are involved, moved around, garbage collected, > compacted, etc, which is obviously not all linear. > > This will behave better than a btree for the purpose it was built for. > > This is just for writes. Reads may be a different story - and I don't > fully understand how reads work in Kafka. A memory mapped circular > buffer will definitely outperform this... mmap support for erlang > would be nice ;p) > > > > > >> > >> On Sat, Jun 8, 2013 at 2:07 AM, Alexis Richardson < > >> alexis.richard...@gmail.com> wrote: > >> > >>> Jonathan > >>> > >>> On Sat, Jun 8, 2013 at 2:09 AM, Jonathan Hodges <hodg...@gmail.com> > wrote: > >>> > Thanks so much for your replies. This has been a great help > >>> understanding > >>> > Rabbit better with having very little experience with it. I have a > few > >>> > follow up comments below. > >>> > >>> Happy to help! > >>> > >>> I'm afraid I don't follow your arguments below. Rabbit contains many > >>> optimisations too. I'm told that it is possible to saturate the disk > >>> i/o, and you saw the message rates I quoted in the previous email. > >>> YES of course there are differences, mostly an accumulation of things. > >>> For example Rabbit spends more time doing work before it writes to > >>> disk. > >>> > >>> You said: > >>> > >>> "Since Rabbit must maintain the state of the > >>> consumers I imagine it’s subjected to random data access patterns on > disk > >>> as opposed to sequential." > >>> > >>> I don't follow the logic here, sorry. > >>> > >>> Couple of side comments: > >>> > >>> * In your Hadoop vs RT example, Rabbit would deliver the RT messages > >>> immediately and write the rest to disk. It can do this at high rates > >>> - I shall try to get you some useful data here. > >>> > >>> * Bear in mind that write speed should be orthogonal to read speed. > >>> Ask yourself - how would Kafka provide a read cache, and when might > >>> that be useful? > >>> > >>> * I'll find out what data structure Rabbit uses for long term > persistence. > >>> > >>> > >>> "Quoting the Kafka design page ( > >>> http://kafka.apache.org/07/design.html) performance of sequential > writes > >>> on > >>> a 6 7200rpm SATA RAID-5 array is about 300MB/sec but the performance of > >>> random writes is only about 50k/sec—a difference of nearly 10000X." > >>> > >>> Depending on your use case, I'd expect 2x-10x overall throughput > >>> differences, and will try to find out more info. As I said, Rabbit > >>> can saturate disk i/o. > >>> > >>> alexis > >>> > >>> > >>> > >>> > >>> > > >>> >> While you are correct the payload is a much bigger concern, > managing the > >>> >> metadata and acks centrally on the broker across multiple clients at > >>> scale > >>> >> is also a concern. This would seem to be exasperated if you have > >>> > consumers > >>> >> at different speeds i.e. Storm and Hadoop consuming the same topic. > >>> >> > >>> >> In that scenario, say storm consumes the topic messages in > real-time and > >>> >> Hadoop consumes once a day. Let’s assume the topic consists of > 100k+ > >>> >> messages/sec throughput so that in a given day you might have 100s > GBs > >>> of > >>> >> data flowing through the topic. > >>> >> > >>> >> To allow Hadoop to consume once a day, Rabbit obviously can’t keep > 100s > >>> > GBs > >>> >> in memory and will need to persist this data to its internal DB to > be > >>> >> retrieved later. > >>> > > >>> > I am not sure why you think this is a problem? > >>> > > >>> > For a fixed number of producers and consumers, the pubsub and > delivery > >>> > semantics of Rabbit and Kafka are quite similar. Think of Rabbit as > >>> > adding an in-memory cache that is used to (a) speed up read > >>> > consumption, (b) obviate disk writes when possible due to all client > >>> > consumers being available and consuming. > >>> > > >>> > > >>> > Actually I think this is the main use case that sets Kafka apart from > >>> > Rabbit and speaks to the poster’s ‘Arguments for Kafka over RabbitMQ’ > >>> > question. As you mentioned Rabbit is a general purpose messaging > system > >>> > and along with that has a lot of features not found in Kafka. There > are > >>> > plenty of times when Rabbit makes more sense than Kafka, but not > when you > >>> > are maintaining large message stores and require high throughput to > disk. > >>> > > >>> > Persisting 100s GBs of messages to disk is a much different problem > than > >>> > managing messages in memory. Since Rabbit must maintain the state > of the > >>> > consumers I imagine it’s subjected to random data access patterns on > disk > >>> > as opposed to sequential. Quoting the Kafka design page ( > >>> > http://kafka.apache.org/07/design.html) performance of sequential > >>> writes on > >>> > a 6 7200rpm SATA RAID-5 array is about 300MB/sec but the performance > of > >>> > random writes is only about 50k/sec—a difference of nearly 10000X. > >>> > > >>> > They go on to say persistent data structure used in messaging systems > >>> > metadata is often a BTree. BTrees are the most versatile data > structure > >>> > available, and make it possible to support a wide variety of > >>> transactional > >>> > and non-transactional semantics in the messaging system. They do come > >>> with > >>> > a fairly high cost, though: Btree operations are O(log N). Normally > O(log > >>> > N) is considered essentially equivalent to constant time, but this > is not > >>> > true for disk operations. Disk seeks come at 10 ms a pop, and each > disk > >>> can > >>> > do only one seek at a time so parallelism is limited. Hence even a > >>> handful > >>> > of disk seeks leads to very high overhead. Since storage systems mix > very > >>> > fast cached operations with actual physical disk operations, the > observed > >>> > performance of tree structures is often superlinear. Furthermore > BTrees > >>> > require a very sophisticated page or row locking implementation to > avoid > >>> > locking the entire tree on each operation. The implementation must > pay a > >>> > fairly high price for row-locking or else effectively serialize all > >>> reads. > >>> > Because of the heavy reliance on disk seeks it is not possible to > >>> > effectively take advantage of the improvements in drive density, and > one > >>> is > >>> > forced to use small (< 100GB) high RPM SAS drives to maintain a sane > >>> ratio > >>> > of data to seek capacity. > >>> > > >>> > Intuitively a persistent queue could be built on simple reads and > appends > >>> > to files as is commonly the case with logging solutions. Though this > >>> > structure would not support the rich semantics of a BTree > implementation, > >>> > but it has the advantage that all operations are O(1) and reads do > not > >>> > block writes or each other. This has obvious performance advantages > since > >>> > the performance is completely decoupled from the data size--one > server > >>> can > >>> > now take full advantage of a number of cheap, low-rotational speed > 1+TB > >>> > SATA drives. Though they have poor seek performance, these drives > often > >>> > have comparable performance for large reads and writes at 1/3 the > price > >>> and > >>> > 3x the capacity. > >>> > > >>> > Having access to virtually unlimited disk space without penalty means > >>> that > >>> > we can provide some features not usually found in a messaging > system. For > >>> > example, in kafka, instead of deleting a message immediately after > >>> > consumption, we can retain messages for a relative long period (say a > >>> week). > >>> > > >>> > Our assumption is that the volume of messages is extremely high, > indeed > >>> it > >>> > is some multiple of the total number of page views for the site > (since a > >>> > page view is one of the activities we process). Furthermore we assume > >>> each > >>> > message published is read at least once (and often multiple times), > hence > >>> > we optimize for consumption rather than production. > >>> > > >>> > There are two common causes of inefficiency: too many network > requests, > >>> and > >>> > excessive byte copying. > >>> > > >>> > To encourage efficiency, the APIs are built around a "message set" > >>> > abstraction that naturally groups messages. This allows network > requests > >>> to > >>> > group messages together and amortize the overhead of the network > >>> roundtrip > >>> > rather than sending a single message at a time. > >>> > > >>> > The MessageSet implementation is itself a very thin API that wraps a > byte > >>> > array or file. Hence there is no separate serialization or > >>> deserialization > >>> > step required for message processing, message fields are lazily > >>> > deserialized as needed (or not deserialized if not needed). > >>> > > >>> > The message log maintained by the broker is itself just a directory > of > >>> > message sets that have been written to disk. This abstraction allows > a > >>> > single byte format to be shared by both the broker and the consumer > (and > >>> to > >>> > some degree the producer, though producer messages are checksumed and > >>> > validated before being added to the log). > >>> > > >>> > Maintaining this common format allows optimization of the most > important > >>> > operation: network transfer of persistent log chunks. Modern unix > >>> operating > >>> > systems offer a highly optimized code path for transferring data out > of > >>> > pagecache to a socket; in Linux this is done with the sendfile system > >>> call. > >>> > Java provides access to this system call with the > FileChannel.transferTo > >>> > api. > >>> > > >>> > To understand the impact of sendfile, it is important to understand > the > >>> > common data path for transfer of data from file to socket: > >>> > > >>> > 1. The operating system reads data from the disk into pagecache in > >>> kernel > >>> > space > >>> > 2. The application reads the data from kernel space into a > user-space > >>> > buffer > >>> > 3. The application writes the data back into kernel space into a > socket > >>> > buffer > >>> > 4. The operating system copies the data from the socket buffer to > the > >>> NIC > >>> > buffer where it is sent over the network > >>> > > >>> > This is clearly inefficient, there are four copies, two system calls. > >>> Using > >>> > sendfile, this re-copying is avoided by allowing the OS to send the > data > >>> > from pagecache to the network directly. So in this optimized path, > only > >>> the > >>> > final copy to the NIC buffer is needed. > >>> > > >>> > We expect a common use case to be multiple consumers on a topic. > Using > >>> the > >>> > zero-copy optimization above, data is copied into pagecache exactly > once > >>> > and reused on each consumption instead of being stored in memory and > >>> copied > >>> > out to kernel space every time it is read. This allows messages to be > >>> > consumed at a rate that approaches the limit of the network > connection. > >>> > > >>> > > >>> > So in the end it would seem Kafka’s specialized nature to write data > >>> first > >>> > really shines over Rabbit when your use case requires a very high > >>> > throughput unblocking firehose with large data persistence to disk. > >>> Since > >>> > this is only one use case this by no means is saying Kafka is better > than > >>> > Rabbit or vice versa. I think it is awesome there are more options > to > >>> > choose from so you can pick the right tool for the job. Thanks open > >>> source! > >>> > > >>> > As always YMMV. > >>> > > >>> > > >>> > > >>> > On Fri, Jun 7, 2013 at 4:40 PM, Alexis Richardson < > >>> > alexis.richard...@gmail.com> wrote: > >>> > > >>> >> Jonathan, > >>> >> > >>> >> > >>> >> On Fri, Jun 7, 2013 at 7:03 PM, Jonathan Hodges <hodg...@gmail.com> > >>> wrote: > >>> >> > Hi Alexis, > >>> >> > > >>> >> > I appreciate your reply and clarifications to my misconception > about > >>> >> > Rabbit, particularly on the copying of the message payloads per > >>> consumer. > >>> >> > >>> >> Thank-you! > >>> >> > >>> >> > >>> >> > It sounds like it only copies metadata like the consumer state > i.e. > >>> >> > position in the topic messages. > >>> >> > >>> >> Basically yes. Of course when a message is delivered to N>1 > >>> >> *machines*, then there will be N copies, one per machine. > >>> >> > >>> >> Also, for various reasons, very tiny (<60b) messages do get copied > as > >>> >> you'd assumed. > >>> >> > >>> >> > >>> >> > I don’t have experience with Rabbit and > >>> >> > was basing this assumption based on Google searches like the > >>> following - > >>> >> > > >>> >> > >>> > http://ilearnstack.com/2013/04/16/introduction-to-amqp-messaging-with-rabbitmq/ > >>> >> . > >>> >> > It seems to indicate with topic exchanges that the messages get > >>> copied > >>> >> to > >>> >> > a queue per consumer, but I am glad you confirmed it is just the > >>> >> metadata. > >>> >> > >>> >> Yup. > >>> >> > >>> >> That's a fairly decent article but even the good stuff uses words > like > >>> >> "copy" without a fixed denotation. Don't believe the internets! > >>> >> > >>> >> > >>> >> > While you are correct the payload is a much bigger concern, > managing > >>> the > >>> >> > metadata and acks centrally on the broker across multiple clients > at > >>> >> scale > >>> >> > is also a concern. This would seem to be exasperated if you have > >>> >> consumers > >>> >> > at different speeds i.e. Storm and Hadoop consuming the same > topic. > >>> >> > > >>> >> > In that scenario, say storm consumes the topic messages in > real-time > >>> and > >>> >> > Hadoop consumes once a day. Let’s assume the topic consists of > 100k+ > >>> >> > messages/sec throughput so that in a given day you might have 100s > >>> GBs of > >>> >> > data flowing through the topic. > >>> >> > > >>> >> > To allow Hadoop to consume once a day, Rabbit obviously can’t keep > >>> 100s > >>> >> GBs > >>> >> > in memory and will need to persist this data to its internal DB > to be > >>> >> > retrieved later. > >>> >> > >>> >> I am not sure why you think this is a problem? > >>> >> > >>> >> For a fixed number of producers and consumers, the pubsub and > delivery > >>> >> semantics of Rabbit and Kafka are quite similar. Think of Rabbit as > >>> >> adding an in-memory cache that is used to (a) speed up read > >>> >> consumption, (b) obviate disk writes when possible due to all client > >>> >> consumers being available and consuming. > >>> >> > >>> >> > >>> >> > I believe when large amounts of data need to be persisted > >>> >> > is the scenario described in the earlier posted Kafka paper ( > >>> >> > > >>> >> > >>> > http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf > >>> >> ) > >>> >> > where Rabbit’s performance really starts to bog down as compared > to > >>> >> Kafka. > >>> >> > >>> >> Not sure what parts of the paper you mean? > >>> >> > >>> >> I read that paper when it came out. I found it strongest when > >>> >> describing Kafka's design philosophy. I found the performance > >>> >> statements made about Rabbit pretty hard to understand. This is not > >>> >> meant to be a criticism of the authors! I have seen very few > >>> >> performance papers about messaging that I would base decisions on. > >>> >> > >>> >> > >>> >> > This Kafka paper is looks to be a few years old > >>> >> > >>> >> Um.... Lots can change in technology very quickly :-) > >>> >> > >>> >> Eg.: At the time this paper was published, Instagram had 5m users. > >>> >> Six months earlier in Dec 2010, it had 1m. Since then it grew huge > >>> >> and got acquired. > >>> >> > >>> >> > >>> >> > >>> >> > so has something changed > >>> >> > within the Rabbit architecture to alleviate this issue when large > >>> amounts > >>> >> > of data are persisted to the internal DB? > >>> >> > >>> >> Rabbit introduced a new internal flow control system which impacted > >>> >> performance under steady load. This may be relevant? I couldn't > say > >>> >> from reading the paper. > >>> >> > >>> >> I don't have a good reference for this to hand, but here is a post > >>> >> about external flow control that you may find amusing: > >>> >> > >>> >> > >>> > http://www.rabbitmq.com/blog/2012/05/11/some-queuing-theory-throughput-latency-and-bandwidth/ > >>> >> > >>> >> > >>> >> > Do the producer and consumer > >>> >> > numbers look correct? If no, maybe you can share some Rabbit > >>> benchmarks > >>> >> > under this scenario, because I believe it is the main area where > Kafka > >>> >> > appears to be the superior solution. > >>> >> > >>> >> This is from about one year ago: > >>> >> > >>> >> > >>> > http://www.rabbitmq.com/blog/2012/04/25/rabbitmq-performance-measurements-part-2/ > >>> >> > >>> >> Obviously none of this uses batching, which is an easy trick for > >>> >> increasing throughput. > >>> >> > >>> >> YMMV. > >>> >> > >>> >> Is this helping? > >>> >> > >>> >> alexis > >>> >> > >>> >> > >>> >> > >>> >> > Thanks for educating me on these matters. > >>> >> > > >>> >> > -Jonathan > >>> >> > > >>> >> > > >>> >> > > >>> >> > On Fri, Jun 7, 2013 at 6:54 AM, Alexis Richardson < > >>> ale...@rabbitmq.com > >>> >> >wrote: > >>> >> > > >>> >> >> Hi > >>> >> >> > >>> >> >> Alexis from Rabbit here. I hope I am not intruding! > >>> >> >> > >>> >> >> It would be super helpful if people with questions, observations > or > >>> >> >> moans posted them to the rabbitmq list too :-) > >>> >> >> > >>> >> >> A few comments: > >>> >> >> > >>> >> >> * Along with ZeroMQ, I consider Kafka to be one of the > interesting > >>> and > >>> >> >> useful messaging projects out there. In a world of cruft, Kafka > is > >>> >> >> cool! > >>> >> >> > >>> >> >> * This is because both projects come at messaging from a specific > >>> >> >> point of view that is *different* from Rabbit. OTOH, many other > >>> >> >> projects exist that replicate Rabbit features for fun, or NIH, > or due > >>> >> >> to misunderstanding the semantics (yes, our docs could be better) > >>> >> >> > >>> >> >> * It is striking how few people describe those differences. In a > >>> >> >> nutshell they are as follows: > >>> >> >> > >>> >> >> *** Kafka writes all incoming data to disk immediately, and then > >>> >> >> figures out who sees what. So it is much more like a database > than > >>> >> >> Rabbit, in that new consumers can appear well after the disk > write > >>> and > >>> >> >> still subscribe to past messages. Instead, Rabbit which tries to > >>> >> >> deliver to consumers and buffers otherwise. Persistence is > optional > >>> >> >> but robust and a feature of the buffer ("queue") not the upstream > >>> >> >> machinery. Rabbit is able to cache-on-arrival via a plugin, but > this > >>> >> >> is a design overlay and not particularly optimal. > >>> >> >> > >>> >> >> *** Kafka is a client server system with end to end semantics. > It > >>> >> >> defines order to include processing order, and keeps state on the > >>> >> >> client to do this. Group management is via a 3rd party service > >>> >> >> (Zookeeper? I forget which). Rabbit is a server-only protocol > based > >>> >> >> system which maintains order on the server and through completely > >>> >> >> language neutral protocol semantics. This makes Rabbit perhaps > more > >>> >> >> natural as a 'messaging service' eg for integration and other > >>> >> >> inter-app data transfer. > >>> >> >> > >>> >> >> *** Rabbit is a general purpose messaging system with extras like > >>> >> >> federation. It speaks many protocols, and has core features > like HA, > >>> >> >> transactions, management, etc. Everything can be switched on or > off. > >>> >> >> Getting all this to work while keeping the install light and > fast, is > >>> >> >> quite fiddly. Kafka by contrast comes from a specific set of use > >>> >> >> cases, which are interesting certainly. I am not sure if Kafka > wants > >>> >> >> to be a general purpose messaging system, but it will become a > bit > >>> >> >> more like Rabbit if that is the goal. > >>> >> >> > >>> >> >> *** Both approaches have costs. In the case of Rabbit the cost > is > >>> >> >> that more metadata is stored on the broker. Kafka can get > >>> performance > >>> >> >> gains by storing less such data. But we are talking about some N > >>> >> >> thousands of MPS versus some M thousands. At those speeds the > >>> clients > >>> >> >> are usually the bottleneck anyway. > >>> >> >> > >>> >> >> * Let me also clarify some things: > >>> >> >> > >>> >> >> *** Rabbit does NOT store multiple copies of the same message > across > >>> >> >> queues, unless they are very small (<60b, iirc). A message > delivered > >>> >> >> to >1 queue on 1 machine is stored once. Metadata about that > message > >>> >> >> may be stored more than once, but, at scale, the big cost is the > >>> >> >> payload. > >>> >> >> > >>> >> >> *** Rabbit's vanilla install does store some index data in memory > >>> when > >>> >> >> messages flow to disk. You can change this by using a plugin, > but > >>> >> >> this is a secret-menu undocumented feature. Very very few people > >>> need > >>> >> >> any such thing. > >>> >> >> > >>> >> >> *** A Rabbit queue is lightweight. It's just an ordered > consumption > >>> >> >> buffer that can persist and ack. Don't assume things about > Rabbit > >>> >> >> queues based on what you know about IBM MQ, JMS, and so forth. > >>> Queues > >>> >> >> in Rabbit and Kafka are not the same. > >>> >> >> > >>> >> >> *** Rabbit does not use mnesia for message storage. It has its > own > >>> >> >> DB, optimised for messaging. You can use other DBs but this is > >>> >> >> Complicated. > >>> >> >> > >>> >> >> *** Rabbit does all kinds of batching and bulk processing, and > can > >>> >> >> batch end to end. If you see claims about batching, buffering, > etc., > >>> >> >> find out ALL the details before drawing conclusions. > >>> >> >> > >>> >> >> I hope this is helpful. > >>> >> >> > >>> >> >> Keen to get feedback / questions / corrections. > >>> >> >> > >>> >> >> alexis > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> On Fri, Jun 7, 2013 at 2:09 AM, Marc Labbe <mrla...@gmail.com> > >>> wrote: > >>> >> >> > We also went through the same decision making and our > arguments for > >>> >> Kafka > >>> >> >> > where in the same lines as those Jonathan mentioned. The fact > that > >>> we > >>> >> >> have > >>> >> >> > heterogeneous consumers is really a deciding factor. Our > >>> requirements > >>> >> >> were > >>> >> >> > to avoid loosing messages at all cost while having multiple > >>> consumers > >>> >> >> > reading the same data at a different pace. On one side, we > have a > >>> few > >>> >> >> > consumers being fed with data coming in from most, if not all, > >>> >> topics. On > >>> >> >> > the other side, we have a good bunch of consumers reading only > >>> from a > >>> >> >> > single topic. The big guys can take their time to read while > the > >>> >> smaller > >>> >> >> > ones are mostly for near real-time events so they need to keep > up > >>> the > >>> >> >> pace > >>> >> >> > of incoming messages. > >>> >> >> > > >>> >> >> > RabbitMQ stores data on disk only if you tell it to while Kafka > >>> >> persists > >>> >> >> by > >>> >> >> > design. From the beginning, we decided we would try to use the > >>> queues > >>> >> the > >>> >> >> > same way, pub/sub with a routing key (an exchange in RabbitMQ) > or > >>> >> topic, > >>> >> >> > persisted to disk and replicated. > >>> >> >> > > >>> >> >> > One of our scenario was to see how the system would cope with > the > >>> >> largest > >>> >> >> > consumer down for a while, therefore forcing the brokers to > keep > >>> the > >>> >> data > >>> >> >> > for a long period. In the case of RabbitMQ, this consumer has > it > >>> owns > >>> >> >> queue > >>> >> >> > and data grows on disk, which is not really a problem if you > plan > >>> >> >> > consequently. But, since it has to keep track of all messages > read, > >>> >> the > >>> >> >> > Mnesia database used by RabbitMQ as the messages index also > grows > >>> >> pretty > >>> >> >> > big. At that point, the amount of RAM necessary becomes very > large > >>> to > >>> >> >> keep > >>> >> >> > the level of performance we need. In our tests, we found that > this > >>> an > >>> >> >> > adverse effect on ALL the brokers, thus affecting all > consumers. > >>> You > >>> >> can > >>> >> >> > always say that you'll monitor the consumers to make sure it > won't > >>> >> >> happen. > >>> >> >> > That's a good thing if you can. I wasn't ready to make that > bet. > >>> >> >> > > >>> >> >> > Another point is the fact that, since we wanted to use pub/sub > >>> with a > >>> >> >> > exchange in RabbitMQ, we would have ended up with a lot data > >>> >> duplication > >>> >> >> > because if a message is read by multiple consumers, it will get > >>> >> >> duplicated > >>> >> >> > in the queue of each of those consumer. Kafka wins on that > side too > >>> >> since > >>> >> >> > every consumer reads from the same source. > >>> >> >> > > >>> >> >> > The downsides of Kafka were the language issues (we are using > >>> mostly > >>> >> >> Python > >>> >> >> > and C#). 0.8 is very new and few drivers are available at this > >>> point. > >>> >> >> Also, > >>> >> >> > we will have to try getting as close as possible to > >>> once-and-only-once > >>> >> >> > guarantee. There are two things where RabbitMQ would have > given us > >>> >> less > >>> >> >> > work out of the box as opposed to Kafka. RabbitMQ also > provides a > >>> >> bunch > >>> >> >> of > >>> >> >> > tools that makes it rather attractive too. > >>> >> >> > > >>> >> >> > In the end, looking at throughput is a pretty nifty thing but > being > >>> >> sure > >>> >> >> > that I'll be able to manage the beast as it grows will allow > me to > >>> >> get to > >>> >> >> > sleep way more easily. > >>> >> >> > > >>> >> >> > > >>> >> >> > On Thu, Jun 6, 2013 at 3:28 PM, Jonathan Hodges < > hodg...@gmail.com > >>> > > >>> >> >> wrote: > >>> >> >> > > >>> >> >> >> We just went through a similar exercise with RabbitMQ at our > >>> company > >>> >> >> with > >>> >> >> >> streaming activity data from our various web properties. Our > use > >>> >> case > >>> >> >> >> requires consumption of this stream by many heterogeneous > >>> consumers > >>> >> >> >> including batch (Hadoop) and real-time (Storm). We pointed > out > >>> that > >>> >> >> Kafka > >>> >> >> >> acts as a configurable rolling window of time on the activity > >>> stream. > >>> >> >> The > >>> >> >> >> window default is 7 days which allows for supporting clients > of > >>> >> >> different > >>> >> >> >> latencies like Hadoop and Storm to read from the same stream. > >>> >> >> >> > >>> >> >> >> We pointed out that the Kafka brokers don't need to maintain > >>> consumer > >>> >> >> state > >>> >> >> >> in the stream and only have to maintain one copy of the > stream to > >>> >> >> support N > >>> >> >> >> number of consumers. Rabbit brokers on the other hand have to > >>> >> maintain > >>> >> >> the > >>> >> >> >> state of each consumer as well as create a copy of the stream > for > >>> >> each > >>> >> >> >> consumer. In our scenario we have 10-20 consumers and with > the > >>> scale > >>> >> >> and > >>> >> >> >> throughput of the activity stream we were able to show Rabbit > >>> quickly > >>> >> >> >> becomes the bottleneck under load. > >>> >> >> >> > >>> >> >> >> > >>> >> >> >> > >>> >> >> >> On Thu, Jun 6, 2013 at 12:40 PM, Dragos Manolescu < > >>> >> >> >> dragos.manole...@servicenow.com> wrote: > >>> >> >> >> > >>> >> >> >> > Hi -- > >>> >> >> >> > > >>> >> >> >> > I am preparing to make a case for using Kafka instead of > Rabbit > >>> MQ > >>> >> as > >>> >> >> a > >>> >> >> >> > broker-based messaging provider. The context is similar to > that > >>> of > >>> >> the > >>> >> >> >> > Kafka papers and user stories: the producers publish > monitoring > >>> >> data > >>> >> >> and > >>> >> >> >> > logs, and a suite of subscribers consume this data (some > store > >>> it, > >>> >> >> others > >>> >> >> >> > perform computations on the event stream). The requirements > are > >>> >> >> typical > >>> >> >> >> of > >>> >> >> >> > this context: low-latency, high-throughput, ability to deal > with > >>> >> >> bursts > >>> >> >> >> and > >>> >> >> >> > operate in/across multiple data centers, etc. > >>> >> >> >> > > >>> >> >> >> > I am familiar with the performance comparison between Kafka, > >>> >> Rabbit MQ > >>> >> >> >> and > >>> >> >> >> > Active MQ from the NetDB 2011 paper< > >>> >> >> >> > > >>> >> >> >> > >>> >> >> > >>> >> > >>> > http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf > >>> >> >> >> >. > >>> >> >> >> > However in the two years that passed since then the number > of > >>> >> >> production > >>> >> >> >> > Kafka installations increased, and people are using it in > >>> different > >>> >> >> ways > >>> >> >> >> > than those imagined by Kafka's designers. In light of these > >>> >> >> experiences > >>> >> >> >> one > >>> >> >> >> > can use more data points and color when contrasting to > Rabbit MQ > >>> >> >> (which > >>> >> >> >> by > >>> >> >> >> > the way also evolved since 2011). (And FWIW I know I am not > the > >>> >> first > >>> >> >> one > >>> >> >> >> > to walk this path; see for example last year's OSCON > session on > >>> the > >>> >> >> State > >>> >> >> >> > of MQ<http://lanyrd.com/2012/oscon/swrcz/>.) > >>> >> >> >> > > >>> >> >> >> > I would appreciate it if you could share measurements, > results, > >>> or > >>> >> >> even > >>> >> >> >> > anecdotal evidence along these lines. How have you avoided > the > >>> >> "let's > >>> >> >> use > >>> >> >> >> > Rabbit MQ because everybody else does it" route when solving > >>> >> problems > >>> >> >> for > >>> >> >> >> > which Kafka is a better fit? > >>> >> >> >> > > >>> >> >> >> > Thanks, > >>> >> >> >> > > >>> >> >> >> > -Dragos > >>> >> >> >> > > >>> >> >> >> > >>> >> >> > >>> >> > >>> >
