Depends how important being able to access every single bit of the messages are, right down to looking at what is on the disk. It's very important to us, we need that control. Ability to scale throughout as needed is also important - too important to do anything but run it ourselves. All these reasons meant we use Kafka.
As for operational overhead we have found Kafka 0.72 to be very stable and easy to manage. If *only* other components were as easy. :-) Philip On Jun 14, 2013, at 7:45 AM, James Newhaven <james.newha...@gmail.com> wrote: > Hi, > > I have a system that needs to process tens of thousands of user events per > second. > > I've looked at both Kafka and Amazon SNS. > > Using SNS would mean I can avoid the operational overhead of maintaining > Kafka and Zookeeper installations and monitoring. I also wouldn't need to > worry about storage and network infrastructure. > > On the other hand, SNS could be very costly, particularly since I must > also use SQS to persist messages until consumed. > > Has anyone else considered using Amazon SNS instead of Kafka? I would be > interested to hear people's opinions on both. > > Thanks, > James
