Amazon SNS and Kafka comparison

2013-06-14 Thread James Newhaven
Hi, I have a system that needs to process tens of thousands of user events per second. I've looked at both Kafka and Amazon SNS. Using SNS would mean I can avoid the operational overhead of maintaining Kafka and Zookeeper installations and monitoring. I also wouldn't need to worry about storage

Events stored in Apache Web Logs - how best to get these into Kafka?

2013-06-11 Thread James Newhaven
I have a lot of user event data being sent to an apache web server and written to web logs. Unfortunately I don't control this flow, but do I have access to the server the logs are being written to. I need a way to access these events as soon as they have been written to the log file and send this