Re: Flume NG and S3

2014-07-08 Thread Dennis Waldron
We've (www.plumbee.co.uk) have been using Flume NG in combination with S3 successfully for about 10 months now without any major issues. Our whole tech stack is hosted in AWS and on average we process 450 million events per day, (approx 120GB) all of which is collected via Flume, aggregated using t

Flume NG SQS Source Plugin

2014-07-11 Thread Dennis Waldron
Hi All, If anyone's interested in consuming messages from Amazon's Simple Queue Service (SQS), I've open sourced a plugin for it available at https://github.com/plumbee/flume-sqs-source Feel free to check it out, comment, suggest RFE's (no guarantees they'll be implemented) Cheers, Dennis Data