Welcome, Ahmed! Cool stuff! Is there a design doc or mission statement that you and Mark are working off?
-=Bill On Wed, May 21, 2014 at 11:15 PM, Ahmed Aley <ahm...@cs.umu.se> wrote: > Hi, > > I am Ahmed Ali-Eldin <https://www8.cs.umu.se/~ahmeda/>, a PhD student at > Umeå University, Sweden (It is up > north< > http://tools.wmflabs.org/geohack/geohack.php?pagename=Ume%C3%A5¶ms=63_49_30_N_20_15_50_E_type:city%2879594%29_region:SE > >:) > ). I am working with @MarkCC on integrating a distributed logging > framework with Aurora and building an analytics framework on top to analyze > the logged data. > We started off by looking into different logging frameworks > (Kafka<http://kafka.apache.org/>, > Scribe <https://github.com/facebook/scribe>, > Chukwa<https://chukwa.apache.org/>, > Suro< > http://techblog.netflix.com/2013/12/announcing-suro-backbone-of-netflixs.html > >, > Calligraphus< > http://www-conf.slac.stanford.edu/xldb2011/talks/xldb2011_tue_0940_facebookrealtimeanalytics.pdf > >and > Flume <http://flume.apache.org/>). We chose Suro coupled with Kafka out of > these for different reasons. > i- It has been built to allow scale-up and down (elastic). > ii- It is quite flexible with a Kafka sink giving us access to all Kafka > sinks. > iii- It has an S3 sink making it a suitable solution for more scenarios. > iv- I got a tip from someone I know at Netflix on Suro benchmarking > results. > v- it is an active project > > Based on the above, I have started some experiments with Suro and will be > looking at its integration with Aurora this weekend. I can not make any > statements on if Suro (coupled with kafka) is "the best" solution for > distributed logging but it looks very promising till now. I will hopefully > send some results/updates late next week. > > Best, > --Ahmed >
