Re: Merging all Spark Streaming RDDs to one RDD

2014-06-12 Thread unorthodox . engineers
eaming apps, we tend to build up state that cannot be regenerated, and hadoop files don't seem to be the best solution. Jeremy Lee BCompSci (Hons) The Unorthodox Engineers > On 10 Jun 2014, at 11:00 am, Henggang Cui wrote: > > Hi, > > I'm wondering whether it&#

Re: Spark-Streaming window processing

2014-06-12 Thread unorthodox . engineers
To get the streaming latency I just look at the stats on the application drivers UI webpage. I don't know if you can do that programatically, but you could CURL and parse the page if you had to. Jeremy Lee BCompSci (Hons) The Unorthodox Engineers > On 10 Jun 2014, at 3:36 pm, Yi

Re: creating new ami image for spark ec2 commands

2014-06-12 Thread unorthodox . engineers
wasn't obvious they were any better with EC2, and a hundred times the complexity. I expect most AWS-heavy companies have a full time person just managing AMIs. They are that annoying. It's what makes Cloudera attractive. Jeremy Lee BCompSci (Hons) The Unorthodox Engineers > On 6