Spark 1.6.1 packages on S3 corrupt?

2016-04-12 Thread Augustus Hong
s 1 tar: Error is not recoverable: exiting now mv: missing destination file operand after `spark' Try `mv --help' for more information. -- [image: Branch] <https://branch.io/?bmp=xink-sig> Augustus Hong Software Engineer

Spark Streaming Running Out Of Memory in 1.5.0.

2015-12-03 Thread Augustus Hong
nceToXML$2.apply(Utility.scala:256) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > at scala.xml.Utility$.sequenceToXML(Utility.scala:256) > at scala.xml.Utility$.serialize(Utility.scala:2

Re: Upgrading Spark in EC2 clusters

2015-11-12 Thread Augustus Hong
rade the cluster in place, you'll probably have to do >> that manually. Otherwise, perhaps spark-ec2 is not the right tool, and >> instead you want one of those "grown-up" management tools like Ansible >> which can be setup to allow in-place upgrades. That'll

Upgrading Spark in EC2 clusters

2015-11-11 Thread Augustus Hong
ficient to spin up a new cluster every time we need to upgrade. Thanks, Augustus -- [image: Branch Metrics mobile deep linking] <http://branch.io/>* Augustus Hong* Data Analytics | Branch Metrics m 650-391-3369 | e augus...@branch.io

Re: Multiple Spark Streaming Jobs on Single Master

2015-10-23 Thread Augustus Hong
t;> # Run on a Spark standalone cluster in client deploy mode >> ./bin/spark-submit \ >> --class org.apache.spark.examples.SparkPi \ >> --master spark://207.184.161.138:7077 \ >> --executor-memory 20G \ >> *--total-executor-cores 100 \* >> /path/to/exam

Multiple Spark Streaming Jobs on Single Master

2015-10-19 Thread Augustus Hong
only one job will be run, even though there are a lot of idle cores. Best, Augustus -- [image: Branch Metrics mobile deep linking] <http://branch.io/>* Augustus Hong* Data Analytics | Branch Metrics m 650-391-3369 | e augus...@branch.io

Re: Adding / Removing worker nodes for Spark Streaming

2015-09-28 Thread Augustus Hong
t; you can run into unexpected issues (e.g. authentication). > > On Mon, Sep 28, 2015 at 1:27 PM, Augustus Hong > wrote: > >> Hey all, >> >> I'm evaluating using Spark Streaming with Kafka direct streaming, and I >> have a couple of questions: >> >&g

Adding / Removing worker nodes for Spark Streaming

2015-09-28 Thread Augustus Hong
lt;http://branch.io/>* Augustus Hong* Data Analytics | Branch Metrics m 650-391-3369 | e augus...@branch.io