Re: Using google cloud storage for spark big data

2014-05-05 Thread Akhil Das
Hi Aureliano, You might want to check this script out, https://github.com/sigmoidanalytics/spark_gce Let me know if you need any help around that. Thanks Best Regards On Tue, Apr 22, 2014 at 7:12 PM, Aureliano Buendia wrote: > > > > On Tue, Apr 22, 2014 at 10:50 AM, Andras Nemeth < > andras.ne

Re: Using google cloud storage for spark big data

2014-04-22 Thread Aureliano Buendia
On Tue, Apr 22, 2014 at 10:50 AM, Andras Nemeth < andras.nem...@lynxanalytics.com> wrote: > We don't have anything fancy. It's basically some very thin layer of > google specifics on top of a stand alone cluster. We basically created two > disk snapshots, one for the master and one for the workers

Re: Using google cloud storage for spark big data

2014-04-22 Thread Andras Nemeth
We don't have anything fancy. It's basically some very thin layer of google specifics on top of a stand alone cluster. We basically created two disk snapshots, one for the master and one for the workers. The snapshots contain initialization scripts so that the master/worker daemons are started on b

Re: Using google cloud storage for spark big data

2014-04-21 Thread Mayur Rustagi
Okay just commented on another thread :) I have one that I use internally. Can give it out but will need some support from you to fix bugs etc. Let me know if you are interested. Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi

Re: Using google cloud storage for spark big data

2014-04-18 Thread Aureliano Buendia
Thanks, Andras. What approach did you use to setup a spark cluster on google compute engine? Currently, there is no production-ready official support for an equivalent of spark-ec2 on gce. Did you roll your own? On Thu, Apr 17, 2014 at 10:24 AM, Andras Nemeth < andras.nem...@lynxanalytics.com> wr

Re: Using google cloud storage for spark big data

2014-04-17 Thread Andras Nemeth
Hello! On Wed, Apr 16, 2014 at 7:59 PM, Aureliano Buendia wrote: > Hi, > > Google has publisheed a new connector for hadoop: google cloud storage, > which is an equivalent of amazon s3: > > > googlecloudplatform.blogspot.com/2014/04/google-bigquery-and-datastore-connectors-for-hadoop.html > This