Re: OutOfMemory error with Spark ML 1.5 logreg example

Hi, Can you try to using save method instead of write? ex: out_df.save("path","parquet") b0c1 -- Skype: boci13, Hangout: boci.b...@gmail.com On Mon, Sep 7, 2015 at 3:

Re: Mesos + Spark

> Author: Programming Scala, 2nd Edition > <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) > Typesafe <http://typesafe.com> > @deanwampler <http://twitter.com/deanwampler> > http://polyglotprogramming.com > > On Fri, Jul 24, 2015 at 4:47 PM,

Re: Mesos + Spark

http://typesafe.com> > @deanwampler <http://twitter.com/deanwampler> > http://polyglotprogramming.com > > On Wed, Jul 22, 2015 at 3:53 AM, boci wrote: > >> Hi guys! >> >> I'm a new in mesos. I have two spark application (one streaming and one >&

Mesos + Spark

Hi guys! I'm a new in mesos. I have two spark application (one streaming and one batch). I want to run both app in mesos cluster. Now for testing I want to run in docker container so I started a simple redjack/mesos-master, but I think a lot of think unclear for me (both mesos and spark-mesos). I

spark submit debugging

Hi! I have a little problem... If I started my spark application as java app (locally) it's work like a charm, but if I start in hadoop cluster (tried spark-submit --master local[5] and --master yarn-client), but it's not working. No error, no exception, periodically run the job but nothing happen

Spark streaming with kafka

Hi guys, I using spark streaming with kafka... In local machine (start as java application without using spark-submit) it's work, connect to kafka and do the job (*). I tried to put into spark docker container (hadoop 2.6, spark 1.3.1, try spark submit wil local[5] and yarn-client too ) but I'm ou

Re: Strange ClassNotFound exeption

ep SelectionPath >521 Mon Sep 29 12:05:36 PDT 2014 akka/actor/SelectionPathElement.class > > Is the above jar in your classpath ? > > On Sat, May 23, 2015 at 5:05 PM, boci wrote: > >> Hi guys! >> >> I have a small spark application. It's query some d

Strange ClassNotFound exeption

Hi guys! I have a small spark application. It's query some data from postgres, enrich it and write to elasticsearch. When I deployed into spark container I got a very fustrating error: https://gist.github.com/b0c1/66527e00bada1e4c0dc3 Spark version: 1.3.1 Hadoop version: 2.6.0 Additional info:

Re: Standalone spark

on Windows. > > On Wed, Feb 25, 2015 at 11:05 PM, boci wrote: > > Thanks your fast answer... > > in windows it's not working, because hadoop (surprise suprise) need > > winutils.exe. Without this it's not working, but if you not set the > hadoop > > d

Re: Standalone spark

5 at 11:50 PM, Sean Owen wrote: > Spark and Hadoop should be listed as 'provided' dependency in your > Maven or SBT build. But that should make it available at compile time. > > On Wed, Feb 25, 2015 at 10:42 PM, boci wrote: > > Hi, > > > > I have

Standalone spark

Hi, I have a little question. I want to develop a spark based application, but spark depend to hadoop-client library. I think it's not necessary (spark standalone) so I excluded from sbt file.. the result is interesting. My trait where I create the spark context not compiled. The error: ... scal

Re: MLLib beginner question

ant to use in prediction? -Xiangrui > > On Mon, Dec 22, 2014 at 1:47 PM, boci wrote: > > Hi! > > > > I want to try out spark mllib in my spark project, but I got a little > > problem. I have training data (external file), but the real data com from > > anoth

MLLib beginner question

Hi! I want to try out spark mllib in my spark project, but I got a little problem. I have training data (external file), but the real data com from another rdd. How can I do that? I try to simple using same SparkContext to boot rdd (first I create rdd using sc.textFile() and after NaiveBayes.train

Re: Out of any idea

eers > > > > On Sat, Jul 19, 2014 at 2:39 PM, boci wrote: > >> Hi guys! >> >> I run out of ideas... I created a spark streaming job (kafka -> spark -> >> ES). >> If I start my app local machine (inside the editor, but connect to the >> rea

Out of any idea

Hi guys! I run out of ideas... I created a spark streaming job (kafka -> spark -> ES). If I start my app local machine (inside the editor, but connect to the real kafka and ES) the application work correctly. If I start it in my docker container (same kafka and ES, local mode (local[4]) like insid

Re: Uber jar with SBT

es has > more than 65536 files, and Java 6 has various issues with jars this > large. If possible, use Java 7 everywhere. > > https://issues.apache.org/jira/browse/SPARK-1520 > > On Sat, Jul 19, 2014 at 2:30 PM, boci wrote: > > Hi Guys, > > > > I try to create sp

Uber jar with SBT

Hi Guys, I try to create spark uber jar with sbt but I have a lot of problem... I want to use the following: - Spark streaming - Kafka - Elsaticsearch - HBase the current jar size is cca 60M and it's not working. - When I deploy with spark-submit: It's running and exit without any error - When I

sbt + idea + test

Hi guys, I want to use Elasticsearch and HBase in my spark project, I want to create a test. I pulled up ES and Zookeeper, but if I put "val htest = new HBaseTestingUtility()" to my app I got a strange exception (compilation time, not runtime). https://gist.github.com/b0c1/4a4b3f6350816090c3b5

Kafka/ES question

Hi! I try to use spark with kafka, everything is work but I found a little problem. I create a small test application which connect to real kafka cluster, send a message and read it back. It's work, but when I run my test second time (send/read) it's read the first and the second stream (maybe the

Re: ElasticSearch enrich

ion. b0c1 -- Skype: boci13, Hangout: boci.b...@gmail.com On Fri, Jun 27, 2014 at 11:31 PM, Holden Karau wrote: > Try setting the master to local[4] > > > On Fri, Jun 27, 2014 at 2:17 PM, boci wrote: > >>

Re: ElasticSearch enrich

4]? > 2) When you say breakpoint, how are you setting this break point? There is > a good chance your breakpoint mechanism doesn't work in a distributed > environment, could you instead cause a side effect (like writing to a file)? > > Cheers, > > Holden :) > > > On

Re: ElasticSearch enrich

unction (where I'm generate the writable) but it's not called) Any idea? b0c1 -- Skype: boci13, Hangout: boci.b...@gmail.com On Fri, Jun 27, 2014 at 4:53 PM

Re: ElasticSearch enrich

s with Elasticsearch & Spark go :) > > > On Thu, Jun 26, 2014 at 3:17 PM, boci wrote: > >> Wow, thanks your fast answer, it's help a lot... >> >> b0c1 >> >> >>

Re: ElasticSearch enrich

Conf) > } > > Hope that helps :) > > Cheers, > > Holden :) > > > On Thu, Jun 26, 2014 at 2:23 PM, boci wrote: > >> Thanks. I without local option I can connect with es remote, now I only >> have one problem. How can I use elasticsearch-hadoop with spark s

Re: ElasticSearch enrich

ode (just > ./bin/elasticsearch) and use the default config (host = localhost, port = > 9200). > > > On Thu, Jun 26, 2014 at 9:04 AM, boci wrote: > >> That's okay, but hadoop has ES integration. what happened if I run >> saveAsHadoopFile without hadoop (or I

Re: ElasticSearch enrich

b0c1 -- Skype: boci13, Hangout: boci.b...@gmail.com On Thu, Jun 26, 2014 at 1:20 AM, Holden Karau wrote: > > > On Wed, Jun 25, 2014 at 4:16 PM, boci wrote: > >> Hi guys, thanks the direction now I have some problem/question: >> - in local (test) mode I want to use Ela

Re: ElasticSearch enrich

Hi guys, thanks the direction now I have some problem/question: - in local (test) mode I want to use ElasticClient.local to create es connection, but in prodution I want to use ElasticClient.remote, to this I want to pass ElasticClient to mapPartitions, or what is the best practices? - my stream ou

Re: ElasticSearch enrich

ytics.com > @mayur_rustagi <https://twitter.com/mayur_rustagi> > > > > On Wed, Jun 25, 2014 at 4:12 AM, boci wrote: > >> Hi guys, >> >> I have a small question. I want to create a "Worker" class which using >> ElasticClient to make query to

Re: ElasticSearch enrich

Ok but in this case where can I store the ES connection? Or all document create new ES connection inside the worker? -- Skype: boci13, Hangout: boci.b...@gmail.com On W

ElasticSearch enrich

Hi guys, I have a small question. I want to create a "Worker" class which using ElasticClient to make query to elasticsearch. (I want to enrich my data with geo search result). How can I do that? I try to create a worker instance with ES host/port parameter but spark throw an exceptino (my class