Re: How to use multi thread in RDD map function ?

Thank for your advise, I have tried your recommended configuration, but SPARK_WORKER_CORES=32 SPARK_WORKER_INSTANCES=1 still work better, in the offical spark-standalone, about the parameter 'SPARK_WORKER_INSTANCES' /You can make this more than 1 if you have have very large machines and would li

Re: jenkins downtime/system upgrade wednesday morning, 730am PDT

reminder: this is happening tomorrow morning. i will be putting jenkins in to quiet mode at ~7am, and then doing the upgrade once any stray builds finish. On Mon, Sep 29, 2014 at 1:43 PM, shane knapp wrote: > happy monday, everyone! > > remember a few weeks back when i upgraded jenkins, and un

Re: Hyper Parameter Optimization Algorithms

Hi Lochana, We are indeed working on hyperparameter optimization as part of the MLbase project. We are writing a paper about this work right now, and also plan to eventually open-source our code. -Ameet On Mon, Sep 29, 2014 at 7:48 PM, Debasish Das wrote: > You should

Re: jenkins downtime/system upgrade wednesday morning, 730am PDT

https://issues.apache.org/jira/browse/SPARK-3745 On Tue, Sep 30, 2014 at 10:22 AM, shane knapp wrote: > (this time, reply to all) > > nice catch. there's a bug in spark/dev/check-license, which i've > confirmed from the CLI. i'll open a bug and PR to fix it. > > On Mon, Sep 29, 2014 at 8:00 PM

Re: jenkins downtime/system upgrade wednesday morning, 730am PDT

(this time, reply to all) nice catch. there's a bug in spark/dev/check-license, which i've confirmed from the CLI. i'll open a bug and PR to fix it. On Mon, Sep 29, 2014 at 8:00 PM, Nan Zhu wrote: > Just noticed these lines in the jenkins log > > =

Re: Cluster tests failing

Try to build the assembly jar first. ClusterSuite uses local-cluster mode, which requires the assembly jar. -Xiangrui On Tue, Sep 30, 2014 at 8:23 AM, Debasish Das wrote: > I have done mvn clean several times... > > Consistently all the mllib tests that are using > LocalClusterSparkContext.scala,

Re: Cluster tests failing

I have done mvn clean several times... Consistently all the mllib tests that are using LocalClusterSparkContext.scala, they fail !

Re: Spark SQL question: why build hashtable for both sides in HashOuterJoin?

Hi Haopu, How about full outer join? One hash table may not be efficient for this case. Liquan On Mon, Sep 29, 2014 at 11:47 PM, Haopu Wang wrote: >Hi, Liquan, thanks for the response. > > > > In your example, I think the hash table should be built on the "right" > side, so Spark can i

Re: Cluster tests failing

first run cmd mvn clean, then try again On 2014/9/30 16:25, Debasish Das wrote: org.apache.spark.SparkException: Job aborted due to stage failure: Master - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For addi

Cluster tests failing

Hi, Inside mllib I am running tests using: mvn -Dhadoop.version=2.3.0-cdh5.1.0 -Phadoop-2.3 -Pyarn install The locat tests run fine but cluster tests are failing.. LBFGSClusterSuite: - task size should be small *** FAILED *** org.apache.spark.SparkException: Job aborted due to stage failure