Re: spark with label nodes in yarn

Yes, of course, capacity scheduler also needs to be configured. On Wed, Dec 16, 2015 at 10:41 AM, 张志强(旺轩) wrote: > one more question , do I have to configure label for my capacity > scheduler? is this mandatory? > > > > *发件人:* AllenZ [mailto:zzq98...@alibaba-inc.com] > *发送时间:* 2015年12月16日 9:21 >

Re: spark with label nodes in yarn

one more question , do I have to configure label for my capacity scheduler? is this mandatory? 发件人: AllenZ [mailto:zzq98...@alibaba-inc.com] 发送时间: 2015年12月16日 9:21 收件人: 'Ted Yu' 抄送: 'Saisai Shao'; 'dev' 主题: Re: spark with label nodes in yarn Oops... I do use spark 1.5.0 and apache had

Re: spark with label nodes in yarn

Oops... I do use spark 1.5.0 and apache hadoop 2.6.0 (spark 1.4.1 + apache hadoop 2.6.0 is a typo), sorry Thanks, Allen 发件人: Ted Yu [mailto:yuzhih...@gmail.com] 发送时间: 2015年12月15日 22:59 收件人: 张志强(旺轩) 抄送: Saisai Shao; dev 主题: Re: spark with label nodes in yarn Please upgrade to Spar

security testing on spark ?

Hi all, Does anyone know of any effort from the community on security testing spark clusters. I.e. Static source code analysis to find security flaws Penetration testing to identify ways to compromise spark cluster Fuzzing to crash spark Thanks, Judy

​Spark 1.6 - H​ive remote metastore not working

​Hello, I am testing out the 1.6 branch (#08aa3b4) and I have just noticed that spark-shell "HiveContext" is no longer able to connect to my remote metastore. Using the same build options and configuration files with 1.5 (#0fdf554) it works. Does anyone know if there have been any mayor changes

java.lang.NoSuchMethodError while saving a random forest model Spark version 1.5

I have recently upgraded spark version but when I try to run save a random forest model using model save command I am getting nosuchmethoderror. My code works fine with 1.3x version. model.save(sc.sc(), "modelsavedir"); ERROR: org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRel

Re: [VOTE] Release Apache Spark 1.6.0 (RC2)

Thanks for the heads up. On Tue, Dec 15, 2015 at 11:40 PM, Michael Armbrust wrote: > This vote is canceled due to the issue with the incorrect version. This > issue will be fixed by https://github.com/apache/spark/pull/10317 > > We can wait a little bit for a fix to > https://issues.apache.org/

Re: [VOTE] Release Apache Spark 1.6.0 (RC2)

This vote is canceled due to the issue with the incorrect version. This issue will be fixed by https://github.com/apache/spark/pull/10317 We can wait a little bit for a fix to https://issues.apache.org/jira/browse/SPARK-12345. However if it looks like there is not an easy fix coming soon, I'm pl

Re: SparkML algos limitations question.

Hi Eugene, The maxDepth parameter exists because the implementation uses Integer node IDs which correspond to positions in the binary tree. This simplified the implementation. I'd like to eventually modify it to avoid depending on tree node IDs, but that is not yet on the roadmap. There is not

Re: BIRCH clustering algorithm

Hi Dzeno, I'm not familiar with the algorithm myself, but if you have an important use case for it, you could open a JIRA to discuss it. However, if it is a less common algorithm, I'd recommend first submitting it as a Spark package (but publicizing the package on the user list). If it gains tra

Re: [VOTE] Release Apache Spark 1.6.0 (RC2)

-1 (non-binding) Cluster mode on Mesos is broken (regression compared to 1.5.2). It seems to be related to the way SPARK_HOME is handled. In the driver logs I see: I1215 15:00:39.411212 28032 exec.cpp:134] Version: 0.25.0 I1215 15:00:39.413512 28037 exec.cpp:208] Executor registered on slave 130b

Re: spark with label nodes in yarn

Please upgrade to Spark 1.5.x 1.4.1 didn't support node label feature. Cheers On Tue, Dec 15, 2015 at 2:20 AM, 张志强(旺轩) wrote: > Hi SaiSai, > > > > OK, it make sense to me , what I need is just to schedule the executors, > AND I leave one nodemanager at least with no any labels. > > > > It’s we

Re: spark with label nodes in yarn

SPARK-6470 only supports node label expression for executors. SPARK-7173 supports node label expression for AM (will be in 1.6). If you want to schedule your whole application through label expression, you have to configure both am and executor label expression. If you only want to schedule execut

Re: spark with label nodes in yarn

SPARK-6470 was integrated to 1.5.0 release. Please use 1.5.0 or newer release. SPARK-7173 adds support for setting "spark.yarn.am.nodeLabelExpression" Cheers On Tue, Dec 15, 2015 at 1:55 AM, 张志强(旺轩) wrote: > Hi Ted, > > > > Thanks for your qu

Re: status of 2.11 support?

Please see related JIRA: https://issues.apache.org/jira/browse/SPARK-8013 This question is better suited for user mailing list. Thanks On Mon, Dec 14, 2015 at 10:29 PM, Sachin Aggarwal < different.sac...@gmail.com> wrote: > Hi, > > > adding question from user group to dev group need expert advi

Re: spark with label nodes in yarn

Hi Ted, Thanks for your quick response, but I think the link you gave it to me is more advanced feature. Yes, I noticed SPARK-6470(https://issues.apache.org/jira/browse/SPARK-6470) And I just tried for this feature with spark 1.5.0, what happened to me was I was blocked to get the YARN con

Re: spark with label nodes in yarn

Please take a look at: https://issues.apache.org/jira/browse/SPARK-7173 Cheers > On Dec 15, 2015, at 1:23 AM, 张志强(旺轩) wrote: > > Hi all, > > Has anyone tried label based scheduling via spark on yarn? I’ve tried that, > it didn’t work, spark 1.4.1 + apache hadoop 2.6.0 > > Any feedbacks are

spark with label nodes in yarn

Hi all, Has anyone tried label based scheduling via spark on yarn? I've tried that, it didn't work, spark 1.4.1 + apache hadoop 2.6.0 Any feedbacks are welcome. Thanks Allen