Re: Does anyone meet the issue that jars under lib_managed is never downloaded ?

BTW, After I revert SPARK-7841, I can see all the jars under lib_managed/jars On Tue, Nov 17, 2015 at 2:46 PM, Jeff Zhang wrote: > Hi Josh, > > I notice the comments in https://github.com/apache/spark/pull/9575 said > that Datanucleus related jars will still be copied to lib_managed/jars. > But

Re: Are map tasks spilling data to disk?

So, in case of Sort Merge join in which a shuffle (exchange) will be performed, I have the following questions (Please correct me if my understanding is not correct): Let's say that relation A is a JSONRelation (640 MB) on the HDFS where the block size is 64MB. This will produce a Scan JSONRelatio

Add a function to support Google's Word2Vec

Hi: I have a function to load Google’s Word2Vec generated binary file and spark can use this model. If it is convenient, I'm going to open a JIRA and Pull Request. My code is: https://github.com/wangyum/spark/commit/7c80d311722d67ed4b9746537e0a21c2dc1a9670 Thanks

Re: Mesos cluster dispatcher doesn't respect most args from the submit req

Hi Jo, I agree that there's something fishy with the cluster dispatcher, I've seen some issues like that. I think it actually tries to send all properties as part of `SPARK_EXECUTOR_OPTS`, which may not be everything that's needed: https://github.com/jayv/spark/blob/mesos_cluster_params/core/src

Re: Add a function to support Google's Word2Vec

Have a look at SPARK-9484, JIRA is already there. Pull request would be good. Robin > On 17 Nov 2015, at 12:10, yuming wang wrote: > > Hi: > > > > I have a function to load Google’s Word2Vec generated binary file and spark > can use this model. If it is convenient, I'm going to open a JIR

Re: Mesos cluster dispatcher doesn't respect most args from the submit req

On Tue, Nov 17, 2015 at 5:16 AM, Iulian Dragoș wrote: > I think it actually tries to send all properties as part of > `SPARK_EXECUTOR_OPTS`, which may not be everything that's needed: > > > https://github.com/jayv/spark/blob/mesos_cluster_params/core/src/main/scala/org/apache/spark/scheduler/clus

Re: Mesos cluster dispatcher doesn't respect most args from the submit req

Hi Jo, Thanks for the links, I would expected the properties to be in scheduler properties but I need to double check. I'll be looking into these problems this week. Tim On Tue, Nov 17, 2015 at 10:28 AM, Jo Voordeckers wrote: > On Tue, Nov 17, 2015 at 5:16 AM, Iulian Dragoș > wrote: >> >> I t

Re: Does anyone meet the issue that jars under lib_managed is never downloaded ?

Is the Hive profile enabled? I think it may need to be turned on in order for those JARs to be deployed. On Tue, Nov 17, 2015 at 2:27 AM Jeff Zhang wrote: > BTW, After I revert SPARK-7841, I can see all the jars under > lib_managed/jars > > On Tue, Nov 17, 2015 at 2:46 PM, Jeff Zhang wrote: > >

Re: Mesos cluster dispatcher doesn't respect most args from the submit req

Hi Tim, I've done more forensics on this bug, see my comment here: https://issues.apache.org/jira/browse/SPARK-11327?focusedCommentId=15009843&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15009843 - Jo Voordeckers On Tue, Nov 17, 2015 at 12:01 PM, Timothy Chen

Re: Does anyone meet the issue that jars under lib_managed is never downloaded ?

Sure, hive profile is enabled. On Wed, Nov 18, 2015 at 6:12 AM, Josh Rosen wrote: > Is the Hive profile enabled? I think it may need to be turned on in order > for those JARs to be deployed. > > On Tue, Nov 17, 2015 at 2:27 AM Jeff Zhang wrote: > >> BTW, After I revert SPARK-7841, I can see al

Re: Does anyone meet the issue that jars under lib_managed is never downloaded ?

Can you file a JIRA issue to help me triage this further? Thanks! On Tue, Nov 17, 2015 at 4:08 PM Jeff Zhang wrote: > Sure, hive profile is enabled. > > On Wed, Nov 18, 2015 at 6:12 AM, Josh Rosen > wrote: > >> Is the Hive profile enabled? I think it may need to be turned on in order >> for tho

Re: Does anyone meet the issue that jars under lib_managed is never downloaded ?

Created https://issues.apache.org/jira/browse/SPARK-11798 On Wed, Nov 18, 2015 at 9:42 AM, Josh Rosen wrote: > Can you file a JIRA issue to help me triage this further? Thanks! > > On Tue, Nov 17, 2015 at 4:08 PM Jeff Zhang wrote: > >> Sure, hive profile is enabled. >> >> On Wed, Nov 18, 2015

Fwd: zeppelin (or spark-shell) with HBase fails on executor level

HI, First of all, I'm sorry if you received mail before (apache spark user mailing group). I posted this mail to user mailing group but I didn't receive any informations about resolving so I'd like to post this to dev mailing group. This link points to the thread I'm forwarding, so if you feel co