Re: Compie error with XML elements

In some cases IntelliJ's Scala compiler can't compile valid Scala source files. Hopefully they fix (or have fixed) this in a newer version. - Patrick On Fri, Aug 29, 2014 at 11:38 AM, Yi Tian wrote: > Hi, Devl! > > I got the same problem. > > You can try to upgrade your scala plugins to 0.41.2

Need to check approach for continuing development on Spark

Hi, We are developing an app in Spring in which we are using Cassandra and calling datastax api's from Java to query it. The internal library is responsible for calling cassandra and other data sources like RDS. We are calling several client API's from Spark provided by the client-jar to perform c

Re: [VOTE] Release Apache Spark 1.1.0 (RC2)

+1. Validated several custom analysis pipelines on a private cluster in standalone mode. Tested new PySpark support for arbitrary Hadoop input formats, works great! -- Jeremy -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/VOTE-Release-Apache-Spark-1-

Re: [VOTE] Release Apache Spark 1.1.0 (RC2)

Just noticed one thing: although --with-hive is deprecated by -Phive, make-distribution.sh still relies on $SPARK_HIVE (which was controlled by --with-hive) to determine whether to include datanucleus jar files. This means we have to do something like SPARK_HIVE=true ./make-distribution.sh ... to e

Re: [VOTE] Release Apache Spark 1.1.0 (RC2)

In our internal projects we use this bit of code in the maven pom to create a properties file with build information (sorry for the messy indentation). Then we have code that reads this property file somewhere and provides that info. This should make it easier to not have to change version numbers

Re: [VOTE] Release Apache Spark 1.1.0 (RC2)

Sounds good. As an FYI, we had this problem with the 1.0.2 release . Is there perhaps some kind of automated check we can make to catch this for us in the future? Where would it go? On Fri, Aug 29, 2014 at 2:18 PM, Patrick Wendell wrote: > Oh da

Re: Compie error with XML elements

Hi, Devl! I got the same problem. You can try to upgrade your scala plugins to 0.41.2 It works on my mac. On Aug 12, 2014, at 15:19, Devl Devel wrote: > When compiling the master checkout of spark. The Intellij compile fails > with: > >Error:(45, 8) not found: value $scope > >

Re: [VOTE] Release Apache Spark 1.1.0 (RC2)

Oh darn - I missed this update. GRR, unfortunately I think this means I'll need to cut a new RC. Thanks for catching this Nick. On Fri, Aug 29, 2014 at 10:18 AM, Nicholas Chammas wrote: > [Let me know if I should be posting these comments in a different thread.] > > Should the default Spark versi

new jenkins plugin installed and ready for use

i have always found the 'Rebuild' plugin super useful: https://wiki.jenkins-ci.org/display/JENKINS/Rebuild+Plugin this is installed and enables. enjoy! shane

Re: [VOTE] Release Apache Spark 1.1.0 (RC2)

[Let me know if I should be posting these comments in a different thread.] Should the default Spark version in spark-ec2 be updated for this release? Nick ​ On Fri, Aug 29, 2014 at 12:55 PM, Pat

Re: Jira tickets for starter tasks

Added you; you should be set! If anyone else wants me to add them, please email me off-list so that we don’t end up flooding the dev list with replies. Thanks! On August 29, 2014 at 10:03:41 AM, Ron's Yahoo! (zlgonza...@yahoo.com) wrote: Hi Josh, Can you add me as well? Thanks, Ron O

Re: Jira tickets for starter tasks

Hi Josh, Can you add me as well? Thanks, Ron On Aug 28, 2014, at 3:56 PM, Josh Rosen wrote: > A JIRA admin needs to add you to the ‘’Contributors” role group in order to > allow you to assign issues to yourself. I’ve added this email address to > that group, so you should be set! > > - Jo

Re: [VOTE] Release Apache Spark 1.1.0 (RC2)

Hey Nicholas, Thanks for this, we can merge in doc changes outside of the actual release timeline, so we'll make sure to loop those changes in before we publish the final 1.1 docs. - Patrick On Fri, Aug 29, 2014 at 9:24 AM, Nicholas Chammas wrote: > There were several formatting and typographic

Re: [VOTE] Release Apache Spark 1.1.0 (RC2)

There were several formatting and typographical errors in the SQL docs that I've fixed in this PR . Dunno if we want to roll that into the release. On Fri, Aug 29, 2014 at 12:17 PM, Patrick Wendell wrote: > Okay I'll plan to add cdh4 binary as well for

Re: [VOTE] Release Apache Spark 1.1.0 (RC2)

Okay I'll plan to add cdh4 binary as well for the final release! --- sent from my phone On Aug 29, 2014 8:26 AM, "Ye Xianjin" wrote: > We just used CDH 4.7 for our production cluster. And I believe we won't > use CDH 5 in the next year. > > Sent from my iPhone > > > On 2014年8月29日, at 14:39, Mate

Re: [VOTE] Release Apache Spark 1.1.0 (RC2)

We just used CDH 4.7 for our production cluster. And I believe we won't use CDH 5 in the next year. Sent from my iPhone > On 2014年8月29日, at 14:39, Matei Zaharia wrote: > > Personally I'd actually consider putting CDH4 back if there are still users > on it. It's always better to be inclusive,

Re: "emergency" jenkins restart, aug 29th, 730am-9am PDT -- plus a postmortem

this is done. On Fri, Aug 29, 2014 at 7:32 AM, shane knapp wrote: > reminder: this is happening right now. jenkins is currently in quiet > mode, and in ~30 minutes, will be briefly going down. > > > On Thu, Aug 28, 2014 at 1:03 PM, shane knapp wrote: > >> as with all software upgrades, some

Re: Jira tickets for starter tasks

Cheng Lian-2 wrote > You can just start the work :) Given 100+ contributors, starting work without a JIRA issue assigned to you could lead to duplication of effort by well meaning people that have no idea they are working on the same issue. This does happen and I don't think it's a good thing. Ju

Re: "emergency" jenkins restart, aug 29th, 730am-9am PDT -- plus a postmortem

reminder: this is happening right now. jenkins is currently in quiet mode, and in ~30 minutes, will be briefly going down. On Thu, Aug 28, 2014 at 1:03 PM, shane knapp wrote: > as with all software upgrades, sometimes things don't always work as > expected. > > a recent change to stapler[1],

Re: Running Spark On Yarn without Spark-Submit

Archit We are using yarn-cluster mode , and calling spark via Client class directly from servlet server. It works fine. To establish a communication channel to give further requests, It should be possible with yarn client, but not with yarn server. Yarn client mode, spark driver i

Re: [VOTE] Release Apache Spark 1.1.0 (RC2)

i suspect there are more cdh4 than cdh5 clusters. most people plan to move to cdh5 within say 6 months. On Fri, Aug 29, 2014 at 3:57 AM, Andrew Ash wrote: > FWIW we use CDH4 extensively and would very much appreciate having a > prebuilt version of Spark for it. > > We're doing a CDH 4.4 to 4.7

Re: Running Spark On Yarn without Spark-Submit

including u...@spark.apache.org. On Fri, Aug 29, 2014 at 2:03 PM, Archit Thakur wrote: > Hi, > > My requirement is to run Spark on Yarn without using the script > spark-submit. > > I have a servlet and a tomcat server. As and when request comes, it > creates a new SC and keeps it alive for the

Running Spark On Yarn without Spark-Submit

Hi, My requirement is to run Spark on Yarn without using the script spark-submit. I have a servlet and a tomcat server. As and when request comes, it creates a new SC and keeps it alive for the further requests, I ma setting my master in sparkConf as sparkConf.setMaster("yarn-cluster") but the

Re: [VOTE] Release Apache Spark 1.1.0 (RC2)

FWIW we use CDH4 extensively and would very much appreciate having a prebuilt version of Spark for it. We're doing a CDH 4.4 to 4.7 upgrade across all the clusters now and have plans for a 5.x transition after that. On Aug 28, 2014 11:57 PM, "Sean Owen" wrote: > On Fri, Aug 29, 2014 at 7:42 AM,