Re: [DISCUSS] Removing or changing maintainer process

+1, some maintainers are hard to find 2016-05-19 9:03 GMT-07:00 Imran Rashid : > +1 (binding) on removal of maintainers > > I dont' have a strong opinion yet on how to have a system for finding the > right reviewers. I agree it would be nice to have something to help you > find reviewers, though

Re: java.lang.OutOfMemoryError: Unable to acquire bytes of memory

@Nezih, can you try again after setting `spark.memory.useLegacyMode` to true? Can you still reproduce the OOM that way? 2016-03-21 10:29 GMT-07:00 Nezih Yigitbasi : > Hi Spark devs, > I am using 1.6.0 with dynamic allocation on yarn. I am trying to run a > relatively big application with 10s of j

Re: [VOTE] Release Apache Spark 1.6.1 (RC1)

+1 2016-03-08 10:59 GMT-08:00 Yin Huai : > +1 > > On Mon, Mar 7, 2016 at 12:39 PM, Reynold Xin wrote: > >> +1 (binding) >> >> >> On Sun, Mar 6, 2016 at 12:08 PM, Egor Pahomov >> wrote: >> >>> +1 >>> >>> Spark ODBC server is fine, SQL is fine. >>> >>> 2016-03-03 12:09 GMT-08:00 Yin Yang : >>> >>

Re: Welcoming two new committers

Welcome! 2016-02-08 10:55 GMT-08:00 Bhupendra Mishra : > Congratulations to both. and welcome to group. > > On Mon, Feb 8, 2016 at 10:45 PM, Matei Zaharia > wrote: > >> Hi all, >> >> The PMC has recently added two new Spark committers -- Herman van Hovell >> and Wenchen Fan. Both have been heavi

Re: [VOTE] Release Apache Spark 1.6.0 (RC4)

+1 2015-12-22 12:43 GMT-08:00 Reynold Xin : > +1 > > > On Tue, Dec 22, 2015 at 12:29 PM, Michael Armbrust > wrote: > >> I'll kick the voting off with a +1. >> >> On Tue, Dec 22, 2015 at 12:10 PM, Michael Armbrust < >> mich...@databricks.com> wrote: >> >>> Please vote on releasing the following c

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

D/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure: > bad substitution > - > > ​Regards, > Sebasti

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

@syepes I just run Spark 1.6 (881f254) on YARN with Hadoop 2.4.0. I was able to run a simple application in cluster mode successfully. Can you verify whether the org.apache.spark.yarn.ApplicationMaster class exists in your assembly jar? jar -tf assembly.jar | grep ApplicationMaster -Andrew 20

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

+1 Mesos cluster mode regression in RC2 is now fixed (SPARK-12345 / PR10332 ). Also tested on standalone client and cluster mode. No problems. 2015-12-16 15:16 GMT-08:00 Rad Gruchalski : > I also not

Re: [VOTE] Release Apache Spark 1.6.0 (RC2)

+1 Ran PageRank on standalone mode with 4 nodes and noticed a speedup after the specific commits that were in RC2 but not RC1: c247b6a Dec 10 [SPARK-12155][SPARK-12253] Fix executor OOM in unified memory management 05e441e Dec 9 [SPARK-12165][SPARK-12189] Fix bugs in eviction of storage memory by

Re: Removing the Mesos fine-grained mode

@Jerry Lam Can someone confirm if it is true that dynamic allocation on mesos "is > designed to run one executor per slave with the configured amount of > resources." I copied this sentence from the documentation. Does this mean > there is at most 1 executor per node? Therefore, if you have a big

Re: Concurrency issue in SQLExecution.withNewExecutionId

@Olivier, did you use scala's parallel collections by any chance? If not, what form of concurrency were you using? 2015-09-10 13:01 GMT-07:00 Andrew Or : > Thanks for reporting this, I have filed > https://issues.apache.org/jira/browse/SPARK-10548. > > 2015-09-10 9:09 GMT-07:

Re: Concurrency issue in SQLExecution.withNewExecutionId

Thanks for reporting this, I have filed https://issues.apache.org/jira/browse/SPARK-10548. 2015-09-10 9:09 GMT-07:00 Olivier Toupin : > Look at this code: > > > https://github.com/apache/spark/blob/branch-1.5/sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala#L42 > > and >

Re: Flaky test in DAGSchedulerSuite?

(merge into master, thanks for the quick fix Pete). 2015-09-04 15:58 GMT-07:00 Cheolsoo Park : > Thank you Pete! > > On Fri, Sep 4, 2015 at 1:40 PM, Pete Robbins wrote: > >> raised https://issues.apache.org/jira/browse/SPARK-10454 and PR >> >> On 4 September 2015 at 21:24, Pete Robbins wrote: >

Re: What is the difference between SlowSparkPullRequestBuilder and SparkPullRequestBuilder?

Hi Yu, As it stands today, they are identical except for trigger mechanism. When you say "test this please" or push a commit, SparkPullRequestBuilder is the one that's running the tests. SlowSparkPullRequestBuilder, however, is not used by default, but only triggered when you say "slow test please

Re: Dynamic resource allocation in Standalone mode

Hi Ray, In standalone mode, you have this thing called the SparkDeploySchedulerBackend, which has this thing called the AppClient. This is the thing on the driver side that already talks to the Master to register the application. As for dynamic allocation in standalone mode, I literally *just* cr

Re: [VOTE] Release Apache Spark 1.4.1 (RC4)

+1 2015-07-09 10:26 GMT-07:00 Michael Armbrust : > +1 > > On Thu, Jul 9, 2015 at 10:07 AM, Mark Hamstra > wrote: > >> +1 >> >> On Wed, Jul 8, 2015 at 10:55 PM, Patrick Wendell >> wrote: >> >>> Please vote on releasing the following candidate as Apache Spark version >>> 1.4.1! >>> >>> This relea

Re: [VOTE] Release Apache Spark 1.4.1 (RC3)

@Sean You actually need to run HiveSparkSubmitSuite with `-Phive` and `-Phive-thriftserver`. The MissingRequirementsError is just complaining that it can't find the right classes. The other one (DataFrameStatSuite) is a little more concerning. 2015-07-08 10:43 GMT-07:00 Pradeep Bashyal : > Hi Shi

Re: [VOTE] Release Apache Spark 1.4.1 (RC3)

+1 Verified that the previous blockers SPARK-8781 and SPARK-8819 are now resolved. 2015-07-07 12:06 GMT-07:00 Patrick Wendell : > Please vote on releasing the following candidate as Apache Spark version > 1.4.1! > > This release fixes a handful of known issues in Spark 1.4.0, listed here: > http

Re: Can not build master

gt;> Java version: 1.8.0_45, vendor: Oracle Corporation >> Java home: /home/hbase/jdk1.8.0_45/jre >> Default locale: en_US, platform encoding: UTF-8 >> OS name: "linux", version: "2.6.32-504.el6.x86_64", arch: "amd64", >> family: "unix" >

Re: Can not build master

48 for *some* combinations of build >> > profiles. >> > >> > 2. MSHADE-148 can be worked around by disabling building of >> > "dependency reduced poms" because then the buggy code path is >> > circumvented. Andrew Or did this in a patch on the

Re: [VOTE] Release Apache Spark 1.4.1

@Sean I believe that is a real issue. I have submitted a patch to fix it: https://github.com/apache/spark/pull/7193. Unfortunately this would mean we need to cut a new RC to include it. When we do so we should also do another careful pass over the commits that are merged since the first RC. -1 20

Re: [VOTE] Release Apache Spark 1.4.1

Hi Ted, We haven't observed a StreamingContextSuite failure on our test infrastructure recently. Given that we cannot reproduce it even locally it is unlikely that this uncovers a real bug. Even if it does I would not block the release on it because many in the community are waiting for a few impo

Re: Welcoming some new committers

Welcome! 2015-06-20 7:30 GMT-07:00 Debasish Das : > Congratulations to All. > > DB great work in bringing quasi newton methods to Spark ! > > On Wed, Jun 17, 2015 at 3:18 PM, Chester Chen > wrote: > >> Congratulations to All. >> >> DB and Sandy, great works ! >> >> >> On Wed, Jun 17, 2015 at 3:1

Re: [VOTE] Release Apache Spark 1.4.0 (RC4)

+1 (binding) Ran the same tests I did for RC3: Tested the standalone cluster mode REST submission gateway - submit / status / kill Tested simple applications on YARN client / cluster modes with and without --jars Tested python applications on YARN client / cluster modes with and without --py-file

Re: [VOTE] Release Apache Spark 1.4.0 (RC3)

+1 (binding) Tested the standalone cluster mode REST submission gateway - submit / status / kill Tested simple applications on YARN client / cluster modes with and without --jars Tested python applications on YARN client / cluster modes with and without --py-files* Tested dynamic allocation on YAR

Re: please use SparkFunSuite instead of ScalaTest's FunSuite from now on

It will be within the next few days 2015-06-01 9:17 GMT-07:00 Reynold Xin : > I don't think so. > > > On Monday, June 1, 2015, Steve Loughran wrote: > >> Is this backported to branch 1.3? >> >> On 31 May 2015, at 00:44, Reynold Xin wrote: >> >> FYI we merged a patch that improves unit test l

Re: [VOTE] Release Apache Spark 1.4.0 (RC2)

-1 Found a new blocker SPARK-7864 that is being resolved by https://github.com/apache/spark/pull/6419. 2015-05-26 11:32 GMT-07:00 Shivaram Venkataraman : > +1 > > Tested the SparkR binaries using a Standalone Hadoop 1 cluster, a YARN > Hadoop 2.

Re: Recent Spark test failures

gt; Cheers > > On Wed, May 6, 2015 at 5:41 PM, Andrew Or wrote: > >> Dear all, >> >> I'm sure you have all noticed that the Spark tests have been fairly >> unstable recently. I wanted to share a tool that I use to track which >> tests >> have been fai

Re: Build fail...

Thanks for pointing this out. I reverted that commit. 2015-05-08 19:01 GMT-07:00 Ted Yu : > Looks like you're right: > > > https://amplab.cs.berkeley.edu/jenkins/view/Spark/job/Spark-1.3-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=centos/427/console > > [error] > /home/jenkins/workspace/Spark

Recent Spark test failures

Dear all, I'm sure you have all noticed that the Spark tests have been fairly unstable recently. I wanted to share a tool that I use to track which tests have been failing most often in order to prioritize fixing these flaky tests. Here is an output of the tool. This spreadsheet reports the top 1

Re: PR 5140

Hey Nathan, thanks for bringing this up I will look at this within the next day or two. 2015-04-08 8:03 GMT-07:00 Nathan Kronenfeld : > Could I get someone to look at PR 5140 please? It's been languishing more > than two weeks. >

Re: multi-line comment style

In my experience I find it much more natural to use // for short multi-line comments (2 or 3 lines), and /* */ for long multi-line comments involving one or more paragraphs. For short multi-line comments, there is no reason not to use // if it just so happens that your first line exceeded 100 chara

Re: Spark client reconnect to driver in yarn-cluster deployment mode

Hi Preeze, > Is there any designed way that the client connects back to the driver (still running in YARN) for collecting results at a later stage? No, there is not support built into Spark for this. For this to happen seamlessly the driver will have to start a server (pull model) or send the res

Re: Unit tests in < 5 minutes

@Patrick and Josh actually we went even further than that. We simply disable the UI for most tests and these used to be the single largest source of port conflict.

Re: keeping PR titles / descriptions up to date

I realize we're not voting, but +1 to this proposal since commit messages can't be changed whereas JIRA issues can always be updated after the fact. 2014-12-02 13:05 GMT-08:00 Patrick Wendell : > Also a note on this for committers - it's possible to re-word the > title during merging, by just run

Announcing Spark 1.1.1!

I am happy to announce the availability of Spark 1.1.1! This is a maintenance release with many bug fixes, most of which are concentrated in the core. This list includes various fixes to sort-based shuffle, memory leak, and spilling issues. Contributions from this release came from 55 developers.

Re: [VOTE] Release Apache Spark 1.2.0 (RC1)

+1. I also tested on Windows just in case, with jars referring other jars and python files referring other python files. Path resolution still works. 2014-12-02 10:16 GMT-08:00 Jeremy Freeman : > +1 (non-binding) > > Installed version pre-built for Hadoop on a private HPC > ran PySpark shell w/ i

[VOTE][RESULT] Release Apache Spark 1.1.1 (RC2)

The vote passes unanimously with 4 binding +1 votes, 5 non-binding +1 votes, and no +0 or -1 votes. The final release will be posted in the next 48 hours. Thanks to everyone who voted. -Andrew +1: Andrew Or* Xiangrui Meng* Krishna Sankar Matei Zaharia* Sean Owen Anant Asthana Marcelo Vanzin

Re: [VOTE] Release Apache Spark 1.1.1 (RC2)

I will start with a +1 2014-11-19 14:51 GMT-08:00 Andrew Or : > Please vote on releasing the following candidate as Apache Spark version 1 > .1.1. > > This release fixes a number of bugs in Spark 1.1.0. Some of the notable > ones are > - [SPARK-3426] Sort-based shuffle compre

[VOTE][RESULT] Release Apache Spark 1.1.1 (RC1)

s issue has been found over there: > > > http://apache-spark-user-list.1001560.n3.nabble.com/Issues-with-partitionBy-FetchFailed-td14760.html > > I wanted to make sure whether 1.1.1 does not have the same bug...-1 from > me till we figure out the root cause... > > Thanks

Re: [VOTE] Release Apache Spark 1.1.1 (RC1)

This seems like a legitimate blocker. We will cut another RC to include the revert. 2014-11-16 17:29 GMT-08:00 Kousuke Saruta : > Now I've finished to revert for SPARK-4434 and opened PR. > > > (2014/11/16 17:08), Josh Rosen wrote: > >> -1 >> >> I found a potential regression in 1.1.1 related to

Re: [VOTE] Release Apache Spark 1.1.1 (RC1)

e/spark/commit/2b7ab814f9bde65ebc57ebd04386e56c97f06f4a#diff-7bfd8d7c8cbb02aa0023e4c3497ee832 > ). > Consider back-porting it if other reasons arise, but this is specific > to tests and to Java 8. > > On Thu, Nov 13, 2014 at 8:01 PM, Andrew Or wrote: > > Yeah, this seems to be som

Re: [VOTE] Release Apache Spark 1.1.1 (RC1)

. rdd operations OK >> > 2.6. recommendation OK >> > 2.7. Good work ! In 1.1.0, there was an error and my program used to >> hang >> > (over memory allocation) consistently running validation using >> itertools, >> > compute optimum rank, lambda,n

Re: [VOTE] Release Apache Spark 1.1.1 (RC1)

I will start the vote with a +1 2014-11-12 20:34 GMT-08:00 Andrew Or : > Please vote on releasing the following candidate as Apache Spark version 1 > .1.1. > > This release fixes a number of bugs in Spark 1.1.0. Some of the notable > ones are > - [SPARK-3426] Sort-based s

Re: Spark 1.1.1 release

(Tonight at midnight being in PST 12am on 11/11) 2014-11-10 14:17 GMT-08:00 Andrew Or : > Hi everyone, > > I am the release manager for 1.1.1, and I am preparing to cut a release > tonight at midnight. 1.1.1 is a maintenance release which will ship several > important bug fixes to

Spark 1.1.1 release

Hi everyone, I am the release manager for 1.1.1, and I am preparing to cut a release tonight at midnight. 1.1.1 is a maintenance release which will ship several important bug fixes to users of Spark 1.1. Many users are waiting for these fixes so I would like to release it as soon as possible. At

Re: [VOTE] Designating maintainers for some Spark components

in > >> > a reasonable time period (say 2 weeks), other committers can merge the > >> > patch, and the PMC will want to discuss adding another maintainer. > >> > > > >> > > If you'd like to see examples for this model, check out the > followin

Re: Spark shuffle consolidateFiles performance degradation numbers

Hey Matt, There's some prior work that compares consolidation performance on some medium-scale workload: http://www.cs.berkeley.edu/~kubitron/courses/cs262a-F13/projects/reports/project16_report.pdf There we noticed about 2x performance degradation in the reduce phase on ext3. I am not aware of a

Re: Spark authenticate enablement

Technology Laboratory in Beijing > > -- > [image: 2D barcode - encoded with contact information] *Phone: > *86-10-82452683 > > * E-mail:* *liuj...@cn.ibm.com* > [image: IBM] > > BLD 28,ZGC Software Park > No.8 Rd.Dong Bei Wang West, Dist.Haidian Beijing

Re: Spark authenticate enablement

Hi Jun, You can still set the authentication variables through `spark-env.sh`, by exporting SPARK_MASTER_OPTS, SPARK_WORKER_OPTS, SPARK_HISTORY_OPTS etc to include "-Dspark.auth.{...}". There is an open pull request that allows these processes to also read from spark-defaults.conf, but this is not

Re: [VOTE] Release Apache Spark 1.1.0 (RC4)

+1 Tested on Yarn and Windows. Also verified that standalone cluster mode is now fixed. 2014-09-03 1:25 GMT-07:00 Xiangrui Meng : > +1. Tested some MLlib example code. > > For default changes, maybe it is useful to mention the default > broadcast factory changed to torrent. > > On Wed, Sep 3, 20

Re: [VOTE] Release Apache Spark 1.1.0 (RC3)

+1. Tested all the basic applications under both deploy modes (where applicable) in the following environments: - locally on OSX 10.9 - locally on Windows 8.1 - standalone cluster - yarn cluster built with Hadoop 2.4 >From this front I have observed no regressions, and verified that standalone-cl

Re: Welcoming two new committers

gt;> Thank you for inviting me to be a committer. I look forward to working >> >> with everyone to ensure the continued success of the Spark project. >> >> >> >> Thanks! >> >> Joey >> >> >> >> >> >> >> &g

Re: replacement for SPARK_JAVA_OPTS

Ah, great to know this is already being fixed. Thanks Patrick, I have marked my JIRA as a duplicate. 2014-08-07 21:42 GMT-07:00 Patrick Wendell : > Andrew - I think your JIRA may duplicate existing work: > https://github.com/apache/spark/pull/1513 > > > On Thu, Aug 7, 2014 at 7:5

Re: replacement for SPARK_JAVA_OPTS

/SPARK-2921. This is a somewhat serious limitation and we will try to fix this for 1.1. -Andrew 2014-08-07 19:42 GMT-07:00 Andrew Or : > Thanks Marcelo, I have moved the changes to a new PR to describe the > problems more clearly: https://github.com/apache/spark/pull/1845 > > @Gary Ye

Re: replacement for SPARK_JAVA_OPTS

Thanks Marcelo, I have moved the changes to a new PR to describe the problems more clearly: https://github.com/apache/spark/pull/1845 @Gary Yeah, the goal is to get this into 1.1 as a bug fix. 2014-08-07 17:30 GMT-07:00 Gary Malouf : > Can this be cherry-picked for 1.1 if everything works out?

Re: [VOTE] Release Apache Spark 1.0.2 (RC1)

+1 Tested on standalone and yarn clusters 2014-07-28 14:59 GMT-07:00 Tathagata Das : > Let me add my vote as well. > Did some basic tests by running simple projects with various Spark > modules. Tested checksums. > > +1 > > On Sun, Jul 27, 2014 at 4:52 PM, Matei Zaharia > wrote: > > +1 > > > >

Re: [VOTE] Release Apache Spark 0.9.2 (RC1)

+1, tested on standalone cluster and ran spark shell, pyspark and SparkPi 2014-07-18 0:03 GMT-07:00 Patrick Wendell : > +1 > > - Looked through the release commits > - Looked through JIRA issues > - Ran the audit tests (one issue with the maven app test, but was also > an issue with 0.9.1 so I t

Re: How pySpark works?

Hi Egor, Here are a few answers to your questions: 1) Python needs to be installed on all machines, but not pyspark. The way the executors get the pyspark code depends on which cluster manager you use. In standalone mode, your executors need to have the actual python files in their working direct

Re: [VOTE] Release Apache Spark 1.0.1 (RC2)

+1, verified that the UI bug is in fact fixed in https://github.com/apache/spark/pull/1255. 2014-07-05 20:01 GMT-07:00 Soren Macbeth : > +1 > > > On Sat, Jul 5, 2014 at 7:41 PM, Krishna Sankar > wrote: > > > +1 > > > >- Compiled rc2 w/ CentOS 6.5, Yarn,Hadoop 2.2.0 - successful > >- Smo

Re: [VOTE] Release Apache Spark 1.0.1 (RC1)

(Forgot to mention, that UI bug is not in Spark 1.0.0, so it is technically a regression) 2014-06-27 15:42 GMT-07:00 Andrew Or : > There is an issue with the SparkUI: the storage page continues to display > RDDs that are dropped from memory. This is fixed in > https://github.com/apa

Re: [VOTE] Release Apache Spark 1.0.1 (RC1)

There is an issue with the SparkUI: the storage page continues to display RDDs that are dropped from memory. This is fixed in https://github.com/apache/spark/commit/21e0f77b6321590ed86223a60cdb8ae08ea4057f but is not part of this RC. 2014-06-27 11:18 GMT-07:00 Matei Zaharia : > +1 > > Tested it

Re: [VOTE] Release Apache Spark 1.0.0 (RC10)

+1 2014-05-20 13:13 GMT-07:00 Tathagata Das : > Please vote on releasing the following candidate as Apache Spark version > 1.0.0! > > This has a few bug fixes on top of rc9: > SPARK-1875: https://github.com/apache/spark/pull/824 > SPARK-1876: https://github.com/apache/spark/pull/819 > SPARK-1878

Re: [VOTE] Release Apache Spark 1.0.0 (rc9)

+1 2014-05-17 8:53 GMT-07:00 Mark Hamstra : > +1 > > > On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell >wrote: > > > I'll start the voting with a +1. > > > > On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell > > wrote: > > > Please vote on releasing the following candidate as Apache Spark >

(test)

Apache has been having some problems lately. Do you guys see this message?

Re: [VOTE] Release Apache Spark 1.0.0 (rc5)

+1 2014-05-13 6:49 GMT-07:00 Sean Owen : > On Tue, May 13, 2014 at 9:36 AM, Patrick Wendell > wrote: > > The release files, including signatures, digests, etc. can be found at: > > http://people.apache.org/~pwendell/spark-1.0.0-rc5/ > > Good news is that the sigs, MD5 and SHA are all correct. >

Re: thoughts on spark_ec2.py?

Hi Art, First of all thanks a lot for your PRs. We are currently in the middle of all the Spark 1.0 release so most of us are swamped with the more core features. To answer your questions: 1. Neither. We welcome changes from developers for all components of Spark, including the EC2 scripts. Once

Re: [VOTE] Release Apache Spark 0.9.1 (RC3)

+1 tested on OSX On Mon, Mar 31, 2014 at 4:33 PM, Kevin Markey wrote: > I had specifically requested that the ASM shading be included in the RC, > hence my testing focused on that, but I ran other tests as well. Tested > with a build of our project, running one of our applications from that > b