Re: ./dev/run-tests failing at master

I did this and it seems to work now. sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 10 even the cpython version got upgraded. Sorry if you feel I was asking too many rudimentary questions, setting up spark has been rough without a mentor. Thanks :-D FYI, I'm working

Re: ./dev/run-tests failing at master

Thanks for the response!! Here is my configuration: flake8 --version 3.8.1 (mccabe: 0.6.1, pycodestyle: 2.6.0, pyflakes: 2.2.0) CPython 2.7.16 on Linux It seems I need to upgrade my CPython... -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

Re: ./dev/run-tests failing at master

this is the flake8 versioning from a jenkins worker: $ flake8 --version 3.6.0 (mccabe: 0.6.1, pycodestyle: 2.4.0, pyflakes: 2.0.0) CPython 3.6.8 on Linux be sure you've got all the right versions of packages in there. On Thu, May 14, 2020 at 12:19 PM suddhu wrote: > Thanks for the response Jeff

Re: ./dev/run-tests failing at master

Thanks for the response Jeff and Sean. It has been quite frustrating setting up the dev environment without any help. Its comforting to have some help finally. I've added "alias python=python3" in my bashrc. So the default python accessed is 3.7.2 I have flake8 installed in both python2 and pyt

Re: ./dev/run-tests failing at master

Thanks for the response Jeff and Sean. It has been quite frustrating setting up the dev environment without any help. Its comforting to have some help finally. I've added "alias python=python3" in my bashrc. So the default python accessed is 3.7.2 I have flake8 installed in both python2 and pyt

Re: ./dev/run-tests failing at master

Are you positive you set up your Python environment correctly? To me, those error messages look like you are running Python 2, but it should be Python 3. On Thu, May 14, 2020 at 1:34 PM Sudharshann D wrote: > Hello! ;) > > I'm new to spark development and have been trying to set up my dev > env

Re: ./dev/run-tests failing at master

Hm, flake8 works OK for me locally in master, and on Jenkins it seems. Could be a version issue? On Thu, May 14, 2020 at 1:34 PM Sudharshann D wrote: > Hello! ;) > > I'm new to spark development and have been trying to set up my dev > environment for hours without much success. :-( > > Firstly,

./dev/run-tests failing at master

Hello! ;) I'm new to spark development and have been trying to set up my dev environment for hours without much success. :-( Firstly, I'm wondering why my ./dev/run-tests fails even though in on the master branch. This is the error: flake8 checks failed: ./examples/src/main/python/sql/arrow.py

Re: Some Spark MLLIB tests failing due to some classes not being registered with Kryo

No luck running the full test suites with mvn test from the main folder or just mvn -pl mllib. Any other suggestion would be much appreciated. Thank you. 2017-11-11 12:46 GMT+00:00 Marco Gaido : > Hi Jorge, > > then try running the tests not from the mllib folder, but on Spark base > directory.

Re: Some Spark MLLIB tests failing due to some classes not being registered with Kryo

Hi Jorge, then try running the tests not from the mllib folder, but on Spark base directory. If you want to run only the tests in mllib, you can specify the project using the -pl argument of mvn. Thanks, Marco 2017-11-11 13:37 GMT+01:00 Jorge Sánchez : > Hi Marco, > > Just mvn test from the m

Some Spark MLLIB tests failing due to some classes not being registered with Kryo

Hi Dev, I'm running the MLLIB tests in the current Master branch and the following Suites are failing due to some classes not being registered with Kryo: org.apache.spark.mllib.MatricesSuite org.apache.spark.mllib.VectorsSuite org.apache.spark.ml.InstanceSuite I can solve it by registering the f

Re: Tests failing with run-tests.py SyntaxError

2016. Didn’t we run it on Python 2.6 > without any problem? > > > > ee74498de37 (Josh Rosen 2016-01-26 14:20:11 -0800 124) {m: > set(m.dependencies).intersection(modules_to_test) for m in modules_to_test}, > sort=True) > > > > Bests, > > Do

Re: Tests failing with run-tests.py SyntaxError

sen 2016-01-26 14:20:11 -0800 124) {m: set(m.dependencies).intersection(modules_to_test) for m in modules_to_test}, sort=True) Bests, Dongjoon. *From: *Hyukjin Kwon *Date: *Friday, July 28, 2017 at 7:06 AM *To: *Sean Owen *Cc: *dev *Subject: *Re: Tests failing with run-tests.p

Re: Tests failing with run-tests.py SyntaxError

sort=True) Bests, Dongjoon. *From: *Hyukjin Kwon *Date: *Friday, July 28, 2017 at 7:06 AM *To: *Sean Owen *Cc: *dev *Subject: *Re: Tests failing with run-tests.py SyntaxError Yes, that's my guess just given information here without a close look. On 28 Jul 2017 11:03 pm, "Sean

Re: Tests failing with run-tests.py SyntaxError

(Josh Rosen 2016-01-26 14:20:11 -0800 124) {m: set(m.dependencies).intersection(modules_to_test) for m in modules_to_test}, sort=True) Bests, Dongjoon. From: Hyukjin Kwon Date: Friday, July 28, 2017 at 7:06 AM To: Sean Owen Cc: dev Subject: Re: Tests failing with run-tests.py

Re: Tests failing with run-tests.py SyntaxError

Yes, that's my guess just given information here without a close look. On 28 Jul 2017 11:03 pm, "Sean Owen" wrote: I see, does that suggest that a machine has 2.6, when it should use 2.7? On Fri, Jul 28, 2017 at 2:58 PM Hyukjin Kwon wrote: > That looks appearently due to dict comprehension wh

Re: Tests failing with run-tests.py SyntaxError

I see, does that suggest that a machine has 2.6, when it should use 2.7? On Fri, Jul 28, 2017 at 2:58 PM Hyukjin Kwon wrote: > That looks appearently due to dict comprehension which is, IIRC, not > allowed in Python 2.6.x. I checked the release note for sure before - > https://issues.apache.org/

Re: Tests failing with run-tests.py SyntaxError

That looks appearently due to dict comprehension which is, IIRC, not allowed in Python 2.6.x. I checked the release note for sure before - https://issues.apache.org/jira/browse/SPARK-20149 On 28 Jul 2017 9:56 pm, "Sean Owen" wrote: > File "./dev/run-tests.py", line 124 > {m: set(m.dependen

Tests failing with run-tests.py SyntaxError

File "./dev/run-tests.py", line 124 {m: set(m.dependencies).intersection(modules_to_test) for m in modules_to_test}, sort=True) ^ SyntaxError: invalid syntax It seems like tests are failing intermittently with this type of error, w

Re: internal unit tests failing against the latest spark master

i confirmed that an Encoder[Array[Int]] is no longer serializable, and with my spark build from march 7 it was. i believe the issue is commit 295747e59739ee8a697ac3eba485d3439e4a04c3 and i send wenchen an email about it. On Wed, Apr 12, 2017 at 4:31 PM, Koert Kuipers wrote: > i believe the erro

Re: internal unit tests failing against the latest spark master

i believe the error is related to an org.apache.spark.sql.expressions.Aggregator where the buffer type (BUF) is Array[Int] On Wed, Apr 12, 2017 at 4:19 PM, Koert Kuipers wrote: > hey all, > today i tried upgrading the spark version we use internally by creating a > new internal release from the

internal unit tests failing against the latest spark master

hey all, today i tried upgrading the spark version we use internally by creating a new internal release from the spark master branch. last time i did this was march 7. with this updated spark i am seeing some serialization errors in the unit tests for our own libraries. looks like a scala reflecti

Re: Tests failing with GC limit exceeded

quick update: things are looking slightly... better. the number of failing builds due to GC overhead has decreased slightly since the reboots last week... in fact, in the last three days the only builds to be affected are spark-master-test-maven-hadoop-2.7 (three failures) and spark-master-test

Re: Tests failing with GC limit exceeded

(adding michael armbrust and josh rosen for visibility) ok. roughly 9% of all spark tests builds (including both PRB builds are failing due to GC overhead limits. $ wc -l SPARK_TEST_BUILDS GC_FAIL 1350 SPARK_TEST_BUILDS 125 GC_FAIL here are the affected builds (over the past ~2 weeks): $ sor

Re: Tests failing with GC limit exceeded

On Fri, Jan 6, 2017 at 12:20 PM, shane knapp wrote: > FYI, this is happening across all spark builds... not just the PRB. s/all/almost all/ - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Tests failing with GC limit exceeded

FYI, this is happening across all spark builds... not just the PRB. i'm compiling a report now and will email that out this afternoon. :( On Thu, Jan 5, 2017 at 9:00 PM, shane knapp wrote: > unsurprisingly, we had another GC: > > https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder

Re: Tests failing with GC limit exceeded

unsurprisingly, we had another GC: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70949/console so, definitely not the system (everything looks hunky dory on the build node). > It can always be some memory leak; if we increase the memory settings > and OOMs still happen, that

Re: Tests failing with GC limit exceeded

But is there any non-memory-leak reason why the tests should need more memory? In theory each test should be cleaning up it's own Spark Context etc. right? My memory is that OOM issues in the tests in the past have been indicative of memory leaks somewhere. I do agree that it doesn't seem likely

Re: Tests failing with GC limit exceeded

On Thu, Jan 5, 2017 at 4:58 PM, Kay Ousterhout wrote: > But is there any non-memory-leak reason why the tests should need more > memory? In theory each test should be cleaning up it's own Spark Context > etc. right? My memory is that OOM issues in the tests in the past have been > indicative of m

Re: Tests failing with GC limit exceeded

Seems like the OOM is coming from tests, which most probably means it's not an infrastructure issue. Maybe tests just need more memory these days and we need to update maven / sbt scripts. On Thu, Jan 5, 2017 at 1:19 PM, shane knapp wrote: > as of first thing this morning, here's the list of rece

Re: Tests failing with GC limit exceeded

Thanks for looking into this Shane! On Thu, Jan 5, 2017 at 1:19 PM, shane knapp wrote: > as of first thing this morning, here's the list of recent GC overhead > build failures: > https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70891/ > console > https://amplab.cs.berkeley.edu/

Re: Tests failing with GC limit exceeded

as of first thing this morning, here's the list of recent GC overhead build failures: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70891/console https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70874/console https://amplab.cs.berkeley.edu/jenkins/job/SparkPul

Tests failing with GC limit exceeded

I've noticed a bunch of the recent builds failing because of GC limits, for seemingly unrelated changes (e.g. 70818, 70840, 70842). Shane, have there been any recent changes in the build configuration that might be causing this? Does anyone else have any ideas about what's going on here? -Kay

Re: Tests failing with GC limit exceeded

preliminary findings: seems to be transient, and affecting 4% of builds from late december until now (which is as far back as we keep build records for the PRB builds). 408 builds 16 builds.gc <--- failures it's also happening across all workers at about the same rate. and best of all, the

Re: Tests failing with GC limit exceeded

nope, no changes to jenkins in the past few months. ganglia graphs show higher, but not worrying, memory usage on the workers when the jobs failed... i'll take a closer look later tonite/first thing tomorrow morning. shane On Tue, Jan 3, 2017 at 4:35 PM, Kay Ousterhout wrote: > I've noticed a

Re: Cluster tests failing

Try to build the assembly jar first. ClusterSuite uses local-cluster mode, which requires the assembly jar. -Xiangrui On Tue, Sep 30, 2014 at 8:23 AM, Debasish Das wrote: > I have done mvn clean several times... > > Consistently all the mllib tests that are using > LocalClusterSparkContext.scala,

Re: Cluster tests failing

I have done mvn clean several times... Consistently all the mllib tests that are using LocalClusterSparkContext.scala, they fail !

Re: Cluster tests failing

first run cmd mvn clean, then try again On 2014/9/30 16:25, Debasish Das wrote: org.apache.spark.SparkException: Job aborted due to stage failure: Master - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For addi

Cluster tests failing

Hi, Inside mllib I am running tests using: mvn -Dhadoop.version=2.3.0-cdh5.1.0 -Phadoop-2.3 -Pyarn install The locat tests run fine but cluster tests are failing.. LBFGSClusterSuite: - task size should be small *** FAILED *** org.apache.spark.SparkException: Job aborted due to stage failure

Re: Tests failing

*Bam. * On Fri, Aug 15, 2014 at 5:04 PM, Patrick Wendell wrote: > Yeah I was thinking something like that. Basically we should just have a > variable for the timeout and I can make sure it's under the configured > Jenkins time. >

Re: Tests failing

Yeah I was thinking something like that. Basically we should just have a variable for the timeout and I can make sure it's under the configured Jenkins time. On Fri, Aug 15, 2014 at 1:55 PM, Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > So 2 hours is a hard cap on how long a build can

Re: Tests failing

So 2 hours is a hard cap on how long a build can run. Okie doke. Perhaps then I'll wrap the run-tests step as you suggest and limit it to 100 minutes or something, and cleanly report if it times out. Sound good? On Fri, Aug 15, 2014 at 4:43 PM, Patrick Wendell wrote: > Hey Nicholas, > > Yeah

Re: Tests failing

Hey Nicholas, Yeah so Jenkins has it's own timeout mechanism and it will just kill the entire build after 120 minutes. But since run-tests is sitting in the middle of the tests, it can't actually post a failure message. I think run-tests-jenkins should just wrap the call to run-tests in a call in

Re: Tests failing

OK, I've captured this in SPARK-3076 . Patrick, Is the problem that this run-tests step times out, and that is currently not handled gracefu

Re: Tests failing

Jenkins runs for this PR https://github.com/apache/spark/pull/1960 timed out without notification. The relevant Jenkins logs are at https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18588/consoleFull https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18592/consoleF

Re: Tests failing

We'll need to build timeouts into our own reporting infrastructure - it shouldn't be too bad but we just need to script it. Unfortunately the Jenkins plug-in is either "all or nothing" in what it reports, so we can't have it report timeouts unless we want all the other fairly noisy messages from it

Re: Tests failing

Shivaram, Can you point us to an example of that happening? The Jenkins console output, that is. Nick On Fri, Aug 15, 2014 at 2:28 PM, Shivaram Venkataraman < shiva...@eecs.berkeley.edu> wrote: > Also I think Jenkins doesn't post build timeouts to github. Is there anyway > we can fix that ? >

Re: Tests failing

Also I think Jenkins doesn't post build timeouts to github. Is there anyway we can fix that ? On Aug 15, 2014 9:04 AM, "Patrick Wendell" wrote: > Hi All, > > I noticed that all PR tests run overnight had failed due to timeouts. The > patch that updates the netty shuffle I believe somehow inflated

Tests failing

Hi All, I noticed that all PR tests run overnight had failed due to timeouts. The patch that updates the netty shuffle I believe somehow inflated to the build time significantly. That patch had been tested, but one change was made before it was merged that was not tested. I've reverted the patch