Re: jenkins failing on Kinesis shard limits

For me https://amplab.cs.berkeley.edu/jenkins/job/SlowSparkPullRequestBuilder/97/console https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38417/console On 25 July 2015 at 09:57, Patrick Wendell wrote: > I've disabled the test and filed a JIRA: > > https://issues.apache.org/ji

Protocol for build breaks

Hi All, If there is a build break (i.e. a compile issue or consistently failing test) that somehow makes it into master, the best protocol is: 1. Revert the offending patch. 2. File a JIRA and assign it to the committer of the offending patch. The JIRA should contain links to broken builds. It's

Re: jenkins failing on Kinesis shard limits

I've disabled the test and filed a JIRA: https://issues.apache.org/jira/browse/SPARK-9335 On Fri, Jul 24, 2015 at 4:05 PM, Steve Loughran wrote: > > Looks like Jenkins is hitting some AWS limits > > https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38396/testReport/org.apache.sp

Jenkins HiveCompatibilitySuite Test Failures

Hi, I've been seeing errors with org.apache.spark.sql.hive.execution.HiveCompatibilitySuite from the Jenkins tests in a PR I proposed as well as in PRs from other members of the community. Is this test stable at the moment? Thanks, Calvin

jenkins failing on Kinesis shard limits

Looks like Jenkins is hitting some AWS limits https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38396/testReport/org.apache.spark.streaming.kinesis/KinesisBackedBlockRDDSuite/_It_is_not_a_test_/

Re: PySpark on PyPi

Hey all, great discussion, just wanted to +1 that I see a lot of value in steps that make it easier to use PySpark as an ordinary python library. You might want to check out this (https://github.com/minrk/findspark ), started by Jupyter project devs, that off

Policy around backporting bug fixes

Hi All, A few times I've been asked about backporting and when to backport and not backport fix patches. Since I have managed this for many of the past releases, I wanted to point out the way I have been thinking about it. If we have some consensus I can put it on the wiki. The trade off when bac

Re: non-deprecation compiler warnings are upgraded to build errors now

Jenkins only run Scala 2.10. I'm actually not sure what the behavior is with 2.11 for that patch. iulian - can you take a look into it and see if it is working as expected? On Fri, Jul 24, 2015 at 10:24 AM, Iulian Dragoș wrote: > On Thu, Jul 23, 2015 at 6:08 AM, Reynold Xin wrote: > > Hi all,

Re: non-deprecation compiler warnings are upgraded to build errors now

On Thu, Jul 23, 2015 at 6:08 AM, Reynold Xin wrote: Hi all, > > FYI, we just merged a patch that fails a build if there is a scala > compiler warning (if it is not deprecation warning). > I’m a bit confused, since I see quite a lot of warnings in semi-legitimate code. For instance, @transient (p

Re: non-deprecation compiler warnings are upgraded to build errors now

You can give it a shot, but we will have to revert it for a project as soon as a project uses a deprecated API somewhere. On Fri, Jul 24, 2015 at 7:43 AM, Punyashloka Biswal wrote: > Would it make sense to isolate the use of deprecated warnings to a subset > of projects? That way we could turn

Re: [ANNOUNCE] Nightly maven and package builds for Spark

Hey Bharath, There was actually an incompatible change to the build process that broke several of the Jenkins builds. This should be patched up in the next day or two and nightly builds will resume. - Patrick On Fri, Jul 24, 2015 at 12:51 AM, Bharath Ravi Kumar wrote: > I noticed the last (1.5)

Re: non-deprecation compiler warnings are upgraded to build errors now

Would it make sense to isolate the use of deprecated warnings to a subset of projects? That way we could turn on more stringent checks for the other ones. Punya On Thu, Jul 23, 2015 at 12:08 AM Reynold Xin wrote: > Hi all, > > FYI, we just merged a patch that fails a build if there is a scala >

Re: review SPARK-8730

I just got those comments and it doesn't merge since few time, the code evolved since I opened the pr 2015-07-24 14:12 GMT+02:00 Sean Owen : > It looks like you have a number of review comments on the PR that you > have not replied to. The PR does not merge at the moment either. > > On Fri, Jul 2

Re: review SPARK-8730

It looks like you have a number of review comments on the PR that you have not replied to. The PR does not merge at the moment either. On Fri, Jul 24, 2015 at 12:03 PM, Eugen Cepoi wrote: > Hey, > > I've opened a PR to fix ser/de issue of primitives classes in the java > serializer. > I already e

review SPARK-8730

Hey, I've opened a PR to fix ser/de issue of primitives classes in the java serializer. I already encountered this problem in different scenarios, so I am bringing it up. Would be great if someone wants to have a look at it! :) https://issues.apache.org/jira/browse/SPARK-8730 https://github.com/a

Re: [ANNOUNCE] Nightly maven and package builds for Spark

I noticed the last (1.5) build has a timestamp of 16th July. Have nightly builds been discontinued since then? Thanks, Bharath On Sun, May 24, 2015 at 1:11 PM, Patrick Wendell wrote: > Hi All, > > This week I got around to setting up nightly builds for Spark on > Jenkins. I'd like feedback on t