Re: [build system] jenkins got itself wedged...

2017-05-18 Thread Sean Owen
I'm not sure if it's related, but I still can't get Jenkins to test PRs. For example, triggering it through the spark-prs.appspot.com UI gives me... https://spark-prs.appspot.com/trigger-jenkins/18012 Internal Server Error That might be from the appspot app though? But posting "Jenkins test this

[commons-text] Regarding code consolidation.

2017-05-18 Thread Rob Tompkins
Hello all, Over the last year or so we in Commons have been working towards a newly released component “commons-text,”, and we were wondering if folks wanted to begin consuming commons-text so that we can consolidate the maintenance of the code performing edit distances and similarity scores (for

Re: Spark Local Pipelines

2017-05-18 Thread Asher Krim
Hi Cristian, There's a jira (https://issues.apache.org/jira/browse/SPARK-16365) where this issue has been discussed as well. I feel very strongly about the need for this feature. I've been implementing local versions of transformers as needed, which has made working with Spark ml much less pleasan

Re: [build system] jenkins got itself wedged...

2017-05-18 Thread shane knapp
yeah, i spoke too soon. jenkins is still misbehaving, but FINALLY i'm getting some error messages in the logs... looks like jenkins is thrashing on GC. now that i know what's up, i should be able to get this sorted today. On Thu, May 18, 2017 at 12:39 AM, Sean Owen wrote: > I'm not sure if it

Re: [build system] jenkins got itself wedged...

2017-05-18 Thread shane knapp
ok, more updates: 1) i audited all of the builds, and found that the spark-*-compile-* and spark-*-test-* jobs were set to the identical cron time trigger, so josh rosen and i updated them to run at H/5 (instead of */5). load balancing ftw. 2) the jenkins master is now running on java8, which ha

Spark madness

2017-05-18 Thread Saikat Kanjilal
Hi Devs, I'm needing to read a json file from hdfs and turn that into a scala string, I have dug around for documentation on how to do this and found this: http://stackoverflow.com/questions/30445263/how-to-read-whole-file-in-one-string [https://cdn.sstatic.net/Sites/stackoverflow/img/apple-t

Re: Spark madness

2017-05-18 Thread Saikat Kanjilal
One additional point, the following line: rdd.collect.foreach(t=>println(t._2)) when set to a scala string prints nothing even when I use toString at the end. This seems to not be something that should be that out of the ordinary but I could be wrong. From: S