Re: Speeding up Spark build during development

Following what Ted said, if you leverage the `mvn` from within the `build/` directory of Spark you¹ll get zinc for free which should help speed up build times. On 5/1/15, 9:45 AM, "Ted Yu" wrote: >Pramod: >Please remember to run Zinc so that the build is faster. > >Cheers > >On Fri, May 1, 2015

Re: Speeding up Spark build during development

Pramod: Please remember to run Zinc so that the build is faster. Cheers On Fri, May 1, 2015 at 9:36 AM, Ulanov, Alexander wrote: > Hi Pramod, > > For cluster-like tests you might want to use the same code as in mllib's > LocalClusterSparkContext. You can rebuild only the package that you change

RE: Speeding up Spark build during development

Hi Pramod, For cluster-like tests you might want to use the same code as in mllib's LocalClusterSparkContext. You can rebuild only the package that you change and then run this main class. Best regards, Alexander -Original Message- From: Pramod Biligiri [mailto:pramodbilig...@gmail.com

Re: Tungsten + Flink

I don't think it's useful to combine them since they are different projects. But I do think that a lot of work went into Flink's paged memory system built on byte buffers and if collaboration can take place to pop that out into like a memory subsystem library that both Spark and Flink can use t

Re: Tungsten + Flink

I think as long as the two frameworks follow the same paradigm for how their interfaces work it’s fine to have 2 competing frameworks. This way the frameworks have some motivation to be the best at what they do rather than being the only choice whether you like it or not. They also seem to have

Re: [discuss] ending support for Java 6?

it seems spark is happy to upgrade scala, drop older java versions, upgrade incompatible library versions (akka), and all of this within spark 1.x does the 1.x mean anything in terms of compatibility of dependencies? or is that limited to its own api? what are the rules? On May 1, 2015 9:04 AM, "S

Re: [discuss] ending support for Java 6?

No. That will be "3.0" some day Sent from my rotary phone. > On May 1, 2015, at 9:04 AM, Steven Shaw wrote: > >> On 1 May 2015 at 21:26, Dean Wampler wrote: > >> FWIW, another reason to start planning for deprecation of Java 7, too, is >> that Scala 2.12 will require Java 8. Scala 2.12 will

Re: [discuss] ending support for Java 6?

On 1 May 2015 at 21:26, Dean Wampler wrote: > FWIW, another reason to start planning for deprecation of Java 7, too, is > that Scala 2.12 will require Java 8. Scala 2.12 will be released early next > year. > ​Will 2.12 be the release that based on dotty ? Chee

Re: [discuss] ending support for Java 6?

FWIW, another reason to start planning for deprecation of Java 7, too, is that Scala 2.12 will require Java 8. Scala 2.12 will be released early next year. Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition (O'Reilly) Typesafe

Re: Speeding up Spark build during development

Hi Pramod, If you are using sbt as your build, then you need to do sbt assembly once and use sbt ~compile. Also export SPARK_PREPEND_CLASSES=1 this in your shell and all nodes. You can may be try this out ? Thanks, Prashant Sharma On Fri, May 1, 2015 at 2:16 PM, Pramod Biligiri wrote: > Hi,

Re: [discuss] ending support for Java 6?

> On 30 Apr 2015, at 21:40, Marcelo Vanzin wrote: > > As for the idea, I'm +1. Spark is the only reason I still have jdk6 > around - exactly because I don't want to cause the issue that started > this discussion (inadvertently using JDK7 APIs). And as has been > pointed out, even J7 is about to

Speeding up Spark build during development

Hi, I'm making some small changes to the Spark codebase and trying it out on a cluster. I was wondering if there's a faster way to build than running the package target each time. Currently I'm using: mvn -DskipTests package All the nodes have the same filesystem mounted at the same mount point.

Re: Custom PersistanceEngine and LeaderAgent implementation in Java

Hi Reynold, Pls find the PR here [1] [1] https://github.com/apache/spark/pull/5832 On Thu, Apr 30, 2015 at 11:34 AM, Reynold Xin wrote: > We should change the trait to abstract class, and then your problem will > go away. > > Do you want to submit a pull request? > > > On Wed, Apr 29, 2015 at

Fwd: Event generator for SPARK-Streaming from csv

I have the real DEBS-TAxi data in csv file , in order to operate over it how to simulate a "Spout" kind of thing as event generator using the timestamps in CSV file. -- Thanks & Regards, Anshu Shukla