Which method do you think is better for making MIN_REMEMBER_DURATION configurable?

Hello, This is about SPARK-3276 and I want to make MIN_REMEMBER_DURATION (that is now a constant) a variable (configurable, with a default value). Before spending effort on developing something and creating a pull request, I wanted to consult with the core developers to see which approach makes mo

Re: Which method do you think is better for making MIN_REMEMBER_DURATION configurable?

What do you say? Kind regards, Emre Sevinç http://www.bigindustries.be/ On Wed, Apr 8, 2015 at 7:16 PM, Tathagata Das wrote: > Approach 2 is definitely better :) > Can you tell us more about the use case why you want to do this? > > TD > > On Wed, Apr 8, 2015 at 1:44 AM, Emre Sev

How to use Spark Streaming .jar file that I've built using a different branch than master?

Hello, I'm building a different version of Spark Streaming (based on a different branch than master) in my application for testing purposes, but it seems like spark-submit is ignoring my newly built Spark Streaming .jar, and using an older version. Here's some context: I'm on a different branch:

Re: How to use Spark Streaming .jar file that I've built using a different branch than master?

ride the SPARK_CLASSPATH with your newly built jar. > > Thanks > Best Regards > > On Mon, Apr 20, 2015 at 2:28 PM, Emre Sevinc > wrote: > >> Hello, >> >> I'm building a different version of Spark Streaming (based on a different >> branch than master) in my ap

Re: How to use Spark Streaming .jar file that I've built using a different branch than master?

Apparently, after *only* building Spark Streaming, I also have to: mvn --projects assembly/ -DskipTests clean install so that my test project uses the new version when I pass it to spark-submit. -- Emre Sevinç On Mon, Apr 20, 2015 at 10:58 AM, Emre Sevinc wrote: > Hello, > > I&#

Is there any particular reason why there's no Java counterpart in Streaming Guide's "Design Patterns for using foreachRDD" section?

Hello, Is there any particular reason why there's no Java counterpart in Streaming Guide's "Design Patterns for using foreachRDD" section? https://spark.apache.org/docs/latest/streaming-programming-guide.html Up to that point, each source code example includes corresponding Java (and sometime

Re: Multi-Line JSON in SparkSQL

;d be glad to hear it). > > Is there any plan to address this or any public recommendation ? > (considering the documentation clearly states that sqlContext.jsonFile will > not work for multi-line json(s)) > > Regards, > > Olivier. > -- Emre Sevinc

Re: Speeding up Spark build during development

t;> > >> Currently I'm using: mvn -DskipTests package > >> > >> > >> > >> All the nodes have the same filesystem mounted at the same mount > >> point. > >> > >> > >> > >> Pramod > >> > >> > >> > > >> > > >> > > >> > The information contained in this e-mail is confidential and/or > >> > proprietary to Capital One and/or its affiliates. The information > >> > transmitted herewith is intended only for use by the individual or > >> entity > >> > to which it is addressed. If the reader of this message is not the > >> > intended recipient, you are hereby notified that any review, > >> > retransmission, dissemination, distribution, copying or other use of, > or > >> > taking of any action in reliance upon this information is strictly > >> > prohibited. If you have received this communication in error, please > >> > contact the sender and delete the material from your computer. > >> > > >> > > >> > > > > > -- Emre Sevinc

Re: Speeding up Spark build during development

Spark. I think you can use a very similar approach. -- Emre Sevinç On Mon, May 4, 2015 at 10:44 AM, Pramod Biligiri wrote: > No, I just need to build one project at a time. Right now SparkSql. > > Pramod > > On Mon, May 4, 2015 at 12:09 AM, Emre Sevinc > wrote: > >>