Re: SPARK-13843 and future of streaming backends

I have worked with various ASF projects for 4+ years now. Sure, ASF projects can delete code as they feel fit. But this is the first time I have really seen code being "moved out" of a project without discussion. I am sure you can do this without violating ASF policy, but the explanation for that w

Re: Long running Spark job on YARN throws "No AMRMToken"

The credentials file approach (using keytab for spark apps) will only update HDFS tokens. YARN's AMRM tokens should be taken care of by YARN internally. Steve - correct me if I am wrong here: If the AMRM tokens are disappearing it might be a YARN bug (does the AMRM token have a 7 day limit as well

Re: Scala 2.11 builds broken/ Can the PR build run also 2.11?

+1, much better than having a new PR each time to fix something for scala-2.11 every time a patch breaks it. Thanks, Hari Shreedharan > On Oct 9, 2015, at 11:47 AM, Michael Armbrust wrote: > > How about just fixing the warning? I get it; it doesn't stop this from > hap

Re: [VOTE] Release Apache Spark 1.4.0 (RC4)

+1. Build looks good, ran a couple apps on YARN Thanks, Hari On Fri, Jun 5, 2015 at 10:52 AM, Yin Huai wrote: > Sean, > > Can you add "-Phive -Phive-thriftserver" and try those Hive tests? > > Thanks, > > Yin > > On Fri, Jun 5, 2015 at 5:19 AM, Sean Owen wrote: > >> Everything checks out agai

Re: Let's set Assignee for Fixed JIRAs

You’d need to add them as a contributor in the JIRA admin page. Once you do that, you should be able to assign the jira to that person Thanks, Hari On Thu, Apr 23, 2015 at 5:33 PM, Shivaram Venkataraman wrote: > A related question that has affected me in the past: If we get a PR from a > ne

Re: finding free ports for tests

One good way to guarantee your tests will work is to have your server bind to an ephemeral port and then query it to find the port it is running on. This ensures that race conditions don’t cause test failures. Thanks, Hari On Wed, Apr 8, 2015 at 3:24 AM, Sean Owen wrote: > Utils.startServi

Re: [VOTE] Release Apache Spark 1.3.1

rk/commit/03e263f5b527cf574f4ffcd5cd886f7723e3756e > - Patrick > On Mon, Apr 6, 2015 at 2:31 PM, Mark Hamstra wrote: >> Is that correct, or is the JIRA just out of sync, since TD's PR was merged? >> https://github.com/apache/spark/pull/5008 >> >> On Mon, Apr 6, 2015 at 11:10 AM, Hari Shreedha

Re: [VOTE] Release Apache Spark 1.3.1

It does not look like https://issues.apache.org/jira/browse/SPARK-6222 made it. It was targeted towards this release.  Thanks, Hari On Mon, Apr 6, 2015 at 11:04 AM, York, Brennon wrote: > +1 (non-binding) > Tested GraphX, build infrastructure, & core test suite on OSX 10.9 w/ Java > 1.7/1.8

Re: PR Builder timing out due to ivy cache lock

Here you are: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28571/consoleFull On Fri, Mar 13, 2015 at 11:58 AM, shane knapp wrote: > link to a build, please? > > On Fri, Mar 13, 2015 at 11:53 AM, Hari Shreedharan < > hshreedha...@cloudera.com> wrot

PR Builder timing out due to ivy cache lock

Looks like something is causing the PR Builder to timeout since this morning with the ivy cache being locked. Any idea what is happening?

Re: Improving metadata in Spark JIRA

+1. Jira cleanup would be good. Please let me know if I can help in some way! Thanks, Hari On Fri, Feb 6, 2015 at 11:56 AM, Nicholas Chammas wrote: > Do we need some new components to be added to the JIRA project? > Like: >- >scheduler > - >YARN > - spark-submit >- …?

Re: Welcoming three new committers

Congrats Cheng, Joseph and Owen! Well done! Thanks, Hari On Tue, Feb 3, 2015 at 2:55 PM, Ted Yu wrote: > Congratulations, Cheng, Joseph and Sean. > On Tue, Feb 3, 2015 at 2:53 PM, Nicholas Chammas > wrote: >> Congratulations guys! >> >> On Tue Feb 03 2015 at 2:36:12 PM Matei Zaharia >> wrot

Re: Which committers care about Kafka?

; 2. begin transaction >>> 3. save data to the store >>> 4. save offsets >>> 5. commit transaction >>> >>> If you fail before 5, the transaction rolls back. To make this less >>> heavyweight, you can write the data outside the transaction and

Re: Which committers care about Kafka?

liability using the primitives >> that spark so elegantly provides. >> >> Cheers, >> >> Sean >> >> >> > On Dec 19, 2014, at 12:06 PM, Hari Shreedharan < >> hshreedha...@cloudera.com> wrote: >> > >> > Hi Dibyendu, >> &g

Re: Which committers care about Kafka?

o achieve this, not to say rebalance >> and fault-tolerance. >> >> Thanks >> Jerry >> >> -Original Message- >> From: Luis Ángel Vicente Sánchez [mailto:langel.gro...@gmail.com] >> Sent: Friday, December 19, 2014 5:57 AM >> To: Cody Koe

Re: Which committers care about Kafka?

independent > implementations of the idea, chances are it's something people need. > On Thu, Dec 18, 2014 at 1:44 PM, Hari Shreedharan > wrote: >> >> Hi Cody, >> >> I am an absolute +1 on SPARK-3146. I think we can implement something >> pretty simple and lightw

Re: Which committers care about Kafka?

Hi Cody, I am an absolute +1 on SPARK-3146. I think we can implement something pretty simple and lightweight for that one. For the Kafka DStream skipping the WAL implementation - this is something I discussed with TD a few weeks ago. Though it is a good idea to implement this to avoid un

Re: Has anyone else observed this build break?

Seems like a comment on that page mentions a fix, which would add yet another profile though — specifically telling mvn that if it is an apple jdk, use the classes.jar as the tools.jar as well, since Apple-packaged JDK 6 bundled them together. Link: http://permalink.gmane.org/gmane.comp.java

Re: Spark-Submit issues

Yep, you’d need to shade jars to ensure all your dependencies are in the classpath. Thanks, Hari On Wed, Nov 12, 2014 at 3:23 AM, Ted Malaska wrote: > Hey this is Ted > Are you using Shade when you build your jar and are you using the bigger > jar? Looks like classes are not included in you

Re: Bind exception while running FlumeEventCount

Did you start a Flume agent to push data to the relevant port? Thanks, Hari On Fri, Nov 7, 2014 at 2:05 PM, Jeniba Johnson wrote: > Hi, > I have installed spark-1.1.0 and apache flume 1.4 for running streaming > example FlumeEventCount. Previously the code was working fine. Now Iam facing

RE: Bind exception while running FlumeEventCount

First, can you try a different port? TIME_WAIT is basically a timeout for a socket to be completely decommissioned for the port to be available for binding. Once you wait for a few minutes and if you still see a startup issue, can you also send the error logs? From what I can see, the port s

RE: Bind exception while running FlumeEventCount

rocess id of the specific port. Still Iam facing > with the similar error. > The command I use is > sudo lsof -i -P | grep -i "listen" > Kill -9 PID > However If I try to work with the port which is available, still the error > remains the same. > Regards, &

Re: Bind exception while running FlumeEventCount

Looks like that port is not available because another app is using that port. Can you take a look at netstat -a and use a port that is free? Thanks, Hari On Fri, Nov 7, 2014 at 2:05 PM, Jeniba Johnson wrote: > Hi, > I have installed spark-1.1.0 and apache flume 1.4 for running streaming >

Re: [VOTE] Designating maintainers for some Spark components

ible.". > I'd like us to start with something simpler and lightweight as > proposed here. Really the proposal on the table is just to codify the > current de-facto process to make sure we stick by it as we scale. If > we want to add more formality to it or strictness, we can do it

Re: [VOTE] Designating maintainers for some Spark components

How would this model work with a new committer who gets voted in? Does it mean that a new committer would be a maintainer for at least one area — else we could end up having committers who really can’t merge anything significant until he becomes a maintainer.  Thanks, Hari On Thu, Nov 6, 2014

Re: Build fails on master (f90ad5d)

I have seen this on sbt sometimes. I usually do an sbt clean and that fixes it. Thanks, Hari On Tue, Nov 4, 2014 at 3:13 PM, Nicholas Chammas wrote: > FWIW, the "official" build instructions are here: > https://github.com/apache/spark#building-spark > On Tue, Nov 4, 2014 at 5:11 PM, Ted Yu wr

Re: Moving PR Builder to mvn

gt;> >> - Patrick >> >> On Fri, Oct 24, 2014 at 1:55 PM, Hari Shreedharan >> wrote: >> > I have zinc server running on my mac, and I see maven compilation to be >> much >> > better than before I had it running. Is the sbt build still faster >

Re: Moving PR Builder to mvn

pull requests > with either one. > - Patrick > On Fri, Oct 24, 2014 at 1:39 PM, Hari Shreedharan > wrote: >> Over the last few months, it seems like we have selected Maven to be the >> "official" build system for Spark. >> >> >> I realize that re

Moving PR Builder to mvn

Over the last few months, it seems like we have selected Maven to be the “official” build system for Spark.  I realize that removing the sbt build may not be easy, but it might be a good idea to start looking into that. We had issues over the past few days where mvn builds were fine, while sbt

Re: Building and Running Spark on OS X

The sbt executable that is in the spark repo can be used to build sbt without any other set up (it will download the sbt jars etc). Thanks, Hari On Mon, Oct 20, 2014 at 5:16 PM, Sean Owen wrote: > Maven is at least built in to OS X (well, with dev tools). You don't > even have to brew install

Re: reference to dstream in package org.apache.spark.streaming which is not available

Sean - I think only the ones in 1726 are enough. It is weird that any class that uses the test-jar actually requires the streaming jar to be added explicitly. Shouldn't maven take care of this? I posted some comments on the PR. -- Thanks, Hari Sean Owen August 2

Re: Spark Avro Generation

Jay running sbt compile or assembly should generate the sources. On Monday, August 11, 2014, Devl Devel wrote: > Hi > > So far I've been managing to build Spark from source but since a change in > spark-streaming-flume I have no idea how to generate classes (e.g. > SparkFlumeProtocol) from the a

Re: sbt/sbt test steals window focus on OS X

Add this to your .bash_profile (or .bashrc) - that will fix it. export _JAVA_OPTIONS=-Djava.awt.headless=true Hari On Sun, Jul 20, 2014 at 1:56 PM, Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > I just created SPARK-2602 < > https://issues.apache.org/jira/browse/SPARK-2602> to > trac