Re: Connectors using new Kafka consumer API

o us making a 5th slightly > different interface to kafka (currently have 0.8 receiver, 0.8 > dstream, 0.10 dstream, 0.10 structured stream) > > On Tue, Nov 8, 2016 at 10:05 PM, Mark Grover wrote: > > I think they are open to others helping, in fact, more than one person > h

Re: Connectors using new Kafka consumer API

te: > Have you asked the assignee on the Kafka jira whether they'd be > willing to accept help on it? > > On Tue, Nov 8, 2016 at 5:26 PM, Mark Grover wrote: > > Hi all, > > We currently have a new direct stream connector, thanks to work by Cody > and > > others o

Connectors using new Kafka consumer API

Hi all, We currently have a new direct stream connector, thanks to work by Cody and others on SPARK-12177. However, that can't be used in secure clusters that require Kerberos authentication. That's because Kafka currently doesn't support delegation tokens (KAFKA-1696

Re: [VOTE] Release Apache Spark 2.0.0 (RC1)

Yeah, I am +1 for including Kafka 0.10 integration as well. We had to wait for Kafka 0.10 because there were incompatibilities between the Kafka 0.9 and 0.10 API. And, yes, the code for 0.8.0 remains unchanged so there shouldn't be any regression for existing users. It's only new code for 0.10. Th

Re: YARN Shuffle service and its compatibility

Great, thanks for confirming, Reynold. Appreciate it! On Tue, Apr 19, 2016 at 4:20 PM, Reynold Xin wrote: > I talked to Lianhui offline and he said it is not that big of a deal to > revert the patch. > > > On Tue, Apr 19, 2016 at 9:52 AM, Mark Grover wrote: > >> Tha

Re: YARN Shuffle service and its compatibility

Thanks. I'm more than happy to wait for more people to chime in here but I do feel that most of us are leaning towards Option B anyways. So, I created a JIRA (SPARK-14731) for reverting SPARK-12130 in Spark 2.0 and file a PR shortly. Mark On Tue, Apr 19, 2016 at 7:44 AM, Tom Graves wrote: > It

Re: YARN Shuffle service and its compatibility

On Tue, Apr 19, 2016 at 2:26 AM, Steve Loughran wrote: > > > On 18 Apr 2016, at 23:05, Marcelo Vanzin wrote: > > > > On Mon, Apr 18, 2016 at 2:02 PM, Reynold Xin > wrote: > >> The bigger problem is that it is much easier to maintain backward > >> compatibility rather than dictating forward comp

Re: YARN Shuffle service and its compatibility

Thanks for responding, Reynold, Marcelo and Marcin. >And I think that's really what Mark is proposing. Basically, "don't >intentionally break backwards compatibility unless it's really >required" (e.g. SPARK-12130). That would allow option B to work. Yeah, that's exactly what Option B is proposin

YARN Shuffle service and its compatibility

Hi all, If you don't use Spark on YARN, you probably don't need to read further. Here's the *user scenario*: There are going to be folks who may be interested in running two versions of Spark (say Spark 1.6.x and Spark 2.x) on the same YARN cluster. And, here's the *problem*: That's all fine, sho

Re: Upgrading to Kafka 0.9.x

is the possibility of doing forwards and > backwards compatibility in the clients. I'm not sure this would actually > make things better, that would probably depend on the details of how it > worked. > > -Jay > > > On Fri, Feb 26, 2016 at 9:46 AM, Mark Grover wrote: >

Upgrading to Kafka 0.9.x

Hi Kafka devs, I come to you with a dilemma and a request. Based on what I understand, users of Kafka need to upgrade their brokers to Kafka 0.9.x first, before they upgrade their clients to Kafka 0.9.x. However, that presents a problem to other projects that integrate with Kafka (Spark, Flume, S

Re: Write access to wiki

Thanks Sean, I will send you the edit on the JIRA to keep email traffic low:-) Thanks Shane, comments in line. On Mon, Jan 11, 2016 at 2:50 PM, shane knapp wrote: > > Shane may be able to fill you in on how the Jenkins build is set up. > > > mark: yes. yes i can. :) > > currently, we have a s

Write access to wiki

Hi all, May I please get write access to the useful tools wiki page? I did some investigation

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

baked, might > be worth including. > > Logs coming offline > > On Fri, Dec 18, 2015 at 5:30 PM, Mark Grover wrote: > > Sean, > > Are you referring to docker integration tests? If so, they were disabled > for > > majority of the release and I recently worked on it (SP

Re: [VOTE] Release Apache Spark 1.6.0 (RC3)

Sean, Are you referring to docker integration tests? If so, they were disabled for majority of the release and I recently worked on it (SPARK-11796) and once it got committed, the tests were re-enabled in Spark builds. I am not sure what OSs the test builds use, but it should be passing there too.

Re: [DISCUSS] Necessity of Maven *and* SBT Build in Spark

Hi Patrick, And, to pile on what Sandy said. In my opinion, it's definitely more than just a matter of convenience. My comment below applies both to distribution builders but also people who have their own internal "distributions" (a few examples of which we have already seen on this thread already