Re: 2.1.2 maintenance release?

2017-09-10 Thread Holden Karau
So I think the consensus is that their is interest in having a few maintenance releases. I'm happy to act as the RM. I think the next step is seeing who the PMC wants as the RM for these (and if people are OK with me I'll start updating my self on the docs, open JIRAs, and relevant Jenkins jobs for

Re: 2.1.2 maintenance release?

2017-09-10 Thread Felix Cheung
Hi - what are the next steps? Pending changes are pushed and checked that there is no open JIRA targeting 2.1.2 and 2.2.1 _ From: Reynold Xin mailto:r...@databricks.com>> Sent: Friday, September 8, 2017 9:27 AM Subject: Re: 2.1.2 maintenance release? To: Felix Cheung m

Re: Supporting Apache Aurora as a cluster manager

2017-09-10 Thread Mark Hamstra
While it may be worth creating the design doc and JIRA ticket so that we at least have a better idea and a record of what you are talking about, I kind of doubt that we are going to want to merge this into the Spark codebase. That's not because of anything specific to this Aurora effort, but rather

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2 read path

2017-09-10 Thread Hemant Bhanawat
+1 (non-binding) I have found the suggestion from Andrew Ash and James about plan push down quite interesting. However, I am not clear about the join push-down support at the data source level. Shouldn't it be the responsibility of the join node to carry out a data source specific join? I mean joi

Supporting Apache Aurora as a cluster manager

2017-09-10 Thread karthik padmanabhan
Hi Spark Devs, We are using Aurora (http://aurora.apache.org/) as our mesos framework for running stateless services. We would like to use Aurora to deploy big data and batch workloads as well. And for this we have forked Spark and implement the ExternalClusterManager trait. The reason for doing

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2 read path

2017-09-10 Thread vaquar khan
+1 Regards, Vaquar khan On Sep 10, 2017 5:18 AM, "Noman Khan" wrote: > +1 > -- > *From:* wangzhenhua (G) > *Sent:* Friday, September 8, 2017 2:20:07 AM > *To:* Dongjoon Hyun; 蒋星博 > *Cc:* Michael Armbrust; Reynold Xin; Andrew Ash; Herman van Hövell tot > Westerflier;

Re: [SS] Bug in StreamExecution? currentBatchId and getBatchDescriptionString for web UI

2017-09-10 Thread Jacek Laskowski
Hi, Please disregard my finding. It does not seem a bug, but just a small "dead code" as "init" will never be displayed in web UI = the minimum batch id can ever be 0 and so getBatchDescriptionString could be a little "improved". Sorry for the noise. Pozdrawiam, Jacek Laskowski https://abou

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2 read path

2017-09-10 Thread Noman Khan
+1 From: wangzhenhua (G) Sent: Friday, September 8, 2017 2:20:07 AM To: Dongjoon Hyun; 蒋星博 Cc: Michael Armbrust; Reynold Xin; Andrew Ash; Herman van Hövell tot Westerflier; Ryan Blue; Spark dev list; Suresh Thalamati; Wenchen Fan Subject: 答复: [VOTE] [SPIP] SPARK-1