Re: Spark Improvement Proposals

2017-03-13 Thread Sean Owen
Responding to your request for a vote, I meant that this isn't required per se and the consensus here was not to vote on it. Hence the jokes about meta-voting protocol. In that sense nothing new happened process-wise, nothing against ASF norms, if that's your concern. I think it's just an agreed c

Re: Spark Improvement Proposals

2017-03-13 Thread Tom Graves
Another thing I think you should send out is when exactly does this take affect.  Is it any major new feature without a pull request?   Is it anything major starting with the 2.3 release?   Tom On Monday, March 13, 2017 1:08 PM, Tom Graves wrote: I'm not sure how you can say its not a

Re: Spark Improvement Proposals

2017-03-13 Thread Tom Graves
I'm not sure how you can say its not a new process.  If that is the case why do we need a page documenting it?   As a developer if I want to put up a major improvement I have to now follow the SPIP whereas before I didn't, that certain seems like a new process.  As a PMC member I now have the ab

Re: Spark Improvement Proposals

2017-03-13 Thread Sean Owen
It's not a new process, in that it doesn't entail anything not already in http://apache.org/foundation/voting.html . We're just deciding to call a VOTE for this type of code modification. To your point -- yes, it's been around a long time with no further comment, and I called several times for mor

Re: Spark Improvement Proposals

2017-03-13 Thread Tom Graves
It seems like if you are adding responsibilities you should do a vote.  SPIP'S require votes from PMC members so you are now putting more responsibility on them. It feels like we should have an official vote to make sure they (PMC members) agree with that and to make sure everyone pays attention

Re: Spark Improvement Proposals

2017-03-13 Thread Sean Owen
This ended up proceeding as a normal doc change, instead of precipitating a meta-vote. However, the text that's on the web site now can certainly be further amended if anyone wants to propose a change from here. On Mon, Mar 13, 2017 at 1:50 PM Tom Graves wrote: > I think a vote here would be goo

Re: Spark Improvement Proposals

2017-03-13 Thread Tom Graves
I think a vote here would be good. I think most of the discussion was done by 4 or 5 people and its a long thread.  If nothing else it summarizes everything and gets people attention to the change. Tom On Thursday, March 9, 2017 10:55 AM, Sean Owen wrote: I think a VOTE is over-thinkin

Re: Spark Improvement Proposals

2017-03-10 Thread Reynold Xin
We can just start using spip label and link to it. On Fri, Mar 10, 2017 at 9:18 AM, Cody Koeninger wrote: > So to be clear, if I translate that google doc to markup and submit a > PR, you will merge it? > > If we're just using "spip" label, that's probably fine, but we still > need shared filt

Re: Spark Improvement Proposals

2017-03-10 Thread Cody Koeninger
Can someone with filter share permissions can make a filter for open SPIP and one for closed SPIP and share it? e.g. project = SPARK AND status in (Open, Reopened, "In Progress") AND labels=SPIP ORDER BY createdDate DESC and another with the status closed equivalent I just made an open ticket w

Re: Spark Improvement Proposals

2017-03-10 Thread Cody Koeninger
So to be clear, if I translate that google doc to markup and submit a PR, you will merge it? If we're just using "spip" label, that's probably fine, but we still need shared filters for open and closed SPIPs so the page can link to them. I do not believe I have jira permissions to share filters,

Re: Spark Improvement Proposals

2017-03-10 Thread Cody Koeninger
I think it ought to be its own page, linked from the more / community menu dropdowns. We also need the jira tag, and for the page to clearly link to filters that show proposed / completed SPIPs On Fri, Mar 10, 2017 at 3:39 AM, Sean Owen wrote: > Alrighty, if nobody is objecting, and nobody calls

Re: Spark Improvement Proposals

2017-03-10 Thread Sean Owen
Alrighty, if nobody is objecting, and nobody calls for a VOTE, then, let's say this document is the SPIP 1.0 process. I think the next step is just to translate the text to some suitable location. I suggest adding it to https://github.com/apache/spark-website/blob/asf-site/contributing.md On Thu,

Re: Spark Improvement Proposals

2017-03-09 Thread Koert Kuipers
gonna end up with a stackoverflow on recursive votes here On Thu, Mar 9, 2017 at 1:17 PM, Mark Hamstra wrote: > -0 on voting on whether we need a vote. > > On Thu, Mar 9, 2017 at 9:00 AM, Reynold Xin wrote: > >> I'm fine without a vote. (are we voting on wether we need a vote?) >> >> >> On Thu,

Re: Spark Improvement Proposals

2017-03-09 Thread Mark Hamstra
-0 on voting on whether we need a vote. On Thu, Mar 9, 2017 at 9:00 AM, Reynold Xin wrote: > I'm fine without a vote. (are we voting on wether we need a vote?) > > > On Thu, Mar 9, 2017 at 8:55 AM, Sean Owen wrote: > >> I think a VOTE is over-thinking it, and is rarely used, but, can't hurt. >>

Re: Spark Improvement Proposals

2017-03-09 Thread vaquar khan
Many of us have issue with "shepherd role " , i think we should go with vote. Regards, Vaquar khan On Thu, Mar 9, 2017 at 11:00 AM, Reynold Xin wrote: > I'm fine without a vote. (are we voting on wether we need a vote?) > > > On Thu, Mar 9, 2017 at 8:55 AM, Sean Owen wrote: > >> I think a VOTE

Re: Spark Improvement Proposals

2017-03-09 Thread Reynold Xin
I'm fine without a vote. (are we voting on wether we need a vote?) On Thu, Mar 9, 2017 at 8:55 AM, Sean Owen wrote: > I think a VOTE is over-thinking it, and is rarely used, but, can't hurt. > Nah, anyone can call a vote. This really isn't that formal. We just want to > declare and document con

Re: Spark Improvement Proposals

2017-03-09 Thread Sean Owen
I think a VOTE is over-thinking it, and is rarely used, but, can't hurt. Nah, anyone can call a vote. This really isn't that formal. We just want to declare and document consensus. I think SPIP is just a remix of existing process anyway, and don't think it will actually do much anyway, which is wh

Re: Spark Improvement Proposals

2017-03-09 Thread Cody Koeninger
I started this idea as a fork with a merge-able change to docs. Reynold moved it to his google doc, and has suggested during this email thread that a vote should occur. If a vote needs to occur, I can't see anything on http://apache.org/foundation/voting.html suggesting that I can call for a vote,

Re: Spark Improvement Proposals

2017-03-07 Thread Sean Owen
Do we need a VOTE? heck I think anyone can call one, anyway. Pre-flight vote check: anyone have objections to the text as-is? See https://docs.google.com/document/d/1-Zdi_W-wtuxS9hTK0P9qb2x-nRanvXmnZ7SUi4qMljg/edit# If so let's hash out specific suggest changes. If not, then I think the next ste

Re: Spark Improvement Proposals

2017-03-07 Thread Cody Koeninger
;>> >>> >> way to >>> >>> >> force attention to a particular change, then, this doesn't do >>> that by >>> >>> >> itself. Therefore I don't want to let a detailed discussion of >>> SPIP >>> >&g

Re: Spark Improvement Proposals

2017-02-27 Thread Sean Owen
To me, no new process is being invented here, on purpose, and so we should just rely on whatever governs any large JIRA or vote, because SPIPs are really just guidance for making a big JIRA. http://apache.org/foundation/voting.html suggests that PMC members have the binding votes in general, and f

Re: Spark Improvement Proposals

2017-02-27 Thread Ryan Blue
re I don't want to let a detailed discussion of SPIP >> >>> >> detract >> >>> >> from the discussion about doing what SPIP implies. It's just a >> process >> >>> >> document. >> >>> >> >> >&

Re: Spark Improvement Proposals

2017-02-24 Thread Joseph Bradley
017 at 4:22 PM Reynold Xin > >>> >> wrote: > >>> >>> > >>> >>> Updated. Any feedback from other community members? > >>> >>> > >>> >>> > >>> >>> On Wed, Feb 15, 2017 at

Re: Spark Improvement Proposals

2017-02-24 Thread Cody Koeninger
>>> >>> wrote: >>> >>>> >>> >>>> Thanks for doing that. >>> >>>> >>> >>>> Given that there are at least 4 different Apache voting processes, >>> >>>> "typical Apa

Re: Spark Improvement Proposals

2017-02-17 Thread vaquar khan
nk the intention is that in order to pass, it needs at least 3 >> +1 >> >>>> votes from PMC members *and no -1 votes from PMC members*. But the >> document >> >>>> doesn't explicitly say that second part. >> >>>> >> >>>

Re: Spark Improvement Proposals

2017-02-16 Thread Ryan Blue
> document > >>>> doesn't explicitly say that second part. > >>>> > >>>> There's also no mention of the duration a vote should remain open. > >>>> There's a mention of a month for finding a shepherd, but that's &

Re: Spark Improvement Proposals

2017-02-16 Thread Sam Elamin
second part. > >>>> > >>>> There's also no mention of the duration a vote should remain open. > >>>> There's a mention of a month for finding a shepherd, but that's > different. > >>>> > >>>> Other than th

Re: Spark Improvement Proposals

2017-02-16 Thread Tim Hunter
n wrote: >>>>> >>>>> Here's a new draft that incorporated most of the feedback: >>>>> https://docs.google.com/document/d/1-Zdi_W-wtuxS9hTK0P9qb2x-nRanvXmnZ7SUi4qMljg/edit# >>>>> >>>>> I added a specific role for SPIP Aut

Re: Spark Improvement Proposals

2017-02-16 Thread Cody Koeninger
st of the feedback: >>>> https://docs.google.com/document/d/1-Zdi_W-wtuxS9hTK0P9qb2x-nRanvXmnZ7SUi4qMljg/edit# >>>> >>>> I added a specific role for SPIP Author and another one for SPIP >>>> Shepherd. >>>> >>>> On Sat, Feb 11, 2017 at

Re: Spark Improvement Proposals

2017-02-16 Thread Sean Owen
e for SPIP Author and another one for SPIP Shepherd. > > On Sat, Feb 11, 2017 at 6:13 PM, Xiao Li wrote: > > During the summit, I also had a lot of discussions over similar topics > with multiple Committers and active users. I heard many fantastic ideas. I > believe Spark improvement prop

Re: Spark Improvement Proposals

2017-02-16 Thread Ryan Blue
c role for SPIP Author and another one for SPIP >>> Shepherd. >>> >>> On Sat, Feb 11, 2017 at 6:13 PM, Xiao Li wrote: >>> >>>> During the summit, I also had a lot of discussions over similar topics >>>> with multiple Committers and a

Re: Spark Improvement Proposals

2017-02-16 Thread Reynold Xin
i_W-wtuxS9h >> TK0P9qb2x-nRanvXmnZ7SUi4qMljg/edit# >> >> I added a specific role for SPIP Author and another one for SPIP Shepherd. >> >> On Sat, Feb 11, 2017 at 6:13 PM, Xiao Li wrote: >> >>> During the summit, I also had a lot of discussions ov

Re: Spark Improvement Proposals

2017-02-14 Thread Cody Koeninger
rote: > >> During the summit, I also had a lot of discussions over similar topics >> with multiple Committers and active users. I heard many fantastic ideas. I >> believe Spark improvement proposals are good channels to collect the >> requirements/designs. >> >>

Re: Spark Improvement Proposals

2017-02-13 Thread Reynold Xin
ummit, I also had a lot of discussions over similar topics > with multiple Committers and active users. I heard many fantastic ideas. I > believe Spark improvement proposals are good channels to collect the > requirements/designs. > > > IMO, we also need to consider the priority w

Re: Spark Improvement Proposals

2017-02-11 Thread Xiao Li
During the summit, I also had a lot of discussions over similar topics with multiple Committers and active users. I heard many fantastic ideas. I believe Spark improvement proposals are good channels to collect the requirements/designs. IMO, we also need to consider the priority when working on

Re: Spark Improvement Proposals

2017-02-11 Thread Cody Koeninger
At the spark summit this week, everyone from PMC members to users I had never met before were asking me about the Spark improvement proposals idea. It's clear that it's a real community need. But it's been almost half a year, and nothing visible has been done. Reynold, are you g

Re: Spark Improvement Proposals

2017-01-11 Thread Reynold Xin
orm and involve", rather than >>>>> just >>>>> >> >>> > "involve". SIPs should also have at least two emails that go >>>>> to >>>>> >> >>> > dev@. >>>>> >> >>>

Re: Spark Improvement Proposals

2017-01-05 Thread Tim Hunter
gt; >> >>> > On Tue, Nov 1, 2016 at 12:09 AM, Reynold Xin < >>>> r...@databricks.com> >>>> >> >>> > wrote: >>>> >> >>> >> >>>> >> >>> >> Most things looked OK to me too, although I do plan to take a >>>> >> >>> >> closer >&g

Re: Spark Improvement Proposals

2017-01-03 Thread Cody Koeninger
PM, Marcelo Vanzin >>> >> >>> >> >>> >> >>> >> wrote: >>> >> >>> >>> >>> >> >>> >>> The proposal looks OK to me. I assume, even though it's not >>> >> >>> >>> explicitly >>> >> >>> >>> called, that voting

Re: Spark Improvement Proposals

2017-01-03 Thread Imran Rashid
t;> >> >>> >>> candidate >> >> >>> >>> for a SIP, given the scope of the work. The document attached >> even >> >> >>> >>> somewhat matches the proposed format. So if anyone wants to try >> >> >&g

Re: Spark Improvement Proposals

2017-01-03 Thread Joseph Bradley
europe is over, are any committers > >> >>> >>> > interested > >> >>> >>> > in > >> >>> >>> > moving forward with this? > >> >>> >>> > > >> >>> >>&g

Re: Spark Improvement Proposals

2017-01-02 Thread Cody Koeninger
; >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> > https://github.com/koeninger/spark-1/blob/SIP-0/docs/spark-improvement-proposals.md >> >>> >>>

Re: Spark Improvement Proposals

2016-11-08 Thread Ryan Blue
;> >>> >> > >>> >>> >> > >>> >>> >> I didn't want to write "lets focus on Flink" or any other > >>> >>> >> framework. > >>> >>> >> The > >>> >>> >> idea with benchmarks was to show two

Re: Spark Improvement Proposals

2016-11-08 Thread Cody Koeninger
hat Spark is >>> >>> >> still on >>> >>> >> the >>> >>> >> top >>> >>> >> >>> >>> >> >>> >>> >> No more, no less. Benchmarks will be helpful, but I don't

Re: Spark Improvement Proposals

2016-11-07 Thread Reynold Xin
gt; >>> >> No more, no less. Benchmarks will be helpful, but I don't think >> >>> >> they're the >> >>> >> most important thing in Spark :) On the Spark main page there is >> still >> >>> >> chart >> >>> &g

Re: Spark Improvement Proposals

2016-11-07 Thread Reynold Xin
I, but much faster and optimized, comparable or > >>> >> even > >>> >> faster than other frameworks. > >>> >> > >>> >> > >>> >> About real-time streaming, I think it would be just good to see it > in > >&g

Re: Odp.: Spark Improvement Proposals

2016-11-07 Thread Cody Koeninger
help them. With SIPs it >>> >> would >>> >> be easier, I've just posted this example as "thing that may be changed >>> >> with >>> >> SIP". >>> >> >>> >> >>> >> I very like unificati

Re: Odp.: Spark Improvement Proposals

2016-11-07 Thread Reynold Xin
let's make easy API, but with strong background (articles, >> >> benchmarks, descriptions, etc) that shows that Spark is still modern >> >> framework. >> >> >> >> >> >> Maybe now my intention will be clearer :) As I said organizational &

Re: Python Spark Improvements (forked from Spark Improvement Proposals)

2016-11-01 Thread Holden Karau
; want some more community iteration on maybe? >> >> >> >> >> >> -- >> View this message in context: http://apache-spark-developers >> -list.1001551.n3.nabble.com/Python-Spark-Improvements- >&g

Re: Odp.: Spark Improvement Proposals

2016-11-01 Thread Reynold Xin
th strong background (articles, > >> benchmarks, descriptions, etc) that shows that Spark is still modern > >> framework. > >> > >> > >> Maybe now my intention will be clearer :) As I said organizational ideas > >> were already mentioned and I agree with them, my mai

Re: Python Spark Improvements (forked from Spark Improvement Proposals)

2016-10-31 Thread Holden Karau
e can try to use it. I think this is something that > we > want some more community iteration on maybe? > > > > > > -- > View this message in context: http://apache-spark- > developers-list.1001551.n3.nabble.com/Python-Spark- > Improvements-forked-from-Spark-Im

Re: Python Spark Improvements (forked from Spark Improvement Proposals)

2016-10-31 Thread mariusvniekerk
Spark-Improvement-Proposals-tp19422p19670.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Odp.: Spark Improvement Proposals

2016-10-31 Thread Marcelo Vanzin
earer :) As I said organizational ideas >> were already mentioned and I agree with them, my mail was just to show some >> aspects from my side, so from theside of developer and person who is trying >> to help others with Spark (via StackOverflow or other ways) >> >> &

Re: Odp.: Spark Improvement Proposals

2016-10-31 Thread Ryan Blue
> > > > > Maybe now my intention will be clearer :) As I said organizational ideas > > were already mentioned and I agree with them, my mail was just to show > some > > aspects from my side, so from theside of developer and person who is > trying > > to help

Re: Odp.: Spark Improvement Proposals

2016-10-31 Thread Cody Koeninger
y mail was just to show some > aspects from my side, so from theside of developer and person who is trying > to help others with Spark (via StackOverflow or other ways) > > > Pozdrawiam / Best regards, > > Tomasz > > > > Od: Cody Koeninger >

Odp.: Spark Improvement Proposals

2016-10-17 Thread Tomasz Gawęda
StackOverflow or other ways) Pozdrawiam / Best regards, Tomasz Od: Cody Koeninger Wysłane: 17 października 2016 16:46 Do: Debasish Das DW: Tomasz Gawęda; dev@spark.apache.org Temat: Re: Spark Improvement Proposals I think narrowly focusing on Flink or benchmarks is miss

Re: Spark Improvement Proposals

2016-10-17 Thread Cody Koeninger
I think narrowly focusing on Flink or benchmarks is missing my point. My point is evolve or die. Spark's governance and organization is hampering its ability to evolve technologically, and it needs to change. On Sun, Oct 16, 2016 at 9:21 PM, Debasish Das wrote: > Thanks Cody for bringing up a v

Re: Spark Improvement Proposals(Internet mail)

2016-10-17 Thread 黄明
n be. --- Sincerely Andy 原始邮件 发件人: Debasish Das 收件人: Tomasz Gawęda 抄送: dev@spark.apache.org; Cody Koeninger 发送时间: 2016年10月17日(周一) 10:21 主题: Re: Spark Improvement Proposals(Internet mail) Thanks Cody for bringing up a valid point...I picked up Spark in 2014 as soon as I looked into it since comp

Re: Spark Improvement Proposals

2016-10-16 Thread Debasish Das
Thanks Cody for bringing up a valid point...I picked up Spark in 2014 as soon as I looked into it since compared to writing Java map-reduce and Cascading code, Spark made writing distributed code fun...But now as we went deeper with Spark and real-time streaming use-case gets more prominent, I thin

Re: Spark Improvement Proposals

2016-10-16 Thread Tomasz Gawęda
Hi everyone, I'm quite late with my answer, but I think my suggestions may help a little bit. :) Many technical and organizational topics were mentioned, but I want to focus on these negative posts about Spark and about "haters" I really like Spark. Easy of use, speed, very good community - it'

Re: Python Spark Improvements (forked from Spark Improvement Proposals)

2016-10-14 Thread mariusvniekerk
s, progress bars, tab completion for spark configuration properties, easier loading of scala objects via py4j. -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Python-Spark-Improvements-forked-from-Spark-Improvement-Proposals-tp19422p19449.html Sent from

Re: Python Spark Improvements (forked from Spark Improvement Proposals)

2016-10-13 Thread Holden Karau
e+ > [hidden email] > <http:///user/SendEmail.jtp?type=node&node=19431&i=0>] > *Sent:* Thursday, October 13, 2016 3:51 AM > *To:* Mendelson, Assaf > *Subject:* Re: Python Spark Improvements (forked from Spark Improvement > Proposals) > > > > As ve

RE: Python Spark Improvements (forked from Spark Improvement Proposals)

2016-10-13 Thread assaf.mendelson
y taste. From: msukmanowsky [via Apache Spark Developers List] [mailto:ml-node+s1001551n19426...@n3.nabble.com] Sent: Thursday, October 13, 2016 3:51 AM To: Mendelson, Assaf Subject: Re: Python Spark Improvements (forked from Spark Improvement Proposals) As very heavy Spark users at Parse.ly, I just wanted

Re: Spark Improvement Proposals

2016-10-12 Thread kant kodali
> First we can always have other people suggest SIPs but mark >> them as >> >>>> >> > “unreviewed” and have committers basically move them forward. >> The >> >>>> >> > problem is >> >>>> >> > that wri

Re: Python Spark Improvements (forked from Spark Improvement Proposals)

2016-10-12 Thread msukmanowsky
is. Maybe I'm more used to this situation from using other frameworks that have similar concepts but incompatible implementations. We're big fans of PySpark and are happy to provide feedback and contribute wherever we can. -- View this message in context: http://apache-spark-developers-li

Re: Spark Improvement Proposals

2016-10-11 Thread Ryan Blue
;> >> > > >>>> >> > As for strategy, in many cases implementation strategy can affect > >>>> >> > the > >>>> >> > goals. > >>>> >> > I will give a small example: In the current structur

Re: Improving governance / committers (split from Spark Improvement Proposals thread)

2016-10-10 Thread Holden Karau
rse my disclaimer from the original conversation applies <http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-Improvement-Proposals-tp19268p19284.html> - I do very much "have a horse in the race" so I will avoid proposing new criteria. I working on Spark is a core part of what I d

Re: Spark Improvement Proposals

2016-10-10 Thread Cody Koeninger
> >>> those is definitely useful sometimes, but if you make this a >>>> >> >>> *required* >>>> >> >>> section, people are just going to fill it in with bogus stuff >>>> >> >>> (I've >>>> >> &g

Re: Spark Improvement Proposals

2016-10-10 Thread Mark Hamstra
n a set of all distinct >>> values. >>> >> > One >>> >> > way to implement this would be to make the set into a map and have >>> the >>> >> > value >>> >> > contain the last time seen. Multiplying it acros

Re: Spark Improvement Proposals

2016-10-10 Thread Mark Hamstra
I'm not a fan of the SEP acronym. Besides it prior established meaning of "Somebody else's problem", the are other inappropriate or offensive connotations such as this Australian slang that often gets shortened to just "sep": http://www.urbandictionary.com/define.php?term=Seppo On Sun, Oct 9, 20

Re: Spark Improvement Proposals

2016-10-10 Thread Mark Hamstra
y, it is easy for whoever goes to the design > >> > document to not think about these cases. Furthermore, it might be > >> > decided > >> > that these cases are rare enough so that the strategy is still good > >> > enough > >> > but how would w

Re: Spark Improvement Proposals

2016-10-10 Thread Cody Koeninger
gt;> problem is >>>>>>>>> that writing a good document takes time. This way we can leverage >>>>>>>>> non >>>>>>>>> committers to do some of this work (it is just another way to >>>>>>>>> contribute). &

Re: Spark Improvement Proposals

2016-10-10 Thread Steve Loughran
gt;>>>>> >>>>>>>> >>>>>>>> As for strategy, in many cases implementation strategy can affect >>>>>>>> the >>>>>>>> goals. >>>>>>>> I will give a small example: In the curren

Re: Spark Improvement Proposals

2016-10-10 Thread Matei Zaharia
mall example: In the current structured streaming >>>>>>> strategy, >>>>>>> we group by the time to achieve a sliding window. This is definitely >>>>>>> an >>>>>>> implementation decision and not a goal. However, I can think of >&

Re: Spark Improvement Proposals

2016-10-10 Thread Cody Koeninger
the time inside their calculation >> >>> > buffer. >> >>> > For example, let’s say we want to return a set of all distinct >> >>> > values. >> >>> > One >> >>> > way to implement this would be to make the set into

Re: Spark Improvement Proposals

2016-10-10 Thread Ryan Blue
> >>> > on > >>> > the type of aggregations and their performance which does affect the > >>> > goal. > >>> > Without adding the strategy, it is easy for whoever goes to the > design > >>> > document to not think about these

Re: Spark Improvement Proposals

2016-10-10 Thread Cody Koeninger
ions and their performance which does affect the >>> > goal. >>> > Without adding the strategy, it is easy for whoever goes to the design >>> > document to not think about these cases. Furthermore, it might be >>> > decided >>> > that these ca

Re: Spark Improvement Proposals

2016-10-10 Thread Ryan Blue
gt;> > I believe this example is exactly what Cody was talking about. Since >> many >> > times implementation strategies have a large effect on the goal, we >> should >> > have it discussed when discussing the goals. In addition, while it is >> often >

Re: Spark Improvement Proposals

2016-10-10 Thread Cody Koeninger
hat the strategy is still good >> > enough >> > but how would we know it without user feedback? >> > >> > I believe this example is exactly what Cody was talking about. Since >> > many >> > times implementation strategies have a large effect o

Re: Spark Improvement Proposals

2016-10-10 Thread Ryan Blue
le is exactly what Cody was talking about. Since many > > times implementation strategies have a large effect on the goal, we > should > > have it discussed when discussing the goals. In addition, while it is > often > > easy to throw out completely infeasible goals, it is often muc

Re: Spark Improvement Proposals

2016-10-10 Thread Cody Koeninger
gt; Assaf. > > > > From: Cody Koeninger-2 [via Apache Spark Developers List] > [mailto:ml-node+[hidden email]] > Sent: Monday, October 10, 2016 2:25 AM > To: Mendelson, Assaf > Subject: Re: Spark Improvement Proposals > > > > Only committers should formally submit S

RE: Spark Improvement Proposals

2016-10-09 Thread assaf.mendelson
ia Apache Spark Developers List] [mailto:ml-node+s1001551n19359...@n3.nabble.com] Sent: Monday, October 10, 2016 2:25 AM To: Mendelson, Assaf Subject: Re: Spark Improvement Proposals Only committers should formally submit SIPs because in an apache project only commiters have explicit political power. If a

Re: Spark Improvement Proposals

2016-10-09 Thread Cody Koeninger
w that the SIP is feasible to implement. One exception, >> >> however, is that I think we'll have some SIPs primarily on internals >> >> (e.g. >> >> if somebody wants to refactor Spark's query optimizer or something). >> >> >> >> -

Re: Spark Improvement Proposals

2016-10-09 Thread Nicholas Chammas
On Sun, Oct 9, 2016 at 5:19 PM Cody Koeninger wrote: > Regarding name, if the SIP overlap is a concern, we can pick a different > name. > > My tongue in cheek suggestion would be > > Spark Lightweight Improvement process (SPARKLI) > If others share my minor concern about the SIP name, I propose

Re: Spark Improvement Proposals

2016-10-09 Thread Matei Zaharia
designing > >> and implementing something? What if you discover that the strategy is > >> actually better when you start doing stuff? > >> > >> At a super high level, it depends on whether you want the SIPs to be PRDs > >> for gettin

Re: Spark Improvement Proposals

2016-10-09 Thread Cody Koeninger
something). >> >> >> >> - Rejected strategies: I personally wouldn't put this, because what's >> >> the >> >> point of voting to reject a strategy before you've really begun >> >> designing >> >> and imple

Re: Spark Improvement Proposals

2016-10-09 Thread Cody Koeninger
getting some quick feedback on the goals of a feature before it is > >> designed, or something more like full-fledged design docs (just a more > >> visible design doc for bigger changes). I looked at Kafka's KIPs, and > they > >> actually seem to be more like desig

Re: Spark Improvement Proposals

2016-10-09 Thread Ofir Manor
tart doing stuff? > >> > >> At a super high level, it depends on whether you want the SIPs to be > PRDs > >> for getting some quick feedback on the goals of a feature before it is > >> designed, or something more like full-fledged design docs (just a more >

Re: Spark Improvement Proposals

2016-10-09 Thread Matei Zaharia
oked at Kafka's KIPs, and they >> actually seem to be more like design docs. This can work too but it does >> require more work from the proposer and it can lead to the same problems you >> mentioned with people already having a design and implementation in mind. >> >&

Re: Spark Improvement Proposals

2016-10-09 Thread Cody Koeninger
e more work from the proposer and it can lead to the same problems you >> mentioned with people already having a design and implementation in mind. >> >> Basically, the question is, are you trying to iterate faster on design by >> adding a step for user feedback earlier? O

Re: Spark Improvement Proposals

2016-10-09 Thread Cody Koeninger
step for user feedback earlier? Or are you just trying to make > design docs for key features more visible (and their approval more formal)? > > BTW note that in either case, I'd like to have a template for design docs > too, which should also include goals. I think that would

Re: Spark Improvement Proposals

2016-10-09 Thread Cody Koeninger
ter they are approved when revisions are agreed upon. PEPs >> are also intended for wide consumption, vs. bug tracker issues which the >> broader Python dev community are not expected to follow closely. >> >> Dunno if we want to follow a similar pattern for Spark, since the >

Re: Spark Improvement Proposals

2016-10-09 Thread Nicholas Chammas
e broader Python dev > community are not expected to follow closely. > > Dunno if we want to follow a similar pattern for Spark, since the > project’s needs are different. But the Python community has used PEPs to > help organize and steer development since 2000; there are plen

Re: Spark Improvement Proposals

2016-10-09 Thread Matei Zaharia
community has used PEPs to help organize > and steer development since 2000; there are plenty of examples there we can > probably take inspiration from. > > By the way, can we call these things something other than Spark Improvement > Proposals? The acronym, SIP, conflicts with Scala S

Re: Spark Improvement Proposals

2016-10-09 Thread Nicholas Chammas
0; there are plenty of examples there we can probably take inspiration from. By the way, can we call these things something other than Spark Improvement Proposals? The acronym, SIP, conflicts with Scala SIPs <http://docs.scala-lang.org/sips/index.html>. Since the Scala and Spark communities hav

Re: Spark Improvement Proposals

2016-10-09 Thread Matei Zaharia
some of the issues you brought up. Matei > On Oct 9, 2016, at 10:40 AM, Cody Koeninger wrote: > > Here's my specific proposal (meta-proposal?) > > Spark Improvement Proposals (SIP) > > > > Background: > > The current problem is that design and implement

Re: Spark Improvement Proposals

2016-10-09 Thread Cody Koeninger
Here's my specific proposal (meta-proposal?) Spark Improvement Proposals (SIP) Background: The current problem is that design and implementation of large features are often done in private, before soliciting user feedback. When feedback is solicited, it is often as to detailed d

Re: Improving governance / committers (split from Spark Improvement Proposals thread)

2016-10-08 Thread Cody Koeninger
It's not about technical design disagreement as to matters of taste, it's about familiarity with the domain. To make an analogy, it's as if a committer in MLlib was firmly intent on, I dunno, treating a collection of categorical variables as if it were an ordered range of continuous variables. It

Re: Improving volunteer management / JIRAs (split from Spark Improvement Proposals thread)

2016-10-08 Thread vaquar khan
>> Thanks, >> >> Assaf. >> >> >> >> >> >> >> >> *From:* Nicholas Chammas [via Apache Spark Developers List] [mailto: >> ml-node+[hidden email] >> <http://user/SendEmail.jtp?type=node&node=19322&

Re: Spark Improvement Proposals

2016-10-08 Thread vaquar khan
+1 for SIP lebles,waiting for Reynolds detailed proposal . Regards, Vaquar khan On 8 Oct 2016 16:22, "Matei Zaharia" wrote: > Sounds good. Just to comment on the compatibility part: > > > I meant changing public user interfaces. I think the first design is > > unlikely to be right, because it'

  1   2   >