That seems reasonable to me. I do not want to see lazy consensus used on one of these proposals though, I want a clear outcome, i.e. call for a vote, wait at least 72 hours, get three +1s and no vetos.
On Mon, Oct 10, 2016 at 2:15 PM, Ryan Blue <rb...@netflix.com> wrote: > Proposal submission: I think we should keep this as open as possible. If > there is a problem with too many open proposals, then we should tackle that > as a fix rather than excluding participation. Perhaps it will end up that > way, but I think it's worth trying a more open model first. > > Majority vs consensus: My rationale is that I don't think we want to > consider a proposal approved if it had objections serious enough that > committers down-voted (or PMC depending on who gets a vote). If these > proposals are like PEPs, then they represent a significant amount of > community effort and I wouldn't want to move forward if up to half of the > community thinks it's an untenable idea. > > rb > > On Mon, Oct 10, 2016 at 12:07 PM, Cody Koeninger <c...@koeninger.org> wrote: >> >> I think this is closer to a procedural issue than a code modification >> issue, hence why majority. If everyone thinks consensus is better, I >> don't care. Again, I don't feel strongly about the way we achieve >> clarity, just that we achieve clarity. >> >> On Mon, Oct 10, 2016 at 2:02 PM, Ryan Blue <rb...@netflix.com> wrote: >> > Sorry, I missed that the proposal includes majority approval. Why >> > majority >> > instead of consensus? I think we want to build consensus around these >> > proposals and it makes sense to discuss until no one would veto. >> > >> > rb >> > >> > On Mon, Oct 10, 2016 at 11:54 AM, Ryan Blue <rb...@netflix.com> wrote: >> >> >> >> +1 to votes to approve proposals. I agree that proposals should have an >> >> official mechanism to be accepted, and a vote is an established means >> >> of >> >> doing that well. I like that it includes a period to review the >> >> proposal and >> >> I think proposals should have been discussed enough ahead of a vote to >> >> survive the possibility of a veto. >> >> >> >> I also like the names that are short and (mostly) unique, like SEP. >> >> >> >> Where I disagree is with the requirement that a committer must formally >> >> propose an enhancement. I don't see the value of restricting this: if >> >> someone has the will to write up a proposal then they should be >> >> encouraged >> >> to do so and start a discussion about it. Even if there is a political >> >> reality as Cody says, what is the value of codifying that in our >> >> process? I >> >> think restricting who can submit proposals would only undermine them by >> >> pushing contributors out. Maybe I'm missing something here? >> >> >> >> rb >> >> >> >> >> >> >> >> On Mon, Oct 10, 2016 at 7:41 AM, Cody Koeninger <c...@koeninger.org> >> >> wrote: >> >>> >> >>> Yes, users suggesting SIPs is a good thing and is explicitly called >> >>> out in the linked document under the Who? section. Formally proposing >> >>> them, not so much, because of the political realities. >> >>> >> >>> Yes, implementation strategy definitely affects goals. There are all >> >>> kinds of examples of this, I'll pick one that's my fault so as to >> >>> avoid sounding like I'm blaming: >> >>> >> >>> When I implemented the Kafka DStream, one of my (not explicitly agreed >> >>> upon by the community) goals was to make sure people could use the >> >>> Dstream with however they were already using Kafka at work. The lack >> >>> of explicit agreement on that goal led to all kinds of fighting with >> >>> committers, that could have been avoided. The lack of explicit >> >>> up-front strategy discussion led to the DStream not really working >> >>> with compacted topics. I knew about compacted topics, but don't have >> >>> a use for them, so had a blind spot there. If there was explicit >> >>> up-front discussion that my strategy was "assume that batches can be >> >>> defined on the driver solely by beginning and ending offsets", there's >> >>> a greater chance that a user would have seen that and said, "hey, what >> >>> about non-contiguous offsets in a compacted topic". >> >>> >> >>> This kind of thing is only going to happen smoothly if we have a >> >>> lightweight user-visible process with clear outcomes. >> >>> >> >>> On Mon, Oct 10, 2016 at 1:34 AM, assaf.mendelson >> >>> <assaf.mendel...@rsa.com> wrote: >> >>> > I agree with most of what Cody said. >> >>> > >> >>> > Two things: >> >>> > >> >>> > First we can always have other people suggest SIPs but mark them as >> >>> > “unreviewed” and have committers basically move them forward. The >> >>> > problem is >> >>> > that writing a good document takes time. This way we can leverage >> >>> > non >> >>> > committers to do some of this work (it is just another way to >> >>> > contribute). >> >>> > >> >>> > >> >>> > >> >>> > As for strategy, in many cases implementation strategy can affect >> >>> > the >> >>> > goals. >> >>> > I will give a small example: In the current structured streaming >> >>> > strategy, >> >>> > we group by the time to achieve a sliding window. This is definitely >> >>> > an >> >>> > implementation decision and not a goal. However, I can think of >> >>> > several >> >>> > aggregation functions which have the time inside their calculation >> >>> > buffer. >> >>> > For example, let’s say we want to return a set of all distinct >> >>> > values. >> >>> > One >> >>> > way to implement this would be to make the set into a map and have >> >>> > the >> >>> > value >> >>> > contain the last time seen. Multiplying it across the groupby would >> >>> > cost a >> >>> > lot in performance. So adding such a strategy would have a great >> >>> > effect >> >>> > on >> >>> > the type of aggregations and their performance which does affect the >> >>> > goal. >> >>> > Without adding the strategy, it is easy for whoever goes to the >> >>> > design >> >>> > document to not think about these cases. Furthermore, it might be >> >>> > decided >> >>> > that these cases are rare enough so that the strategy is still good >> >>> > enough >> >>> > but how would we know it without user feedback? >> >>> > >> >>> > I believe this example is exactly what Cody was talking about. Since >> >>> > many >> >>> > times implementation strategies have a large effect on the goal, we >> >>> > should >> >>> > have it discussed when discussing the goals. In addition, while it >> >>> > is >> >>> > often >> >>> > easy to throw out completely infeasible goals, it is often much >> >>> > harder >> >>> > to >> >>> > figure out that the goals are unfeasible without fine tuning. >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > Assaf. >> >>> > >> >>> > >> >>> > >> >>> > From: Cody Koeninger-2 [via Apache Spark Developers List] >> >>> > [mailto:ml-node+[hidden email]] >> >>> > Sent: Monday, October 10, 2016 2:25 AM >> >>> > To: Mendelson, Assaf >> >>> > Subject: Re: Spark Improvement Proposals >> >>> > >> >>> > >> >>> > >> >>> > Only committers should formally submit SIPs because in an apache >> >>> > project only commiters have explicit political power. If a user >> >>> > can't >> >>> > find a commiter willing to sponsor an SIP idea, they have no way to >> >>> > get the idea passed in any case. If I can't find a committer to >> >>> > sponsor this meta-SIP idea, I'm out of luck. >> >>> > >> >>> > I do not believe unrealistic goals can be found solely by >> >>> > inspection. >> >>> > We've managed to ignore unrealistic goals even after implementation! >> >>> > Focusing on APIs can allow people to think they've solved something, >> >>> > when there's really no way of implementing that API while meeting >> >>> > the >> >>> > goals. Rapid iteration is clearly the best way to address this, but >> >>> > we've already talked about why that hasn't really worked. If adding >> >>> > a >> >>> > non-binding API section to the template is important to you, I'm not >> >>> > against it, but I don't think it's sufficient. >> >>> > >> >>> > On your PRD vs design doc spectrum, I'm saying this is closer to a >> >>> > PRD. Clear agreement on goals is the most important thing and >> >>> > that's >> >>> > why it's the thing I want binding agreement on. But I cannot agree >> >>> > to >> >>> > goals unless I have enough minimal technical info to judge whether >> >>> > the >> >>> > goals are likely to actually be accomplished. >> >>> > >> >>> > >> >>> > >> >>> > On Sun, Oct 9, 2016 at 5:35 PM, Matei Zaharia <[hidden email]> >> >>> > wrote: >> >>> > >> >>> > >> >>> >> Well, I think there are a few things here that don't make sense. >> >>> >> First, >> >>> >> why >> >>> >> should only committers submit SIPs? Development in the project >> >>> >> should >> >>> >> be >> >>> >> open to all contributors, whether they're committers or not. >> >>> >> Second, I >> >>> >> think >> >>> >> unrealistic goals can be found just by inspecting the goals, and >> >>> >> I'm >> >>> >> not >> >>> >> super worried that we'll accept a lot of SIPs that are then >> >>> >> infeasible >> >>> >> -- >> >>> >> we >> >>> >> can then submit new ones. But this depends on whether you want this >> >>> >> process >> >>> >> to be a "design doc lite", where people also agree on >> >>> >> implementation >> >>> >> strategy, or just a way to agree on goals. This is what I asked >> >>> >> earlier >> >>> >> about PRDs vs design docs (and I'm open to either one but I'd just >> >>> >> like >> >>> >> clarity). Finally, both as a user and designer of software, I >> >>> >> always >> >>> >> want >> >>> >> to >> >>> >> give feedback on APIs, so I'd really like a culture of having those >> >>> >> early. >> >>> >> People don't argue about prettiness when they discuss APIs, they >> >>> >> argue >> >>> >> about >> >>> >> the core concepts to expose in order to meet various goals, and >> >>> >> then >> >>> >> they're >> >>> >> stuck maintaining those for a long time. >> >>> >> >> >>> >> Matei >> >>> >> >> >>> >> On Oct 9, 2016, at 3:10 PM, Cody Koeninger <[hidden email]> wrote: >> >>> >> >> >>> >> Users instead of people, sure. Commiters and contributors are (or >> >>> >> at >> >>> >> least >> >>> >> should be) a subset of users. >> >>> >> >> >>> >> Non goals, sure. I don't care what the name is, but we need to >> >>> >> clearly >> >>> >> say >> >>> >> e.g. 'no we are not maintaining compatibility with XYZ right now'. >> >>> >> >> >>> >> API, what I care most about is whether it allows me to accomplish >> >>> >> the >> >>> >> goals. >> >>> >> Arguing about how ugly or pretty it is can be saved for design/ >> >>> >> implementation imho. >> >>> >> >> >>> >> Strategy, this is necessary because otherwise goals can be out of >> >>> >> line >> >>> >> with >> >>> >> reality. Don't propose goals you don't have at least some idea of >> >>> >> how >> >>> >> to >> >>> >> implement. >> >>> >> >> >>> >> Rejected strategies, given that commiters are the only ones I'm >> >>> >> saying >> >>> >> should formally submit SPARKLIs or SIPs, if they put junk in a >> >>> >> required >> >>> >> section then slap them down for it and tell them to fix it. >> >>> >> >> >>> >> >> >>> >> On Oct 9, 2016 4:36 PM, "Matei Zaharia" <[hidden email]> wrote: >> >>> >>> >> >>> >>> Yup, this is the stuff that I found unclear. Thanks for clarifying >> >>> >>> here, >> >>> >>> but we should also clarify it in the writeup. In particular: >> >>> >>> >> >>> >>> - Goals needs to be about user-facing behavior ("people" is broad) >> >>> >>> >> >>> >>> - I'd rename Rejected Goals to Non-Goals. Otherwise someone will >> >>> >>> dig >> >>> >>> up >> >>> >>> one of these and say "Spark's developers have officially rejected >> >>> >>> X, >> >>> >>> which >> >>> >>> our awesome system has". >> >>> >>> >> >>> >>> - For user-facing stuff, I think you need a section on API. >> >>> >>> Virtually >> >>> >>> all >> >>> >>> other *IPs I've seen have that. >> >>> >>> >> >>> >>> - I'm still not sure why the strategy section is needed if the >> >>> >>> purpose is >> >>> >>> to define user-facing behavior -- unless this is the strategy for >> >>> >>> setting >> >>> >>> the goals or for defining the API. That sounds squarely like a >> >>> >>> design >> >>> >>> doc >> >>> >>> issue. In some sense, who cares whether the proposal is >> >>> >>> technically >> >>> >>> feasible >> >>> >>> right now? If it's infeasible, that will be discovered later >> >>> >>> during >> >>> >>> design >> >>> >>> and implementation. Same thing with rejected strategies -- listing >> >>> >>> some >> >>> >>> of >> >>> >>> those is definitely useful sometimes, but if you make this a >> >>> >>> *required* >> >>> >>> section, people are just going to fill it in with bogus stuff >> >>> >>> (I've >> >>> >>> seen >> >>> >>> this happen before). >> >>> >>> >> >>> >>> Matei >> >>> >>> >> >>> > >> >>> >>> > On Oct 9, 2016, at 2:14 PM, Cody Koeninger <[hidden email]> >> >>> >>> > wrote: >> >>> >>> > >> >>> >>> > So to focus the discussion on the specific strategy I'm >> >>> >>> > suggesting, >> >>> >>> > documented at >> >>> >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> > https://github.com/koeninger/spark-1/blob/SIP-0/docs/spark-improvement-proposals.md >> >>> >>> > >> >>> >>> > "Goals: What must this allow people to do, that they can't >> >>> >>> > currently?" >> >>> >>> > >> >>> >>> > Is it unclear that this is focusing specifically on >> >>> >>> > people-visible >> >>> >>> > behavior? >> >>> >>> > >> >>> >>> > Rejected goals - are important because otherwise people keep >> >>> >>> > trying >> >>> >>> > to argue about scope. Of course you can change things later >> >>> >>> > with a >> >>> >>> > different SIP and different vote, the point is to focus. >> >>> >>> > >> >>> >>> > Use cases - are something that people are going to bring up in >> >>> >>> > discussion. If they aren't clearly documented as a goal ("This >> >>> >>> > must >> >>> >>> > allow me to connect using SSL"), they should be added. >> >>> >>> > >> >>> >>> > Internal architecture - if the people who need specific behavior >> >>> >>> > are >> >>> >>> > implementers of other parts of the system, that's fine. >> >>> >>> > >> >>> >>> > Rejected strategies - If you have none of these, you have no >> >>> >>> > evidence >> >>> >>> > that the proponent didn't just go with the first thing they had >> >>> >>> > in >> >>> >>> > mind (or have already implemented), which is a big problem >> >>> >>> > currently. >> >>> >>> > Approval isn't binding as to specifics of implementation, so >> >>> >>> > these >> >>> >>> > aren't handcuffs. The goals are the contract, the strategy is >> >>> >>> > evidence that contract can actually be met. >> >>> >>> > >> >>> >>> > Design docs - I'm not touching design docs. The markdown file I >> >>> >>> > linked specifically says of the strategy section "This is not a >> >>> >>> > full >> >>> >>> > design document." Is this unclear? Design docs can be worked >> >>> >>> > on >> >>> >>> > obviously, but that's not what I'm concerned with here. >> >>> >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> > On Sun, Oct 9, 2016 at 2:34 PM, Matei Zaharia <[hidden email]> >> >>> >>> > wrote: >> >>> >>> >> Hi Cody, >> >>> >>> >> >> >>> >>> >> I think this would be a lot more concrete if we had a more >> >>> >>> >> detailed >> >>> >>> >> template >> >>> >>> >> for SIPs. Right now, it's not super clear what's in scope -- >> >>> >>> >> e.g. >> >>> >>> >> are >> >>> >>> >> they >> >>> >>> >> a way to solicit feedback on the user-facing behavior or on the >> >>> >>> >> internals? >> >>> >>> >> "Goals" can cover both things. I've been thinking of SIPs more >> >>> >>> >> as >> >>> >>> >> Product >> >>> >>> >> Requirements Docs (PRDs), which focus on *what* a code change >> >>> >>> >> should >> >>> >>> >> do >> >>> >>> >> as >> >>> >>> >> opposed to how. >> >>> >>> >> >> >>> >>> >> In particular, here are some things that you may or may not >> >>> >>> >> consider >> >>> >>> >> in >> >>> >>> >> scope for SIPs: >> >>> >>> >> >> >>> >>> >> - Goals and non-goals: This is definitely in scope, and IMO >> >>> >>> >> should >> >>> >>> >> focus on >> >>> >>> >> user-visible behavior (e.g. "system supports SQL window >> >>> >>> >> functions" >> >>> >>> >> or >> >>> >>> >> "system continues working if one node fails"). BTW I wouldn't >> >>> >>> >> say >> >>> >>> >> "rejected >> >>> >>> >> goals" because some of them might become goals later, so we're >> >>> >>> >> not >> >>> >>> >> definitively rejecting them. >> >>> >>> >> >> >>> >>> >> - Public API: Probably should be included in most SIPs unless >> >>> >>> >> it's >> >>> >>> >> too >> >>> >>> >> large >> >>> >>> >> to fully specify then (e.g. "let's add an ML library"). >> >>> >>> >> >> >>> >>> >> - Use cases: I usually find this very useful in PRDs to better >> >>> >>> >> communicate >> >>> >>> >> the goals. >> >>> >>> >> >> >>> >>> >> - Internal architecture: This is usually *not* a thing users >> >>> >>> >> can >> >>> >>> >> easily >> >>> >>> >> comment on and it sounds more like a design doc item. Of course >> >>> >>> >> it's >> >>> >>> >> important to show that the SIP is feasible to implement. One >> >>> >>> >> exception, >> >>> >>> >> however, is that I think we'll have some SIPs primarily on >> >>> >>> >> internals >> >>> >>> >> (e.g. >> >>> >>> >> if somebody wants to refactor Spark's query optimizer or >> >>> >>> >> something). >> >>> >>> >> >> >>> >>> >> - Rejected strategies: I personally wouldn't put this, because >> >>> >>> >> what's >> >>> >>> >> the >> >>> >>> >> point of voting to reject a strategy before you've really begun >> >>> >>> >> designing >> >>> >>> >> and implementing something? What if you discover that the >> >>> >>> >> strategy >> >>> >>> >> is >> >>> >>> >> actually better when you start doing stuff? >> >>> >>> >> >> >>> >>> >> At a super high level, it depends on whether you want the SIPs >> >>> >>> >> to >> >>> >>> >> be >> >>> >>> >> PRDs >> >>> >>> >> for getting some quick feedback on the goals of a feature >> >>> >>> >> before >> >>> >>> >> it is >> >>> >>> >> designed, or something more like full-fledged design docs (just >> >>> >>> >> a >> >>> >>> >> more >> >>> >>> >> visible design doc for bigger changes). I looked at Kafka's >> >>> >>> >> KIPs, >> >>> >>> >> and >> >>> >>> >> they >> >>> >>> >> actually seem to be more like design docs. This can work too >> >>> >>> >> but >> >>> >>> >> it >> >>> >>> >> does >> >>> >>> >> require more work from the proposer and it can lead to the same >> >>> >>> >> problems you >> >>> >>> >> mentioned with people already having a design and >> >>> >>> >> implementation >> >>> >>> >> in >> >>> >>> >> mind. >> >>> >>> >> >> >>> >>> >> Basically, the question is, are you trying to iterate faster on >> >>> >>> >> design >> >>> >>> >> by >> >>> >>> >> adding a step for user feedback earlier? Or are you just trying >> >>> >>> >> to >> >>> >>> >> make >> >>> >>> >> design docs for key features more visible (and their approval >> >>> >>> >> more >> >>> >>> >> formal)? >> >>> >>> >> >> >>> >>> >> BTW note that in either case, I'd like to have a template for >> >>> >>> >> design >> >>> >>> >> docs >> >>> >>> >> too, which should also include goals. I think that would've >> >>> >>> >> avoided >> >>> >>> >> some of >> >>> >>> >> the issues you brought up. >> >>> >>> >> >> >>> >>> >> Matei >> >>> >>> >> >> >>> >>> >> On Oct 9, 2016, at 10:40 AM, Cody Koeninger <[hidden email]> >> >>> >>> >> wrote: >> >>> >>> >> >> >>> >>> >> Here's my specific proposal (meta-proposal?) >> >>> >>> >> >> >>> >>> >> Spark Improvement Proposals (SIP) >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> Background: >> >>> >>> >> >> >>> >>> >> The current problem is that design and implementation of large >> >>> >>> >> features >> >>> >>> >> are >> >>> >>> >> often done in private, before soliciting user feedback. >> >>> >>> >> >> >>> >>> >> When feedback is solicited, it is often as to detailed design >> >>> >>> >> specifics, not >> >>> >>> >> focused on goals. >> >>> >>> >> >> >>> >>> >> When implementation does take place after design, there is >> >>> >>> >> often >> >>> >>> >> disagreement as to what goals are or are not in scope. >> >>> >>> >> >> >>> >>> >> This results in commits that don't fully meet user needs. >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> Goals: >> >>> >>> >> >> >>> >>> >> - Ensure user, contributor, and committer goals are clearly >> >>> >>> >> identified >> >>> >>> >> and >> >>> >>> >> agreed upon, before implementation takes place. >> >>> >>> >> >> >>> >>> >> - Ensure that a technically feasible strategy is chosen that is >> >>> >>> >> likely >> >>> >>> >> to >> >>> >>> >> meet the goals. >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> Rejected Goals: >> >>> >>> >> >> >>> >>> >> - SIPs are not for detailed design. Design by committee >> >>> >>> >> doesn't >> >>> >>> >> work. >> >>> >>> >> >> >>> >>> >> - SIPs are not for every change. We dont need that much >> >>> >>> >> process. >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> Strategy: >> >>> >>> >> >> >>> >>> >> My suggestion is outlined as a Spark Improvement Proposal >> >>> >>> >> process >> >>> >>> >> documented >> >>> >>> >> at >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> https://github.com/koeninger/spark-1/blob/SIP-0/docs/spark-improvement-proposals.md >> >>> >>> >> >> >>> >>> >> Specifics of Jira manipulation are an implementation detail we >> >>> >>> >> can >> >>> >>> >> figure >> >>> >>> >> out. >> >>> >>> >> >> >>> >>> >> I'm suggesting voting; the need here is for a _clear_ outcome. >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> Rejected Strategies: >> >>> >>> >> >> >>> >>> >> Having someone who understands the problem implement it first >> >>> >>> >> works, >> >>> >>> >> but >> >>> >>> >> only if significant iteration after user feedback is allowed. >> >>> >>> >> >> >>> >>> >> Historically this has been problematic due to pressure to limit >> >>> >>> >> public >> >>> >>> >> api >> >>> >>> >> changes. >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> On Fri, Oct 7, 2016 at 5:16 PM, Reynold Xin <[hidden email]> >> >>> >>> >> wrote: >> >>> >>> >>> >> >>> >>> >>> Alright looks like there are quite a bit of support. We should >> >>> >>> >>> wait >> >>> >>> >>> to >> >>> >>> >>> hear from more people too. >> >>> >>> >>> >> >>> >>> >>> To push this forward, Cody and I will be working together in >> >>> >>> >>> the >> >>> >>> >>> next >> >>> >>> >>> couple of weeks to come up with a concrete, detailed proposal >> >>> >>> >>> on >> >>> >>> >>> what >> >>> >>> >>> this >> >>> >>> >>> entails, and then we can discuss this the specific proposal as >> >>> >>> >>> well. >> >>> >>> >>> >> >>> >>> >>> >> >>> >>> >>> On Fri, Oct 7, 2016 at 2:29 PM, Cody Koeninger <[hidden >> >>> >>> >>> email]> >> >>> >>> >>> wrote: >> >>> >>> >>>> >> >>> >>> >>>> Yeah, in case it wasn't clear, I was talking about SIPs for >> >>> >>> >>>> major >> >>> >>> >>>> user-facing or cross-cutting changes, not minor feature adds. >> >>> >>> >>>> >> >>> >>> >>>> On Fri, Oct 7, 2016 at 3:58 PM, Stavros Kontopoulos >> >>> >>> >>>> <[hidden email]> wrote: >> >>> >>> >>>>> >> >>> >>> >>>>> +1 to the SIP label as long as it does not slow down things >> >>> >>> >>>>> and >> >>> >>> >>>>> it >> >>> >>> >>>>> targets optimizing efforts, coordination etc. For example >> >>> >>> >>>>> really >> >>> >>> >>>>> small >> >>> >>> >>>>> features should not need to go through this process >> >>> >>> >>>>> (assuming >> >>> >>> >>>>> they >> >>> >>> >>>>> dont >> >>> >>> >>>>> touch public interfaces) or re-factorings and hope it will >> >>> >>> >>>>> be >> >>> >>> >>>>> kept >> >>> >>> >>>>> this >> >>> >>> >>>>> way. So as a guideline doc should be provided, like in the >> >>> >>> >>>>> KIP >> >>> >>> >>>>> case. >> >>> >>> >>>>> >> >>> >>> >>>>> IMHO so far aside from tagging things and linking them >> >>> >>> >>>>> elsewhere >> >>> >>> >>>>> simply >> >>> >>> >>>>> having design docs and prototypes implementations in PRs is >> >>> >>> >>>>> not >> >>> >>> >>>>> something >> >>> >>> >>>>> that has not worked so far. What is really a pain in many >> >>> >>> >>>>> projects >> >>> >>> >>>>> out there >> >>> >>> >>>>> is discontinuity in progress of PRs, missing features, slow >> >>> >>> >>>>> reviews >> >>> >>> >>>>> which is >> >>> >>> >>>>> understandable to some extent... it is not only about Spark >> >>> >>> >>>>> but >> >>> >>> >>>>> things can >> >>> >>> >>>>> be improved for sure for this project in particular as >> >>> >>> >>>>> already >> >>> >>> >>>>> stated. >> >>> >>> >>>>> >> >>> >>> >>>>> On Fri, Oct 7, 2016 at 11:14 PM, Cody Koeninger <[hidden >> >>> >>> >>>>> email]> >> >>> >>> >>>>> wrote: >> >>> >>> >>>>>> >> >>> >>> >>>>>> +1 to adding an SIP label and linking it from the website. >> >>> >>> >>>>>> I >> >>> >>> >>>>>> think >> >>> >>> >>>>>> it >> >>> >>> >>>>>> needs >> >>> >>> >>>>>> >> >>> >>> >>>>>> - template that focuses it towards soliciting user goals / >> >>> >>> >>>>>> non >> >>> >>> >>>>>> goals >> >>> >>> >>>>>> - clear resolution as to which strategy was chosen to >> >>> >>> >>>>>> pursue. >> >>> >>> >>>>>> I'd >> >>> >>> >>>>>> recommend a vote. >> >>> >>> >>>>>> >> >>> >>> >>>>>> Matei asked me to clarify what I meant by changing >> >>> >>> >>>>>> interfaces, >> >>> >>> >>>>>> I >> >>> >>> >>>>>> think >> >>> >>> >>>>>> it's directly relevant to the SIP idea so I'll clarify >> >>> >>> >>>>>> here, >> >>> >>> >>>>>> and >> >>> >>> >>>>>> split >> >>> >>> >>>>>> a thread for the other discussion per Nicholas' request. >> >>> >>> >>>>>> >> >>> >>> >>>>>> I meant changing public user interfaces. I think the first >> >>> >>> >>>>>> design >> >>> >>> >>>>>> is >> >>> >>> >>>>>> unlikely to be right, because it's done at a time when you >> >>> >>> >>>>>> have >> >>> >>> >>>>>> the >> >>> >>> >>>>>> least information. As a user, I find it considerably more >> >>> >>> >>>>>> frustrating >> >>> >>> >>>>>> to be unable to use a tool to get my job done, than I do >> >>> >>> >>>>>> having to >> >>> >>> >>>>>> make minor changes to my code in order to take advantage of >> >>> >>> >>>>>> features. >> >>> >>> >>>>>> I've seen committers be seriously reluctant to allow >> >>> >>> >>>>>> changes >> >>> >>> >>>>>> to >> >>> >>> >>>>>> @experimental code that are needed in order for it to >> >>> >>> >>>>>> really >> >>> >>> >>>>>> work >> >>> >>> >>>>>> right. You need to be able to iterate, and if people on >> >>> >>> >>>>>> both >> >>> >>> >>>>>> sides >> >>> >>> >>>>>> of >> >>> >>> >>>>>> the fence aren't going to respect that some newer apis are >> >>> >>> >>>>>> subject >> >>> >>> >>>>>> to >> >>> >>> >>>>>> change, then why even mark them as such? >> >>> >>> >>>>>> >> >>> >>> >>>>>> Ideally a finished SIP should give me a checklist of things >> >>> >>> >>>>>> that >> >>> >>> >>>>>> an >> >>> >>> >>>>>> implementation must do, and things that it doesn't need to >> >>> >>> >>>>>> do. >> >>> >>> >>>>>> Contributors/committers should be seriously discouraged >> >>> >>> >>>>>> from >> >>> >>> >>>>>> putting >> >>> >>> >>>>>> out a version 0.1 that doesn't have at least a prototype >> >>> >>> >>>>>> implementation of all those things, especially if they're >> >>> >>> >>>>>> then >> >>> >>> >>>>>> going >> >>> >>> >>>>>> to argue against interface changes necessary to get the the >> >>> >>> >>>>>> rest >> >>> >>> >>>>>> of >> >>> >>> >>>>>> the things done in the 0.2 version. >> >>> >>> >>>>>> >> >>> >>> >>>>>> >> >>> >>> >>>>>> On Fri, Oct 7, 2016 at 2:18 PM, Reynold Xin <[hidden >> >>> >>> >>>>>> email]> >> >>> >>> >>>>>> wrote: >> >>> >>> >>>>>>> I like the lightweight proposal to add a SIP label. >> >>> >>> >>>>>>> >> >>> >>> >>>>>>> During Spark 2.0 development, Tom (Graves) and I suggested >> >>> >>> >>>>>>> using >> >>> >>> >>>>>>> wiki >> >>> >>> >>>>>>> to >> >>> >>> >>>>>>> track the list of major changes, but that never really >> >>> >>> >>>>>>> materialized >> >>> >>> >>>>>>> due to >> >>> >>> >>>>>>> the overhead. Adding a SIP label on major JIRAs and then >> >>> >>> >>>>>>> link >> >>> >>> >>>>>>> to >> >>> >>> >>>>>>> them >> >>> >>> >>>>>>> prominently on the Spark website makes a lot of sense. >> >>> >>> >>>>>>> >> >>> >>> >>>>>>> >> >>> >>> >>>>>>> On Fri, Oct 7, 2016 at 10:50 AM, Matei Zaharia >> >>> >>> >>>>>>> <[hidden email]> >> >>> >>> >>>>>>> wrote: >> >>> >>> >>>>>>>> >> >>> >>> >>>>>>>> For the improvement proposals, I think one major point >> >>> >>> >>>>>>>> was >> >>> >>> >>>>>>>> to >> >>> >>> >>>>>>>> make >> >>> >>> >>>>>>>> them >> >>> >>> >>>>>>>> really visible to users who are not contributors, so we >> >>> >>> >>>>>>>> should >> >>> >>> >>>>>>>> do >> >>> >>> >>>>>>>> more than >> >>> >>> >>>>>>>> sending stuff to dev@. One very lightweight idea is to >> >>> >>> >>>>>>>> have >> >>> >>> >>>>>>>> a >> >>> >>> >>>>>>>> new >> >>> >>> >>>>>>>> type of >> >>> >>> >>>>>>>> JIRA called a SIP and have a link to a filter that shows >> >>> >>> >>>>>>>> all >> >>> >>> >>>>>>>> such >> >>> >>> >>>>>>>> JIRAs from >> >>> >>> >>>>>>>> http://spark.apache.org. I also like the idea of SIP and >> >>> >>> >>>>>>>> design >> >>> >>> >>>>>>>> doc >> >>> >>> >>>>>>>> templates (in fact many projects have them). >> >>> >>> >>>>>>>> >> >>> >>> >>>>>>>> Matei >> >>> >>> >>>>>>>> >> >>> >>> >>>>>>>> On Oct 7, 2016, at 10:38 AM, Reynold Xin <[hidden email]> >> >>> >>> >>>>>>>> wrote: >> >>> >>> >>>>>>>> >> >>> >>> >>>>>>>> I called Cody last night and talked about some of the >> >>> >>> >>>>>>>> topics >> >>> >>> >>>>>>>> in >> >>> >>> >>>>>>>> his >> >>> >>> >>>>>>>> email. >> >>> >>> >>>>>>>> It became clear to me Cody genuinely cares about the >> >>> >>> >>>>>>>> project. >> >>> >>> >>>>>>>> >> >>> >>> >>>>>>>> Some of the frustrations come from the success of the >> >>> >>> >>>>>>>> project >> >>> >>> >>>>>>>> itself >> >>> >>> >>>>>>>> becoming very "hot", and it is difficult to get clarity >> >>> >>> >>>>>>>> from >> >>> >>> >>>>>>>> people >> >>> >>> >>>>>>>> who >> >>> >>> >>>>>>>> don't dedicate all their time to Spark. In fact, it is in >> >>> >>> >>>>>>>> some >> >>> >>> >>>>>>>> ways >> >>> >>> >>>>>>>> similar >> >>> >>> >>>>>>>> to scaling an engineering team in a successful startup: >> >>> >>> >>>>>>>> old >> >>> >>> >>>>>>>> processes that >> >>> >>> >>>>>>>> worked well might not work so well when it gets to a >> >>> >>> >>>>>>>> certain >> >>> >>> >>>>>>>> size, >> >>> >>> >>>>>>>> cultures >> >>> >>> >>>>>>>> can get diluted, building culture vs building process, >> >>> >>> >>>>>>>> etc. >> >>> >>> >>>>>>>> >> >>> >>> >>>>>>>> I also really like to have a more visible process for >> >>> >>> >>>>>>>> larger >> >>> >>> >>>>>>>> changes, >> >>> >>> >>>>>>>> especially major user facing API changes. Historically we >> >>> >>> >>>>>>>> upload >> >>> >>> >>>>>>>> design docs >> >>> >>> >>>>>>>> for major changes, but it is not always consistent and >> >>> >>> >>>>>>>> difficult >> >>> >>> >>>>>>>> to >> >>> >>> >>>>>>>> quality >> >>> >>> >>>>>>>> of the docs, due to the volunteering nature of the >> >>> >>> >>>>>>>> organization. >> >>> >>> >>>>>>>> >> >>> >>> >>>>>>>> Some of the more concrete ideas we discussed focus on >> >>> >>> >>>>>>>> building a >> >>> >>> >>>>>>>> culture >> >>> >>> >>>>>>>> to improve clarity: >> >>> >>> >>>>>>>> >> >>> >>> >>>>>>>> - Process: Large changes should have design docs posted >> >>> >>> >>>>>>>> on >> >>> >>> >>>>>>>> JIRA. >> >>> >>> >>>>>>>> One >> >>> >>> >>>>>>>> thing >> >>> >>> >>>>>>>> Cody and I didn't discuss but an idea that just came to >> >>> >>> >>>>>>>> me >> >>> >>> >>>>>>>> is we >> >>> >>> >>>>>>>> should >> >>> >>> >>>>>>>> create a design doc template for the project and ask >> >>> >>> >>>>>>>> everybody >> >>> >>> >>>>>>>> to >> >>> >>> >>>>>>>> follow. >> >>> >>> >>>>>>>> The design doc template should also explicitly list goals >> >>> >>> >>>>>>>> and >> >>> >>> >>>>>>>> non-goals, to >> >>> >>> >>>>>>>> make design doc more consistent. >> >>> >>> >>>>>>>> >> >>> >>> >>>>>>>> - Process: Email dev@ to solicit feedback. We have some >> >>> >>> >>>>>>>> this >> >>> >>> >>>>>>>> with >> >>> >>> >>>>>>>> some >> >>> >>> >>>>>>>> changes, but again very inconsistent. Just posting >> >>> >>> >>>>>>>> something >> >>> >>> >>>>>>>> on >> >>> >>> >>>>>>>> JIRA >> >>> >>> >>>>>>>> isn't >> >>> >>> >>>>>>>> sufficient, because there are simply too many JIRAs and >> >>> >>> >>>>>>>> the >> >>> >>> >>>>>>>> signal >> >>> >>> >>>>>>>> get lost >> >>> >>> >>>>>>>> in the noise. While this is generally impossible to >> >>> >>> >>>>>>>> enforce >> >>> >>> >>>>>>>> because >> >>> >>> >>>>>>>> we can't >> >>> >>> >>>>>>>> force all volunteers to conform to a process (or they >> >>> >>> >>>>>>>> might >> >>> >>> >>>>>>>> not >> >>> >>> >>>>>>>> even >> >>> >>> >>>>>>>> be >> >>> >>> >>>>>>>> aware of this), those who are more familiar with the >> >>> >>> >>>>>>>> project >> >>> >>> >>>>>>>> can >> >>> >>> >>>>>>>> help by >> >>> >>> >>>>>>>> emailing the dev@ when they see something that hasn't >> >>> >>> >>>>>>>> been. >> >>> >>> >>>>>>>> >> >>> >>> >>>>>>>> - Culture: The design doc author(s) should be open to >> >>> >>> >>>>>>>> feedback. >> >>> >>> >>>>>>>> A >> >>> >>> >>>>>>>> design >> >>> >>> >>>>>>>> doc should serve as the base for discussion and is by no >> >>> >>> >>>>>>>> means >> >>> >>> >>>>>>>> the >> >>> >>> >>>>>>>> final >> >>> >>> >>>>>>>> design. Of course, this does not mean the author has to >> >>> >>> >>>>>>>> accept >> >>> >>> >>>>>>>> every >> >>> >>> >>>>>>>> feedback. They should also be comfortable accepting / >> >>> >>> >>>>>>>> rejecting >> >>> >>> >>>>>>>> ideas on >> >>> >>> >>>>>>>> technical grounds. >> >>> >>> >>>>>>>> >> >>> >>> >>>>>>>> - Process / Culture: For major ongoing projects, it can >> >>> >>> >>>>>>>> be >> >>> >>> >>>>>>>> useful >> >>> >>> >>>>>>>> to >> >>> >>> >>>>>>>> have >> >>> >>> >>>>>>>> some monthly Google hangouts that are open to the world. >> >>> >>> >>>>>>>> I >> >>> >>> >>>>>>>> am >> >>> >>> >>>>>>>> actually not >> >>> >>> >>>>>>>> sure how well this will work, because of the volunteering >> >>> >>> >>>>>>>> nature >> >>> >>> >>>>>>>> and >> >>> >>> >>>>>>>> we need >> >>> >>> >>>>>>>> to adjust for timezones for people across the globe, but >> >>> >>> >>>>>>>> it >> >>> >>> >>>>>>>> seems >> >>> >>> >>>>>>>> worth >> >>> >>> >>>>>>>> trying. >> >>> >>> >>>>>>>> >> >>> >>> >>>>>>>> - Culture: Contributors (including committers) should be >> >>> >>> >>>>>>>> more >> >>> >>> >>>>>>>> direct >> >>> >>> >>>>>>>> in >> >>> >>> >>>>>>>> setting expectations, including whether they are working >> >>> >>> >>>>>>>> on >> >>> >>> >>>>>>>> a >> >>> >>> >>>>>>>> specific >> >>> >>> >>>>>>>> issue, whether they will be working on a specific issue, >> >>> >>> >>>>>>>> and >> >>> >>> >>>>>>>> whether >> >>> >>> >>>>>>>> an >> >>> >>> >>>>>>>> issue or pr or jira should be rejected. Most people I >> >>> >>> >>>>>>>> know >> >>> >>> >>>>>>>> in >> >>> >>> >>>>>>>> this >> >>> >>> >>>>>>>> community >> >>> >>> >>>>>>>> are nice and don't enjoy telling other people no, but it >> >>> >>> >>>>>>>> is >> >>> >>> >>>>>>>> often >> >>> >>> >>>>>>>> more >> >>> >>> >>>>>>>> annoying to a contributor to not know anything than >> >>> >>> >>>>>>>> getting >> >>> >>> >>>>>>>> a >> >>> >>> >>>>>>>> no. >> >>> >>> >>>>>>>> >> >>> >>> >>>>>>>> >> >>> >>> >>>>>>>> On Fri, Oct 7, 2016 at 10:03 AM, Matei Zaharia >> >>> >>> >>>>>>>> <[hidden email]> >> >>> >>> >>>>>>>> wrote: >> >>> >>> >>>>>>>>> >> >>> >>> >>>>>>>>> >> >>> >>> >>>>>>>>> Love the idea of a more visible "Spark Improvement >> >>> >>> >>>>>>>>> Proposal" >> >>> >>> >>>>>>>>> process that >> >>> >>> >>>>>>>>> solicits user input on new APIs. For what it's worth, I >> >>> >>> >>>>>>>>> don't >> >>> >>> >>>>>>>>> think >> >>> >>> >>>>>>>>> committers are trying to minimize their own work -- >> >>> >>> >>>>>>>>> every >> >>> >>> >>>>>>>>> committer >> >>> >>> >>>>>>>>> cares >> >>> >>> >>>>>>>>> about making the software useful for users. However, it >> >>> >>> >>>>>>>>> is >> >>> >>> >>>>>>>>> always >> >>> >>> >>>>>>>>> hard to >> >>> >>> >>>>>>>>> get user input and so it helps to have this kind of >> >>> >>> >>>>>>>>> process. >> >>> >>> >>>>>>>>> I've >> >>> >>> >>>>>>>>> certainly >> >>> >>> >>>>>>>>> looked at the *IPs a lot in other software I use just to >> >>> >>> >>>>>>>>> see >> >>> >>> >>>>>>>>> the >> >>> >>> >>>>>>>>> biggest >> >>> >>> >>>>>>>>> things on the roadmap. >> >>> >>> >>>>>>>>> >> >>> >>> >>>>>>>>> When you're talking about "changing interfaces", are you >> >>> >>> >>>>>>>>> talking >> >>> >>> >>>>>>>>> about >> >>> >>> >>>>>>>>> public or internal APIs? I do think many people hate >> >>> >>> >>>>>>>>> changing >> >>> >>> >>>>>>>>> public APIs >> >>> >>> >>>>>>>>> and I actually think that's for the best of the project. >> >>> >>> >>>>>>>>> That's >> >>> >>> >>>>>>>>> a >> >>> >>> >>>>>>>>> technical >> >>> >>> >>>>>>>>> debate, but basically, the worst thing when you're using >> >>> >>> >>>>>>>>> a >> >>> >>> >>>>>>>>> piece >> >>> >>> >>>>>>>>> of >> >>> >>> >>>>>>>>> software >> >>> >>> >>>>>>>>> is that the developers constantly ask you to rewrite >> >>> >>> >>>>>>>>> your >> >>> >>> >>>>>>>>> app >> >>> >>> >>>>>>>>> to >> >>> >>> >>>>>>>>> update to a >> >>> >>> >>>>>>>>> new version (and thus benefit from bug fixes, etc). Cue >> >>> >>> >>>>>>>>> anyone >> >>> >>> >>>>>>>>> who's used >> >>> >>> >>>>>>>>> Protobuf, or Guava. The "let's get everyone to change >> >>> >>> >>>>>>>>> their >> >>> >>> >>>>>>>>> code >> >>> >>> >>>>>>>>> this >> >>> >>> >>>>>>>>> release" model works well within a single large company, >> >>> >>> >>>>>>>>> but >> >>> >>> >>>>>>>>> doesn't work >> >>> >>> >>>>>>>>> well for a community, which is why nearly all *very* >> >>> >>> >>>>>>>>> widely >> >>> >>> >>>>>>>>> used >> >>> >>> >>>>>>>>> programming >> >>> >>> >>>>>>>>> interfaces (I'm talking things like Java standard >> >>> >>> >>>>>>>>> library, >> >>> >>> >>>>>>>>> Windows >> >>> >>> >>>>>>>>> API, etc) >> >>> >>> >>>>>>>>> almost *never* break backwards compatibility. All this >> >>> >>> >>>>>>>>> is >> >>> >>> >>>>>>>>> done >> >>> >>> >>>>>>>>> within reason >> >>> >>> >>>>>>>>> though, e.g. we do change things in major releases (2.x, >> >>> >>> >>>>>>>>> 3.x, >> >>> >>> >>>>>>>>> etc). >> >>> >>> >>>>>>>> >> >>> >>> >>>>>>>> >> >>> >>> >>>>>>>> >> >>> >>> >>>>>>>> >> >>> >>> >>>>>>> >> >>> >>> >>>>>> >> >>> >>> >>>>>> >> >>> >>> >>>>>> >> >>> >>> >>>>>> >> >>> >>> >>>>>> >> >>> >>> >>>>>> --------------------------------------------------------------------- >> >>> >>> >>>>>> To unsubscribe e-mail: [hidden email] >> >>> >>> >>>>>> >> >>> >>> >>>>> >> >>> >>> >>>>> >> >>> >>> >>>>> >> >>> >>> >>>>> -- >> >>> >>> >>>>> Stavros Kontopoulos >> >>> >>> >>>>> Senior Software Engineer >> >>> >>> >>>>> Lightbend, Inc. >> >>> >>> >>>>> p: +30 6977967274 >> >>> >>> >>>>> e: [hidden email] >> >>> >>> >>>>> >> >>> >>> >>>>> >> >>> >>> >>>> >> >>> >>> >>> >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> >>> >> >> >>> > >> >>> > >> >>> > >> >>> > --------------------------------------------------------------------- >> >>> > To unsubscribe e-mail: [hidden email] >> >>> > >> >>> > >> >>> > ________________________________ >> >>> > >> >>> > If you reply to this email, your message will be added to the >> >>> > discussion >> >>> > below: >> >>> > >> >>> > >> >>> > >> >>> > http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-Improvement-Proposals-tp19268p19359.html >> >>> > >> >>> > To start a new topic under Apache Spark Developers List, email >> >>> > [hidden >> >>> > email] >> >>> > To unsubscribe from Apache Spark Developers List, click here. >> >>> > NAML >> >>> > >> >>> > >> >>> > ________________________________ >> >>> > View this message in context: RE: Spark Improvement Proposals >> >>> > Sent from the Apache Spark Developers List mailing list archive at >> >>> > Nabble.com. >> >>> >> >>> --------------------------------------------------------------------- >> >>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >>> >> >> >> >> >> >> >> >> -- >> >> Ryan Blue >> >> Software Engineer >> >> Netflix >> > >> > >> > >> > >> > -- >> > Ryan Blue >> > Software Engineer >> > Netflix > > > > > -- > Ryan Blue > Software Engineer > Netflix --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
