Re: Design docs: consolidation and discoverability

2015-04-27 Thread Punyashloka Biswal
Github's wiki is just another Git repo. If we use a separate repo, it's probably easiest to use the wiki git repo rather than the "primary" git repo. Punya On Mon, Apr 27, 2015 at 1:50 PM Nicholas Chammas wrote: > Oh, a GitHub wiki (which is separate from having docs in a repo) is yet > another

Re: Design docs: consolidation and discoverability

2015-04-27 Thread Nicholas Chammas
Oh, a GitHub wiki (which is separate from having docs in a repo) is yet another approach we could take, though if we want to do that on the main Spark repo we'd need permission from Apache, which may be tough to get... On Mon, Apr 27, 2015 at 1:47 PM Punyashloka Biswal wrote: > Nick, I like your

Re: Design docs: consolidation and discoverability

2015-04-27 Thread Punyashloka Biswal
Nick, I like your idea of keeping it in a separate git repository. It seems to combine the advantages of the present Google Docs approach with the crisper history, discoverability, and text format simplicity of GitHub wikis. Punya On Mon, Apr 27, 2015 at 1:30 PM Nicholas Chammas wrote: > I like

Re: Design docs: consolidation and discoverability

2015-04-27 Thread Nicholas Chammas
I like the idea of having design docs be kept up to date and tracked in git. If the Apache repo isn't a good fit, perhaps we can have a separate repo just for design docs? Maybe something like github.com/spark-docs/spark-docs/ ? If there's other stuff we want to track but haven't, perhaps we can

Re: Design docs: consolidation and discoverability

2015-04-27 Thread Sandy Ryza
My only issue with Google Docs is that they're mutable, so it's difficult to follow a design's history through its revisions and link up JIRA comments with the relevant version. -Sandy On Mon, Apr 27, 2015 at 7:54 AM, Steve Loughran wrote: > > One thing to consider is that while docs as PDFs in

Re: Design docs: consolidation and discoverability

2015-04-27 Thread Steve Loughran
One thing to consider is that while docs as PDFs in JIRAs do document the original proposal, that's not the place to keep living specifications. That stuff needs to live in SCM, in a format which can be easily maintained, can generate readable documents, and, in an unrealistically ideal world,

Re: Design docs: consolidation and discoverability

2015-04-26 Thread Patrick Wendell
I actually don't totally see why we can't use Google Docs provided it is clearly discoverable from the JIRA. It was my understanding that many projects do this. Maybe not (?). If it's a matter of maintaining public record on ASF infrastructure, perhaps we can just automate that if an issue is clos

Re: Design docs: consolidation and discoverability

2015-04-24 Thread Sean Owen
I know I recently used Google Docs from a JIRA, so am guilty as charged. I don't think there are a lot of design docs in general, but the ones I've seen have simply pushed docs to a JIRA. (I did the same, mirroring PDFs of the Google Doc.) I don't think this is hard to follow. I think you can do w

Re: Design docs: consolidation and discoverability

2015-04-24 Thread Punyashloka Biswal
Okay, I can understand wanting to keep Git history clean, and avoid bottlenecking on committers. Is it reasonable to establish a convention of having a label, component or (best of all) an issue type for issues that are associated with design docs? For example, if we used the existing "Brainstormin

Re: Design docs: consolidation and discoverability

2015-04-24 Thread Patrick Wendell
Using our ASF git repository as a working area for design docs, it seems potentially concerning to me. It's difficult process wise because all commits need to go through committers and also, we'd pollute our git history a lot with random incremental design updates. The git history is used a lot by

Re: Design docs: consolidation and discoverability

2015-04-24 Thread Cody Koeninger
Why can't pull requests be used for design docs in Git if people who aren't committers want to contribute changes (as opposed to just comments)? On Fri, Apr 24, 2015 at 2:57 PM, Sean Owen wrote: > Only catch there is it requires commit access to the repo. We need a > way for people who aren't co

Re: Design docs: consolidation and discoverability

2015-04-24 Thread Sean Owen
Only catch there is it requires commit access to the repo. We need a way for people who aren't committers to write and collaborate (for point #1) On Fri, Apr 24, 2015 at 3:56 PM, Punyashloka Biswal wrote: > Sandy, doesn't keeping (in-progress) design docs in Git satisfy the history > requirement?

Re: Design docs: consolidation and discoverability

2015-04-24 Thread Punyashloka Biswal
Sandy, doesn't keeping (in-progress) design docs in Git satisfy the history requirement? Referring back to my Gradle example, it seems that https://github.com/gradle/gradle/commits/master/design-docs/build-comparison.md is a really good way to see why the design doc evolved the way it did. When kee

Re: Design docs: consolidation and discoverability

2015-04-24 Thread Sandy Ryza
I think there are maybe two separate things we're talking about? 1. Design discussions and in-progress design docs. My two cents are that JIRA is the best place for this. It allows tracking the progression of a design across multiple PRs and contributors. A piece of useful feedback that I've go

Re: Design docs: consolidation and discoverability

2015-04-24 Thread Punyashloka Biswal
The Gradle dev team keep their design documents *checked into* their Git repository -- see https://github.com/gradle/gradle/blob/master/design-docs/build-comparison.md for example. The advantages I see to their approach are: - design docs stay on ASF property (since Github is synced to the

Re: Design docs: consolidation and discoverability

2015-04-24 Thread Sean Owen
I think it's OK to have design discussions on github, as emails go to ASF lists. After all, loads of PR discussions happen there. It's easy for anyone to follow. I also would rather just discuss on Github, except for all that noise. It's not great to put discussions in something like Google Docs

Re: Design docs: consolidation and discoverability

2015-04-24 Thread Reynold Xin
I'd love to see more design discussions consolidated in a single place as well. That said, there are many practical challenges to overcome. Some of them are out of our control: 1. For large features, it is fairly common to open a PR for discussion, close the PR taking some feedback into account, a

Re: Design docs: consolidation and discoverability

2015-04-24 Thread Cody Koeninger
My 2 cents - I'd rather see design docs in github pull requests (using plain text / markdown). That doesn't require changing access or adding people, and github PRs already allow for conversation / email notifications. Conversation is already split between jira and github PRs. Having a third str

Re: Design docs: consolidation and discoverability

2015-04-24 Thread Sean Owen
That would require giving wiki access to everyone or manually adding people any time they make a doc. I don't see how this helps though. They're still docs on the internet and they're still linked from the central project JIRA, which is what you should follow. On Apr 24, 2015 8:14 AM, "Punyashlok