On Tue, Feb 16, 2016 at 11:44:52AM +1000, Dave Airlie wrote: > On 15 February 2016 at 20:06, Daniel Vetter <daniel.vetter at ffwll.ch> wrote: > > Hi all, > > > > I've already chatted with some of you in private, here's the entire idea > > with a > > bit more thought. My motiviation for group maintainership of drm-misc was > > that I > > got a bit a guilty feeling the last few vacations/conferences when folks > > pinged > > about reviewed/tested pretty patches not landing. But also just increasing > > the > > bus factor and sharing the load better is good. And finally a shared misc > > drm > > tree would allow fringe drivers to get faster into Dave's drm-next by > > piggy-packing on top of the one pull request train. And it would also > > reduce a > > bit tree proliferation (at one point we had > > drm-misc/-bridge/-panel/-trivial and > > may more even). And at least everyone I chatted with seems to like the idea > > in > > principle. > > > > But what's still open is how to do it exactly. One big change with group > > maintainership is that you can't rebase a tree anymore. And right now I > > need to > > rebase drm-misc fairly often to throw out bad apples again. I think solving > > that > > is the important bit to make a shared drm-misc work. A few ideas: > > This is kinda what I don't like. I don't want a tree that can't rebase > out bad stuff > coming to me that often. If we are being too over eager in merging > stuff the answer > is to merge less not try and merge more. If we have a tree where things > are thrown until they stick I'm not sure the history will ever be nice > to pull from. > > I'd rather consider a staging tree where everything from patchwork can > get thrown > into, CIed, but that we cherry-pick fixes things to go back to me from the > what > works category.
I fully agree on your concern that drm-misc could become a mess - that's why I think everything bigger than e.g. 2 patches or touching drivers needs to soak first in a topic branch. We could still get it into drm-misc with cherry-picks to avoid too many merges. The other part where I often had to squash in fixups is build fail on arm. But I fixed myself with some decent cross-compile toolchain, so I'm positive that won't happen any more. I think a good idea would be that I test-drive this process a bit without adding more maintainers, i.e. - no more rebases for drm-misc - topic branches for everything big - dutifully testing arm before pushing I'll do that over the next few weeks, then we can see how bad it would look and whether we need more to avoid bad history. > I'm finding with i915 for example there is a massive latency in the pipeline > now > waiting for fixes, and the pipeline to the end of drm-intel-next is > very long and > hard to figure out what fixes should be pulled back, and how much of the > driver has been rewritten between the -next and the -fixes pulls. > > > - I think CI is super-important. We're starting to finally roll that out for > > real for i915, and it's catching an awful lot of stuff already. Not yet > > ready > > for prime-time on public mailing lists, and for misc we probably can't > > test > > every patch before they land like we do for i915. But CI should have veto > > power before a pull request goes to Dave imo. > > > > For non-i915 Daniel Stone and others are working on ARM CI using the > > generic > > igt testcases for kms. And I'm open to merging driver-specific tests into > > igt > > too, if it makes sense, and e.g. Eric has already pushed some vc4 tests. > > > > - Stuff needs to at least compile cleanly before pushing. I've been really > > bad > > at that wrt arm drivers with my own drm-misc, but turns out it's fairly > > easy > > to get this right: > > http://blog.ffwll.ch/2016/02/arm-kernel-cross-compiling.html > > > > - Bad apples need to be kicked out with reverts, not rebases. I think that's > > fine for simple patches, and hence those can go directly to drm-misc. But > > a > > bunch of subsystem-wide refactorings go in through mis trees, and for > > those > > constantly mass-reverting until it's all solid is silly. And ime you need > > some > > soak time in a shared tree to iron out bugs with those kind of > > endeavours. We > > can address that with ad-hoc&short-lived topic branches which are then > > again > > owned by a single maintainer, but automatically pulled into an integration > > tree. After some soaking time to give CI systems time to crunch through > > those > > topic branches can then be merged into the main drm-misc and removed. > > I think integration trees are probably the way forward, but I > understand they come > with a massive overhead for constant merging, I like the idea of topic > branches until > it becomes my turn to spaghetti merge 3-4 features at once. At that > point I still > fallback to the slow things down, pick one feature merge it, back the > others off, > and I think this is something we should do more off, the pile everything in > and > hope that magical CI will stabilise things doesn't seem to be a responsible > process to me. My idea is that you won't see the integration tree mess ever, but that tested stuff would land in drm-misc first to resolve conflicts. But really there never should be, since when a series takes longer than 1-2 weeks to get ready it's just not ready for merging. And the entire topic branch should be kicked out. Also integration tree mangling is why I suggested we start out using dim scripts - we're integrating 5-10 branches for drm-intel-nightly since years, since a few months with multiple maintainers pushing conflict resolutions, and it all works really well. Also agreed that CI won't magically make stuff stabilise - if CI complaints the rule needs to be to revert or drop the topic branch, not try to wrestle it until it's less unhappy. That indeed just doesn't work, and we're learning all that with drm-intel right now (before we had no CI but ran on hope, which works even less). > so I'd like CI to be happening but I'd like it to be happening before the git > history is baked into stone. Yeah that's what we do for drm-intel. At least right now we don't even have anyone doing arm CI, much less CI before merging. Daniel Stone is working on it though, so I'm hopeful. But yeah for my own stuff or patches from intel folks we already throw them all at CI before they land anywhere. > > - Also we can't roll forward to latest drm-next easily with rebases any > > more. So > > that needs to happen either right after the pull request lands (when no > > patch > > has been merged meanwhile). Or with backmerges, which then need a short > > commit > > message as to way the backmerge is needed ("Backmerge because we need > > $feature" is what I usually type), plus sob line from the committer. And > > of > > course the backmerged treed must be stable/non-rebasing and preferrably > > the > > backmerge should be a release/pull-request tag. > > > > - For tooling I suggest we just go with drm-intel maintainer-tools for a > > start. > > Picking dim has a few reasons: > > * Well tested, documented and fairly complete (includes e.g. > > bash-completion). > > * Integrated support for topic branches, including support for git > > worktree. > > * Well-exercised and robust integration tree support. > > Short-term we could add some convenience functions (e.g. for > > creating/merging > > drm-misc topic branches). Long-term we might or might not want to have > > this > > separated from drm-intel - that should be possible but a bit of work. > > Also, > > this way drm-misc would stay at it's current location while we figure > > things > > out. Longer term we might also want to look into adding other big drivers > > into an over drm integration. > > > > Of course group maintainership needs an initial group. My experience from > > drm-intel is that a bigger group of maintainer has benefits: It's clear that > > part-time maintaining is ok too, with maintainers focusing on their area of > > interest/expertise and only helping out in other places when there's a gap > > (due > > to e.g. vacations). Anyway, here's my thoughts for the starting group: > > Archit, Jani, Thierry & me as existing maintainers of drm-misc/bridge/panel, > > Alex&Rob as maintainers of big drivers and engaged in core drm stuff, > > Daniel Stone so that he has no more excuses to stall on arm drm ci. > > > > I think if this goes well we can extend it to more driver maintainers, e.g. > > Patrick would like to just push gma500 patches to some tree and not fiddle > > with > > pull requests all the time himself. > > Is pull request sending a big issue? I don't really want a misc tree > as a way to avoid sending distinct pull requests for drivers. If we do > this I want > it to be drm core patches and subsequent fallout only and if we want > an unmaintained > drivers patches tree to do that separately. Eric and Patrick mentioned in private that pull reqs for drivers is somewhat a nuisance for them. And iirc Thierry also said he's drowning, so often only gets around to stuff very late. But really driver trees for small drivers is an entire different topic that I'd like to maybe keep in mind. But certainly not for initial. > I don't really like the misc tree replacing panel/bridge tree either, > those were nicely > compartmentalised in my head as trees I pulled but didn't really care > about as Thierry > did a good job and I don't have ARM hw that I personally care about. Hm, I mentioned that since I've heard complaints from ARM folks about tree proliferation, so figured we could try something to get things together a bit more. But since I care about x86 first I don't mind at all if we keep drm-misc restricted to what I pulled in thus far if you don't want this. > It's important to me that stuff still comes in some sort of > compartments, as it decides > for each pull request how much time/effort I put into testing it on my > end, I'm pretty much > at the mercy of ARM hw maintainers, but with x86 I still like to smoke > test it on some > hardware as much as possible, also with DRM core stuff. If we keep bridge/panel out I don't expect stuff will change at all in that regard for you. drm-misc will still be misc random patches, plus the oddball treewide refactor. And I'll still try to pace the bigger refactors reasonable. That needs some coordination among maintainers, but I think the bigger upside here is to get the oddball ones in with less overhead - big patch series are much harder to forget about ;-) > So I'm still not sure where I stand on this, personally I don't care > if you take holidays > and stuff stalls, delays aren't the big enemy here, merging crap is > much worse than > delaying. One part of my motivation is that this went much, much better than I ever hoped for in drm-intel. There's teething issues for sure, but it helped massively in increasing the bus factor, and in reducing people pinging me directly for every little tiny patch. So much more time for me to look at patches that are actually important for me to look at wrt "should this go in or not". Anyway no rush at all here for drm-misc, and I think next step is for me to test-run the "no rebasing" to figure that part out. But still would like comments/ideas from others, too. Cheers, Daniel -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
