A discussion of draft document
<http://rakudo.org/parrot/index.cgi?parallel_development_requirements>,
with some side threads edited out. See
<http://www.parrotcode.org/misc/parrotsketch-logs/irclog.parrotsketch-200703/irclog.parrotsketch.20070327>
for the full text.
...
<Coke> ok. This is basically a followup to discussions that have
occurred on #parrot and pp - what are our expectations about trunk in
terms of stability? how can we use branching and tools to support these
expectations?
<Coke> the outline of the various topics is on the wiki.
<Coke> (and I have a meeting in 8 minutes. =-)
<allison> ok, let's continue the conversation on the list, then
<bernhard> Have a bugfix branch for every release ?
<Coke> That's certainly a possibility, b.
<bernhard> Gives some freedom on trunk
<chromatic> Branches suck though.
<Coke> I would need you to specify the ways in which they suck. =-)
<chromatic> Merging between branches sucks.
<chromatic> Really sucks.
<Coke> chromatic: thank you.
<particle> parallel development sucks, just like multiple
inheritance.
<chromatic> Pulling code from experimental branches into a stable
branch is a lot of work.
<particle> but we need it.
<chromatic> It's a lot of not-fun work.
<Coke> I think that svk or some other "sits in front of svn" method of
keeping track of thigns would work. especially if those svk repos were
public.
<chromatic> It doesn't work very well in Perl 5, and I fear that it
won't work very well anywhere else.
<particle> we need an svn-only solution
<Coke> then it's easy to keep things in sync.
<allison> coke: I've had terrible trouble with bugs in svk messing up
my repository, I've gone back to svn because it's reliable.
<Coke> particle: I disagree. We need something that does svn in the
back end, but I don't think we have to have the LCD be the solution
everyone uses.
<Steve_p> It works OK if everyone agrees that one ( or a very few)
access the maintanence branch
<bernhard> How many branches are we talking about 1,2 or 10 ?
<chromatic> Steve_p, I think Nicholas might disagree that it
works okay.
<Coke> alrightee. General plan: update the wiki page with your notes.
Someone (probably me) will start a thread on p2p. Chromatic will herd
you for the remainder of this meeting. =-)
<Coke> any discussion that continues here, I or someone else will
capture and add to the wiki.
<pmichaud> thanks, Coke!
<particle> my proposed solution would support infinite branches,
infinite repositories. the main repository would have (at least) an
integration branch, and a stable branch
<Steve_p> Oh yes, and unfortunately, scm tools tend to mess up
branching. The human factor of branching comes int too.
<tewk> (chromatic's comments)++
<Coke> Roger. Hope everyone is having fun. (and let me know if
you're not!)
<allison> the most important question is "what problem are we solving?"
<allison> once we have that, it'll make it much easier to pick the
right solution
<particle> we're aiming to solve multiple problems
<particle> ~ provide a stable branch
<chromatic> It sounds like the problem is "Some changes are too large
to keep stable in small patches."
<bernhard> Lessen fear of breaking things
<particle> parallel work on multiple subsystems may result in diverging
code. this is the hard part to manage
<pmichaud> out of curiosity (because I don't know), how do other large
systems like the Linux kernel handle it?
<jisom_> then we need a better test suite! :)
<jisom_> pmichaud, luck
<chromatic> pmichaud, the Linux kernel has lots of public trees, from
which various maintainers pull patchsets when they want to include them.
<chromatic> It's highly distributed, so there's no single stable
trunk.
<pmichaud> chromatic: is that a function of maturity, then?
<chromatic> More a function of chaos.
<pmichaud> heh
<pmichaud> okay :-)
<pmichaud> answers my question!
<particle> pmichaud: linux kernel folk use 'git', a tool linus
wrote
<particle> it's not portable enough for us, yet
<chromatic> They use git now, yes, but they used to mail patches around
and there was no central SCM besides Linus' inbox.
<pmichaud> particle: I meant more at a meta-level, in terms of managing
multiple repositories and dealing with parallel work on multiple subsystems
<particle> yes, it's a distributed vcs solution. we're used to, and i
think aiming for, a centralized solution
<pmichaud> I didn't follow some of the earlier discussions, but what
are some examples of specific obstacles we're encountering with the
present system?
<particle> presently, trunk breaks.
<particle> far too often.
<pmichaud> isn't that the historical purpose of trunk/head,
though?
<allison> isn't that a matter of developer discipline?
<bernhard> And trunk gets fixed
<chromatic> My answers are no, yes, and hopefully.
<pmichaud> okay, I see the issue now
<particle> we have the ability to provide a stable codeline, so we
always have a release candidate.
<pmichaud> so, trunk is where stable things go, and developers should
be working in branches
<particle> why don't we offer it? it takes a little know-how, some
discipline, and programming.
<particle> not necessarially
<allison> pmichaud: or trunk is where work happens and branches are for
release candidates
<pmichaud> I meant as a general approach
<chromatic> Because branches have the tendency to grow and grow and
diverge further from trunk, which makes them much more difficult to merge.
<particle> i propose we have branches for shared/solo subsystem
coding
<particle> a branch for integration
<particle> and a branch for stable code.
<particle> trunk is just another branch.
<particle> perhaps we rename 'trunk' to 'integration', or define it
that way.
<allison> particle: but we have the same problem then of branches
diverging rapidly, unless release branches are short-lived
<particle> or, maybe we rename/identify it as 'stable'
<particle> allison: as long as the development branches are kept
up-to-date wrt the stable branch, they won't diverge
<pmichaud> well, if we treat the trunk branch as being the stable one,
then there's a built-in incentive to get things checked into the trunk
because without that new features don't make it into the release
<chromatic> particle, that's never been my experience.
<pmichaud> s/checked in/merged in/
<allison> particle: it depends on someone actively watching and merging
changes, which is pretty much a full-time job for a volunteer
<allison> (and not a particularly fun job)
<particle> it's a job we all do already, in a way
<chromatic> As soon as someone checks in a change that's not
appropriate for whatever point you want to merge to, it blocks that
branch forever.
<allison> aye, but we'd be doing it twice
<particle> crap, somebody changed the code i was working on, i have
conflicts
<particle> i need to merge them in my working copy
<particle> chromatic, you can peel off your changes since the last
baseline (synchronization point between branches), apply the changes
from 'stable', and reapply your diffs
<allison> ouch
<particle> yes, it's merging. it's painful. but you have to do it at
some point anyway
<tewk> ouch++
<chromatic> Oh I agree that it's possible, but in my experience the
longer a branch lives on, the more difficult this is and the more likely
it is that there's an imcompatible change you just can't work around.
<allison> the question is whether the sacrifice is worth the gain
<particle> i don't get it. you MUST do this before your code is
committed to trunk now.
<particle> the pain is no different
<particle> if you wish to synchronize more frequently, you may.
<allison> merge and integrate changes? yeah, but I do it on a daily
basis
<chromatic> The difference is that if you're always working off of
trunk, you have much less possibility to diverge strongly.
<pmichaud> also, when working off trunk you notice divergences
earlier
<particle> so synch daily, and it's still no different
<pmichaud> s/divergences/conflicts/
<allison> but if we're keeping the two branches in sync daily, then
what's the point of having branches?
<particle> okay, let's think of trunk as an integration branch, instead
of a stable one.
<tewk> I svnup whenever I see a check-in, 5-10 times a day.
<particle> commits are only merged with the stable branch after they've
been smoked
<chromatic> It takes the same kind of developer discipline to manage a
long-lived branch as it does to keep the trunk stable, and managing
branches adds administrative overhead and latency.
<allison> either they diverge, in which case it leads to painful merges
down the road, or they stay the same and they're irrelevant. is there a
3rd option?
<allison> chromatic: point
<chromatic> The smoke/integrate option is interesting. Is this
automatable?
<particle> of course it can be automated :)
<particle> but, we need better smoke tools
<allison> and what's the process when smokes fail?
<chromatic> When smokes fail, Paul Cochrane fixes a bunch of whitespace
issues.
<particle> pugs has a smokebot which notifys #pugs
<mugwump> Here's a little analagy: if a branch is a software
development transaction, trunk is like forced AUTOCOMMIT
<allison> particle: I meant human process
<tewk> guilty party get an email with the build report, and he checks
in a fix.
<particle> failures must be dealt with. a revision can't be considered
a release candidate until it smokes.
<allison> tewk: and if guilty party is on vacation?
<particle> we can have smoke results for each platform, and release
candidates for each platform
<particle> when the revisions converge across all platforms, we have a
true release candidate
<tewk> someone fixes it or reverts it.
<particle> smoke failures should be RT tix
<particle> no different than parrotbug
<allison> particle: I see the appeal, but I'm not yet convinced on the
cost/benefit side
<mugwump> so with branch based development, you snapshot the current
state, work on it, and eventually "commit" the transaction (merge)
<chromatic> Before we discuss branches, perhaps we need reliable
automated smokes.
<mugwump> with trunk based development, every change you make gets
"committed", there are no long-running transactions
<particle> chromatic: the smokes themselves aren't unreliable, but the
reports aren't great
<allison> that is, if we make it significantly harder to work on
parrot, it will reduce the number of new contributors we get, and we're
not at a good point to be turning people away because they don't
understand our arcane commit system
<particle> firstly, smoke reports are slow to render, secondly, they're
not easy to compare to others
<tewk> Lets get a rockin smoke system with IM and email notification to
guilty parties working first, then revisit the branching issue if it is
still relevant.
<chromatic> Do we have automated smokes though?
<particle> chromatic: yes, we do
<chromatic> Okay. Good. I didn't realize that we did.
<allison> I'm totally on board for improving our smoke system
<particle> ^conner_ and others have donated time and machines for that.
we can use more, of course.
<particle> Steve_p will be helping with that effort, i suspect :)
<particle> there has been one or more folks on #parrot and #perl6
talking about better smoke tools
<particle> *writing better ...
<tewk> So what we are really talking about here is a continuous
integration server on a bunch of platforms,
<chromatic> It sounds like we need to know, first off, how often trunk
breaks and then see how we can keep it stable.
<allison> tewk: yeah, which may have nothing to do with branching, when
we get down to it
<tewk> It has to be up and running all the time to be useful.
<allison> I'd be interested to talk to the mozilla group about their
build process
<tewk> I don't think trunk breaks across all platforms very often. A
check-in quite often doesn't work on say WIN32 or with FreeBSD's make
<particle> i have tests failing on win32 now.
<chromatic> It's much easier to fix things if we can relate them to one
specific checkin.
<particle> patrick committed a TODO just before his release ,which
caused an unexpected success on my platform
<particle> we get bug reports of "this test is broken" all the
time
<bernhard> Can we smoke for every checkin and see the diffs?
<tewk> bernhard: that is what a continuous build server does, plus it
emails out reports and auto creates rt tickets
<particle> we can do every checkin, or every <time it takes to
realclean/config/make/test>, whichever is longer
<bernhard> continuous build server ++
<mugwump> if you're going to do that, why not get out of it a stable
trunk
<mugwump> instead of just a bunch of rt tickets which is going to
create lots of work
<tewk> usually a continuous build server can do every checkin, it may
slip behind during heavy periods but can catch back up during lighter
workload periods.
<tewk> When the workload gets too high you add another build server.
<tewk> We don't have that many checkins per day.
<particle> we don't have many resources
<particle> and i'm not interested in testing every revision, when so
many might be coding standards
<allison> an hourly build is good enough
<allison> or even twice daily
<particle> if we can specify that all smokers should build a particular
revision...
<particle> then we can do that prior to release day
<allison> particle: not the latest revision?
<chromatic> The fewer changes between smokes, the better... but being
able to narrow a failure down to a particular range (10 commits perhaps)
is better than what we have now.
<tewk> every build lets you identify who broke it though.
<allison> tewk: aye, but it's a cost/benefit thing again
<allison> tewk: identifying it within a few hours of commits is good
enough, and doesn't require the purchase of additional hardware
<particle> now, we're only talking about smoking *one* branch,
here
<allison> particle: yes
<mugwump> you only need to smoke the tip
<chromatic> Maybe in the future, we could do a binary search to the
previous commit point to find the failure.
<leo_> hi folks - greetings @all - if it's not too late I've got some
comments WRT recent discussion
<allison> please, dive in
<bernhard> Andreas König knows about binary searches
<mugwump> git-bisect++
<leo_> 'smoke ok' ne 'make test'
<allison> leo: true
<leo_> the former has multiple platforms
<leo_> the latter is your home machine
<leo_> but - I usually did not ci w/o 'make test'
<leo_> that already helps a lot
<particle> we can probably learn some from PITA
<chromatic> If more people did that, we'd have fewer coding standards
failures....
<leo_> re binary searches
<allison> leo: I hope most people are doing that much
<leo_> coke write a tool which is on CPAN, which just does that
<allison> chromatic: if not, we could use a developer education day
<leo_> wrote*
<leo_> and I'd really prefer to have stable := trunk
<particle> App::SVNBinarySearch ?? something like that
<allison> leo: stable == trunk == mainline
<leo_> search by author might help
<leo_> allison: yep
<allison> leo: agreed
<leo_> re branches - if you have a branch: merge trunk as often as
you can
<pmichaud> I should note that for the case that particle cited (my
todo'd test), the problem is that I was on a platform (x86_64) where a
test failed but every other platform succeeded, and I didn't have a
convenient way to determine that. 'make smoke' doesn't mean anything in
that case
<allison> leo: but also agreed that we do need to do some work on
stability, it's just a matter if deciding the best use of efforts toward
stability
<pmichaud> thus we need a smoke server
<leo_> but that doesn't help against collisions of multiple branches of
course
<allison> pmichaud: hmm... the ability to smoke un checked in versions?
<mugwump> no, when a branch is merged the branch fails its first smoke
metric, which is mergability
<leo_> pmichaud: yep - being on x86-64 too I'm seeing this too
<mugwump> er, when a branch is merged, other branches that conflict
lose their mergability
<allison> mugwump: we're talking outside the branching model
<leo_> it just reduces failing tets - i.e. whitespace and such or
non-ansi, which is/ought to be tested anyway
<leo_> mugwump: if folks are working in different directions in the
same part of code ...
<leo_> that's a conflict, yes ;)
<allison> leo: then we have a bigger problem than just repository
maintenance :)
<pmichaud> allison: I don't know that "ability to smoke unchecked-in
versions" is what I was implying... I just know that some of the
problems being cited have to do with the fact that some developers don't
have access to all platforms
<leo_> indeed
<allison> pmichaud: yeah, that's where I was heading
<leo_> pmichaud: s/some/most of the/
<particle> how about s/unchecked-in/unstable/
<allison> pmichaud: access to devel servers
<mugwump> every change is an unstable change until it is smoked
<pmichaud> particle: but how do I know my particular commit is
'unstable' for a platform I'm not using?
<allison> particle: not exactly
<particle> a promotion model of dev -> integration -> stable
<pmichaud> mugwump: every change is unstable until it has been smoked
on every supported platform
<mugwump> right
<particle> it's unstable until it passes smoke, by definition
<pmichaud> thus my comment that 'make smoke' is insufficient
<pmichaud> it has to be 'make smoke' on lots of platforms
<leo_> as said: 'make smoke' := sum('make test') for @platforms -
IMHO
<allison> leo: yes
<particle> yes, modulo new/unsupported platforms
<particle> like, say zaurus
<leo_> of course
<pmichaud> okay, I'd prefer a different name for this, then, because
'make smoke' is also a command we often use to submit smoke reports
<leo_> 'make smoke' is just: I submit a smoke test for my $platform
<particle> devsmoke/intsmoke?
<pmichaud> if 'smoke' := sum('make test') for @platforms, then I'm okay
with that
<allison> platform smoking
<allison> porter smoking
<chromatic> If the smokes fail, does the porter bellow?
<allison> he just asks for better tobacco
<allison> smoke farm
* particle drinks porter, he doesn't smoke it
<allison> hit the smoke farm
<chromatic> Alright, with bad puns it sounds like we're winding
down.
<particle> finally!
<chromatic> To my mind, the discussion seems to have turned from
branching to a discussion of improving our smoke farm.
<allison> mmmm... yes
<particle> yes, we still need to support parallel development
better
<chromatic> Specifically, we desire to have regular smokes run on our
necessary platforms on a regular basis.
<allison> which doesn't change the fact that we need a branching policy
<leo_> branching is just part of the whole thing
<mugwump> I've got to go to work, but I'll just leave with this
comment: overcome your fear of branches. branches are the way forward.
trunk is the way of pain. That's all :)
<leo_> as well as moking
<leo_> s*
<allison> but, smoking is the most important problem
<leo_> and it does need some machines
<allison> and is useful whether or not we decide branching is worth the
pain
<leo_> or many folks that just submit tests
<leo_> BTW the report at http://smoke.parrotcode.org/smoke/ is still
suboptimal as dates / platforms and versions are just summarized, but
you can't see any trend
<chromatic> So noted.
<leo_> maybe looks like one of the microgrant todo projects to make
that more useful
......
