Gregory Szorc <mailto:g...@mozilla.com>
Wednesday, March 26, 2014 17:40
On 3/26/14, 4:53 PM, Taras Glek wrote:
*User Repos*
TLDR: I would like to make user repos read-only by April 30th. We should
archive them by May 31st.
Time spent operating user repositories could be spent reducing our
end-to-end continuous integration cycles. These do not seem like
mission-critical repos, seems like developers would be better off
hosting these on bitbucket or github. Using a 3rd-party host has obvious
benefits for collaboration & self-service that our existing system will
never meet.
How much time do we spend operating user repositories? I follow the
repos bugzilla components and most of the requests I see have little
if anything to do with user repositories. And I reckon that's because
user repositories are self-service. Are user repositories more than
just disk space and seldom CPU usage and page cache eviction?
Some significant portion Ben's time was spent on user repos during the
multi-week webhead migration(http://bke.ro/?p=380).
bugs like https://bugzilla.mozilla.org/show_bug.cgi?id=983085 cropped up.
The fact that repos keep growing means that we'll have to do this
migration again soon. We are at 260gb/300gb.
As long as our footprint is >40gb we can't migrate to fast, cheap &
cheerful AWS nodes. Have to do something complex or expensive
instead...which means more devop time. As long as our footprint keeps
growing we'll keep revisiting this problem.
B2G guys seem to prefer github already.
We are happy to help move specific hg repos to bitbucket.
Once you have migrated your repository, please comment in
https://bugzilla.mozilla.org/show_bug.cgi?id=988628so we can free some
disk space.
*Non-User Repos*
There are too many non-user repos. I'm not convinced we should host
ash, oak, other project branches internally. I think we should focus on
mission-critical repos only. There should be less than a dozen of those.
I would like to stop hosting non-mission-critical repositories by end of
Q2.
What about making non-user repos more self-service? (They currently
require bugs for everything AFAICT.)
This requires more devop time. Handling bugs wastes time on both sides.
Self-serve is the biggest advantage of breaking the habit and moving to
a hosted service.
This is a soft target. I don't have a concrete plan here. I'd like to
start experimenting with moving project branches elsewhere and see where
that takes us.
I would *really* like the ability to trigger automation on any repo,
regardless of its URL. Moving project branches elsewhere might make
this happen, so +1.
Right. I'm all for increased productivity through self-serve. Having a
weird ad-hoc system does not seem like a win.
*What my hg repo needs X/Y that 3rd-party services do not provide?*
If you have a good reason to use a feature not supported by
github/bitbucket, we should continue hosting your repo at Mozilla.
*Why Not Move Everything to Github/Bitbucket/etc?*
Mozilla prefers to keep repositories public by-default. This does not
fit Github's business model which is built around private repos.
Github's free service does not provide any availability guarantee.
There is also a problem of github not supporting hg.
I'm not completely sure why we can't move everything to bitbucket. Some
of it is to do with anecdotal evidence of robustness problems. Some of
it is lack of hooks (sans post-receive POSTs).Additionally, as with
Github there is no availability guarantee.
A lot of it has to do with lack of hooks. Without pre-push hooks on
Bitbucket or Github, there will be footgunning. The counter argument
is "just back out bad commits." But excessive backouts can be
problematic (see our Firefox tree management and ask Jesse about
bisecting impact).
No. Counterargument is: stop using hg like cvs. Have a staging repo and
automation to transfer changesets as part of a c-i process.
There is also the issue with size. Remember when GitHub disabled our
mirror without notice because it became too large and became a
performance problem? I can only speculate what Bitbucket will do when
1000 new 1.5+ GB clones of the Firefox repo show up. Have we asked them?
We are asking. Part of the reason mc got pulled is heavy traffic on that
repo. User & project repos should generate a lot less load. It's more of
a lets try this.
In the case of Mercurial, we'll want to someday deploy Facebook's
remotefilelog extension to enable "shallow" clones (drastically
reducing clone time in the process - a game changer for new
contributors who can't download 1+ GB of repo data). We may also want
to deploy a "bundle lookaside" extension that automatically uses a
bundle for initial clones. Obviously we can do these things for repos
on hg.mozilla.org. But what about the "user clones" on Bitbucket? We
may run into compatibility problems.
There are lots of potential solutions here. I would like to deploy hg
proxies(something like
https://bitbucket.org/Unity-Technologies/hgwebcachingproxy/src/e65fb34f6fa819c39c0c44277ff37e325221c407/hgwebcachingproxy.py?at=default)
for data-center/office locality. We can tweak the proxies as much as we
like. Those proxies can do this kind of magic. I'd much rather focus on
perf problems to do with version control than ALL version control problems.
Hosting arbitrary Moz-related hg repositories does not make strategic
sense. We should do the absolute minimum(eg http://bke.ro/?p=380)
required to keep Firefox shipping smoothly and focus our efforts on
making Firefox better.
Strategic, no. Necessary because we have no better alternative, quite
possibly.
I think you are agreeing.
If this boils down to maintaining the code behind
hg.mozilla.org/git.mozilla.org, I and others have offered to help.
I've volunteered to improve the self-service capabilities of user
repos, for example. But, the code is in some private IT repository and
it's difficult to get your hands on initially, to test, and deploy.
Whatever the outcome of this proposal is, I hope that roadblock can be
eliminated.
Yes. My preference here is to not compete with commodity tech. I don't
buy the argument that Mozilla's hg usecase is so unique that it can't
use popular DVCS services at all.
Taras
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
