This message applies to anyone who clones a Firefox repository from hg.mozilla.org. *Failure to read this message may result in degraded Mercurial performance.* If you use git-cinnabar or use the mozilla-unified repository exclusively, you are likely shielded from degraded performance and can stop reading.
Some changes were made to the hosting of high-volume Firefox repositories on hg.mozilla.org in the past week. The full list of repositories impacted so far is available at https://hg.cdn.mozilla.net/. Client-side operations against repositories cloned from hg.mozilla.org may be slower until a one-time operation is performed. Once complete, operations against hg.mozilla.org should be *faster* than they were before. tldr ---- To ensure your Firefox repository clone runs at peak performance: 1) Upgrade to the latest Mercurial release (currently 4.1.2) - instructions at [1]. 2a) Run `hg debugupgraderepo --optimize redeltaparent --optimize redeltamultibase --run`. Learn more at [2]. This command will likely take a few hours to run. But your existing repo will function just as it did before, just faster. The command keeps a backup of your existing data and doesn't modify your repo until the last few milliseconds of execution. 2b) Create a fresh clone from hg.mozilla.org using Mercurial 4.1 then push/pull data from your existing clone to it. See [3] for specific instructions. Technical Changes ----------------- The main Firefox repositories on hg.mozilla.org were using a legacy storage format with known deficiencies. The repositories have been upgraded to use a modern storage format (generaldelta). This storage format results in smaller repositories (300-500 MB on-disk reduction is common) and faster operations against optimally encoded repositories. The main Firefox repositories on hg.mozilla.org have not only had their storage formats upgraded, they have also been optimized so data is stored within as efficiently as possible. This is effectively the Mercurial equivalent of `git repack -a -f`. While your local clone may be using generaldelta (run `hg debugupgraderepo` to see if it is already present), the deltas may not be optimal and this can result in repo bloat and slow operations. A re-clone or one-time data upgrade with certain optimizations (using the `hg debugupgraderepo` described in the tldr section) will align your local clone with hg.mozilla.org and result in optimal performance. Performance Implications ------------------------ If 2 Mercurial peers (say your local clone and hg.mozilla.org) are using different repository storage formats, one side has to do work to normalize exchanged data so it can be applied. For 95% of repositories, this data normalization is fast and nobody likely notices. But because the Firefox repository is so large, it can add noticeable to extreme overhead to operations. With the Firefox repositories on hg.mozilla.org upgraded to generaldelta, the following can occur: * A Mercurial 3.7+ client with a generaldelta repo will essentially stream data from the server to local disk with minimal processing. (Read: this is ideal and fastest.) * A Mercurial 3.5+ client with a non-generaldelta repo will have to heavily process incoming data so it is compatible with local storage. This will slow down operations like `hg pull` in proportion to the amount of incoming data. The slowdown will be less pronounced on Mercurial 4.1 due to performance improvements in Mercurial. * A Mercurial 3.4 or older client isn't capable of receiving the server's native storage format over the wire protocol so the *server* has to normalize things to a format the client can understand. This slows down `hg clone` and `hg pull` operations drastically (45+ minute clone times regardless of network speed) and can create load problems on the server. (This is why we waited a while before performing this server-side change and is why we'll have to lock out legacy clients if this load becomes a problem.) In addition, a Mercurial 4.1+ client will use zstandard instead of zlib for compression, resulting in faster compression/decompression and fewer bytes transferred. This translates to faster operations against hg.mozilla.org. Why We Made This Change ----------------------- Many of you may be asking: "since this change slows clients down, why make it?" Good question. The viable choices were between doing nothing and not allowing peak performance or making a 1-time change that enabled faster operations while temporarily inconveniencing (to various extents) existing clones and their users (namely Firefox developers). In other words, it was about letting the past and legacy software continue to hinder us or to break free and move to the future. With Firefox and Mozilla, we have to play the long game. And, we need to enable developers to move fast. This includes things like Firefox's automation. This change will enable VCS interactions in automation to complete faster which will reduce end-to-end times - something many complain about. We could have made this change over a year ago. But we waited for better client-side adoption so the impact of this transition would be mitigated. We rolled out repositories like mozilla-unified (that have been encoded in the optimal format since day 0) and encouraged their use. There would never be an optimal time to make this transition: someone would always lose. In my mind at least, we had postponed allowing clients to achieve optimal performance for long enough (over a year) and it was time to make the transition. So we did. Additional Work and Related Bugs -------------------------------- Bug 1351859 is the bug tracking upgrading all repos to the latest storage format. If you notice any fallout from this work, please track against it. There may be issues with the pre-generated zstd bundles and 32-bit Python/Mercurial processes [on Windows] running out of memory. That's bug 1344790. If you run into a memory error, run `hg --config ui.clonebundleprefers=COMPRESSION=gzip clone ...` as a workaround. [1] https://mozilla-version-control-tools.readthedocs.io/ en/latest/hgmozilla/installing.html <https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/common.html#upgrading-repository-storage> [2] https://mozilla-version-control-tools.readthedocs.io/en/ latest/hgmozilla/common.html#upgrading-storage-via-debugupgraderepo [3] https://mozilla-version-control-tools.readthedocs.io/en/ latest/hgmozilla/common.html#upgrading-repository-storage <https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/common.html#upgrading-repository-storage> _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
