Hello, all. I'm using squashfs to hold my Gentoo repositories on all of my systems for some time. As you probably know, this allows me to save space while keeping portage fast. However, it makes updating the tree quite burdensome and time-consuming.
We're already hosting daily gx86 tarballs on our mirrors, and deltas made using diffball. Those can be used with Zac's emerge-delta-webrsync to get daily updates done with minimal network overhead. Sadly, it takes the whole process even more time consuming :). Therefore, I'd like to suggest an alternative solution that could help out Gentoo users that use squashfs for gx86 and would like to be able to get daily updates fast and easy. The idea is to host -- along with the tarballs -- daily squashfs images of gx86 in a chosen format. Additionally, the images would come with deltas made using xdelta3 or a similar tool. Those deltas -- with a slight download overhead -- would allow very fast updates of the squashfs. Now some numbers. I did some tests 'converting' late gx86 daily tarballs to squashfs. I've used squashfs 4.2 with LZO compression since it's quite good and very fast. 96M portage-20140108.sqfs 96M portage-20140109.sqfs 96M portage-20140110.sqfs 96M portage-20140111.sqfs 96M portage-20140112.sqfs 96M portage-20140113.sqfs 97M portage-20140114.sqfs 97M portage-20140115.sqfs For deltas, I've used xdelta3 with max compression (-9) and djw secondary compression (it gave ~0.1M smaller files than fgk and ~0.5M gain than with no secondary compression). 4,9M portage-20140108.sqfs-portage-20140109.sqfs.vcdiff.djw 6,3M portage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw 5,6M portage-20140110.sqfs-portage-20140111.sqfs.vcdiff.djw 8,9M portage-20140111.sqfs-portage-20140112.sqfs.vcdiff.djw 6,3M portage-20140112.sqfs-portage-20140113.sqfs.vcdiff.djw 7,8M portage-20140113.sqfs-portage-20140114.sqfs.vcdiff.djw 8,5M portage-20140114.sqfs-portage-20140115.sqfs.vcdiff.djw As you can see, the deltas are quite large compared to the actual changes. However, we could have expected that since we're diffing a compressed filesystem. What's important, however, is that applying it takes ~2.5 second on my 2 GHz Athlon64. So, even with the extra download time, the update is much faster than recreating the squashfs. And unlike some types of unionfs, it doesn't come with extra runtime slowdown. What do you think? -- Best regards, Michał Górny
signature.asc
Description: PGP signature
