Hi! [ I'll try to summarize the current discussion and status, what might be blockers, and a potential incremental way forward. ]
On Wed, 2024-09-25 at 10:48:50 +0200, Mark Brown wrote: > On Tue, Sep 24, 2024 at 05:45:49PM +0200, Guillem Jover wrote: > > On Tue, 2024-09-24 at 15:58:10 +0200, Mark Brown wrote: > > > Obviously it's far too late to do anything with the default for trixie, > > > we might want to evaluate doing something after the release but for now > > > it's too late. > > > Personally I don't think it's too late, there should be several months > > until the freeze, and I think if we wanted to switch we could perhaps > > do a staged transition and see how it goes and only do the final > > replacement if everything seems fine. > > We do OTOH package more software than most distros on more architectures > so we got a lot more exposure for testing coverage, and the revert would > involve switching the entire implementation which complicates things a > bit compared to a risky patch within a package. I'm not totally > opposed, and if everything goes smoothly we could definitely implement > it within the timeframe, but it feels like an impactful change to > introduce now not having considered it sooner. True, also two months have passed since (that's on me!). At this time, I'm now not sure whether it is feasible to consider such a switch, even if there was agreement to do it. As it is, I think there are too many unknowns! > > (perhaps) exceeding changed circumstances. But given your mail, I'm > > happy to work on this again and start with say uploading some initial > > stuff into experimental for example, after this thread settles a bit? > > > (I'll start by refreshing the packaging first though.) > > Sure. I did that, and the current WIP zlib-ng packaging provides now two builds, one with the new native zng_* API and another (tentatively) with the compat API/ABI one in libz-dev and libz1 binary packages. I've tentatively chosen those package names for the compat libraries to avoid having to go through NEW multiple times (with the assumption that we'd either go ahead with the switch or the packages could then simply be dropped). I think this should initially only be uploaded to experimental, to avoid getting packages built with either zlib or zlib-ng. But depending on the outcome of this discussion, I think other (probably better) options would be to perhaps name the compat packages something like libz-ng-compat*, or drop them completely? WIP package at <https://git.hadrons.org/cgit/wip/debian/pkgs/zlib-ng.git/>. > > > Does anyone have thoughts on this? > > > Personally, I think fully migrating from zlib to zlib-ng would sound > > great (even for trixie), but I guess we can take it slow if you do not > > feel confident or have concerns over this. > > > Also if you'd prefer to take over the zlib-ng ITP, as a continuation > > of zlib, that'd seem fine with me too. > > I'm fine with you carrying on with it (actually there is some slight > non-technical complication for me with doing it myself), or we could > also consider a packaging team. I think there was some other interest > in helping out but ICBW. If you're packaging it I'm also more confident > in letting you worry about how risky it is to transition and deal with > any fallout! :P Ok, so after the feedback on this thread, and Sebastian asking how we can proceed, here are the concerns brought on this thread, along my own and things I think we need to check or consider: * There were concerns (from Fay) about whether given same input the output changes per arch or hw setup, we'd need to check this; I'd expect this not to be the case for different arches, but it might be an issue with number of cores for example, but if either is true this would be a serious blocker. * I've had concerns both about providing the zlib compat API and the native zlib-ng API in sid, and then getting a mess of packages linking against (true) zlib and against (native) zlib-ng, or packages relying on specific behaviors from either and breaking when switching from (true) zlib to zlib-ng-compat or vice versa, for example. * There were concerns (from Fay) about the output stream changing due to a potential implementation switch and that affecting external reproducibility. Personally I think while I can see how this is annoying for the involved parties, it's part of the "you need the same tools to generate the same output" premise that we also assume in Debian. I guess keeping both implementations around indefinitely, I think, would make this less of an issue, with the potential drawbacks mentioned in the previous point. * There was a concern (from Konstantin) about at least one known upstream (Angie) misbehaving with zlib-ng generated streams. * There were concerns (from Mark) that even though projects like Fedora have done such switch, we have way more packages and architectures, so we might see more fallout that has not already been handled. * There was a concern (from Mike) about whether the performance gain at the cost of stream size makes sense, given that the compression level could be reduced instead to similar effect (?). I'm not sure how these compare, so it would be interesting to analyze this, because perhaps that's a less traumatic way to look at it (but that might require redefining compression level semantics globally in zlib, or patching users, with neither look very enticing options). My perception from when I tested it is that the speed up was significant enough and the size increase not so much, but… In any case switching to zlib-ng upstream would also imply other benefits, like (supposedly) a more responsive upstream with more frequent releases, the new native API, and an implementation other distributions are switching to. * Some upstreams have started to use the zlib-ng native API, so regardless of whether we plan a switch or not, I guess packaging zlib-ng (w/ or w/ the compat API) might still make sense. * To consider a switch we'd need to do a mass rebuild of the archive. Ideally running autopkgtests and similar to exercise the packages? After having written the above, and if Mark agrees, I think I'd opt for uploading zlib-ng to experimental, with the compat packages renamed to libz-ng-compat* or similar (even if that implies later on another trip through NEW if we want to perform a full switch), because that might make it easier to move them to sid as a way less disruptive change, even if we decide not to switch the default zlib implementation. OTOH and unfortunately I don't think I'm currently prepared to drive any of what I think might be required mass archive rebuilds and testing or the analysis mentioned above. Thanks, Guillem
