I received no comment to technical review questions, no acknowledgement of the regressions laid out and also no pointer to where the plan for these changes is.
I can only assume that nobody saw my questions or doesn't want to answer for non-technical reasons. I can live with that, but I also feel that for an issue that involves rewriting the central trust store of the years past (ca_root_nss) this behaviour should raise a red flag. Here is my final technical assessment of the current situation, whether someone cares or not, including base. I've also added all current and past authors to this reply just for once to make sure nobody is actually "missing" this who could benefit from it for the sake of everyone else using FreeBSD having to deal with the elevation of certctl and the FreeBSD managed trust store. 1 -- The untrusted/blacklisted situation There was more than enough time to get the following commit[1] into stable/13, but it does not appear there. It would have been practical to have it in 13.2 at the very latest because in January ports consumers are going to be forced to deal with certctl through ca_root_nss[2] (and a large number of other sudden bundle dependency removals) through quarterly packages if not sooner if it's going to be put there directly (I have no idea if it will but it doesn't matter because it's going to happen either way). One of the bigger concerns is that why is this work carried out after 13.2 is out and established. People have migrated and set it up and the next quarterly at the latest will change their expectations and it's not documented anywhere I could find. It's not even being actively discussed by the authors as you can see from the radio silence mentioned earlier. [1] https://cgit.freebsd.org/src/commit/usr.sbin/certctl?id=64e6e1e46363de5d4843cf0fc79406060ec44c03 [2] https://cgit.freebsd.org/ports/commit/security/ca_root_nss?id=483e74f44b82f20bddd5608beef74b2a5ab38a88 2 --No annotation of (desired) behavioural changes This is a point from the last mail and the reason why a plan would be good, preferably written as a UPDATING entry or a port revision bump and a commit message or levelling expectations through the certctl man page that additional trust stores will be merged into the base provider location and what to do if this is undesired. This also goes for inconsistency changes here[3]. Yes, in the best case these files are the same but now we have one direct link and two overrides. Is the user supposed to override each file with something else not knowing what the actual port is going to use[4]? It's unlikely. And UFS corruptions WILL leave the effective file empty on a crash (0 bytes) which pkg/ca_root_nss install WILL NOT recover from. Debugging this is no fun, and I've seen it enough times to say this is a very bad thing to change, because in most cases people do not edit these files and a corruption can mean their application stops working because the bundle being used is empty. And if there are technical requirements to have sample files a compromise would be possible, but that requires responding to technical review. Wouldn't it have been better to remove the ETCSYMLINK files one by one and give enough space in between these removals so that the ports committers can weed one out per quarter? And I don't mean now. This could be years away, but it would be a plan. First /usr/local/openssl/cert.pem moving to /etc/ssl/cert.pem Then /usr/local/etc/ssl/cert.pem moving to /etc/ssl/cert.pem Lastly /etc/ssl/cert.pem [3] https://cgit.freebsd.org/ports/commit/security/ca_root_nss?id=574c939eccd322f546365bff8a68c7a5b7c3dc92 [4] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=274322 3 -- Merging trust stores and the global untrusted dilemma I was lucky enough to run into an unexpected situation with a CA that I was installing locally. Appending it as the first certificate ignores the whole bundle on 13.2 and I see it was fixed here[5] and it was at least marked for stable merge. Will it also be in 13.2 or is 13.2 as a production release not a target like point 1) not also? The wider remark here: FreeBSD reserves its right to provide further untrusted certificates through base updates which can heavily impact operation of installations in a quite unexpected way by untrusting certificates bundled with other trust stores or the user's choices. What is the benefit of this as opposed to revoking (as in removing) certificates on base updates and leaving the untrusted list for the user to deal with on their own? I do know why that one untrusted certificate was actually problematic, but only since I worked with the signer closely over a number of years. That FreeBSD tries to manage this different than ca_root_nss has always done is strange... Flat bundle, update and done? Why make it complicated, but perhaps I'm missing a technical reason that is not relevant to ca_root_nss. But bottom line untrusted is good, but why is it partially managed by FreBSD and also globally effective? It appears this was written without ca_root_nss in mind, which could technically become untrusted. And I do not know how updates are handled but could removed untrusted certificates come back? It wouldn't be very expected to the user dealing with the problem either. But it might not be the case. [5] https://cgit.freebsd.org/src/commit/usr.sbin/certctl?id=a401c8cb26b22688087ad7c5ee527718459df15a 4 -- On certctl rehash being slow I'm not saying it's too slow but the result of chaining everything into the rehash process from a shell script makes sure that people will be affected by this later when they try to sidestep[6] it. There is probably room for improvement, but there are no obvious things to do here that would make this faster on short notice, especially if additional certs are going to be dissected for valid reasons checking against untrusted references. [6] https://github.com/pfsense/pfsense/commit/72c441e9e0c0f 5 -- Missing bundle emulation for certctl The simple cure for ETCSYMLINK removal would be a bundle emulation mode in certctl which appears to be missing for no apparent reason. It would have made the whole situation so much clearer and smoother (especially when it wold have been carried out in 2019-2020). Keep the option, modify the state of the ETCSYMLINK=off to use the bundle emulation (certctl rehash is hooked up anyway for better or worse). After rehash append a bundle stage or a call a separate stage which could be used from ca_root_nss (or not). Providing the bundle files for base as well would be less error prone. I expect some bundle use to linger for a couple of years: old tutorials are used, old code is copied, people make mistakes, hardcoding etc. In OPNsense we've been disabling ETCSYMLINK for a long time and added a light emulation layer for the files provided with the merged user certificates in it. The process is very quick and sturdy, because you neither need @sample overrides nor is this prone to UFS corruption mentioned in point 2) because one ends up rewriting and copying these files fresh on each invoke fixing any corruption (on a reinstall of ca_root_nss via pkg too). 6 -- Missing global lock for certctl Running wedged tasks like[2] is going to force collision when certctl rehash is going to be backgrounded[6] by users because of point 4). It can be said that it's undesired but we all know it's going to happen. The user can't access "certctl rehash" inside ca_root_nss. It may end up building a partial trust store and for some people "just run certctl rehash" again is not a great support strategy (automation, unreachable destination, administration errors, script bugs etc.). And rehash cron jobs are also problematic in the same way and I would not exclude this as it can be a valid use case. 7 -- Missing pre- and post-processing hooks for rehash Async[6] is also bad because if users want to manually override the structure and certctl rehash is in ca_root_nss (possibly triggering twice for removal and re-adding on upgrade?). In these cases the best solution for the user would be to be able to drop files into the right location at the right time -- when certctl actually is ready to perform a rehash -- and clean up the current untrusted situation if it is kept like it is. It's not something desperately needed, but it shows how simple certctl was built for the base system where this works fine, but not if there are more trust store locations incoming from the ports or user side and multiple points try toreload it to get their updates in. 8 -- Bundle file handling I've wondered that OpenSSL can deal with full bundles in /etc/ssl/certs just fine like done with ca_root_nss but the blacklisting appears to defeat that purpose completely if the FIRST CA happens to match. Good thing is this was already found and fixed[7]. What's the plan for 13.2 inclusion? [7] https://cgit.freebsd.org/src/commit/usr.sbin/certctl?id=a401c8cb26b22688087ad7c5ee527718459df15a 9 -- The problem of thinking the ports situation is as easy as the base situation The tooling appeared ready for base side consumption in 2020, but solving the challenges in ports WRT ca_root_nss was not prepared for. The code trickled into all the releases in a first version which was a very good approach to keep it under the radar. I've seen nice results bootstrapping projects from GitHub with just base tools now on FreeBSD 13.2 and that certainly is a great achievement I applaud. Keeping the former points mentioned here in mind now both certcl is going to be improved on the fly while the ports tree is changed mid-release at least from the FreeBSD 13.2 perspective. I don't see how this will not create more ports consumer problems doing all of it on the fly at the same time and the ports tree moving faster than the base tree from a user perspective, but it is what it is. The behaviour for 14.0 will also be a bit different perhaps, and for CURRENT users get the latest and greatest, but I doubt a lot of users are aiming for production systems on CURRENT. Again, most of this is a view from the FreeBSD 13.2 release side looking at commits and reverts in the ports tree, core feature changes and all this will be ready once quarterly hits and doesn't depend on any work for certctl that is not in FreeBSD 13.2? 10 -- Breaking people's trust The sad truth for me personally is that there is a lot of knowledge out there (in the bug tracker, the ports code, committers and contributors) and a number of easily curable mishaps have already been made. Seeing this drag out in real time is not very pleasant, especially since FreeBSD users are going to be at risk needlessly if only there was more communication and most of this could be avoidable and smoother. The aim shouldn't be bug free implementations, but breaking the trust on the Internet for already voiced concerns is not really worth it. That's especially for the current authors of the changes to decide, maybe already decided. Fair enough. Feel free to correct me on technical aspects I'm getting wrong or point me to documentation that explains these points raised here. Favourably where users will find them quickly if they run into any such issues laid out here when pkg- upgrade will change their machine's behaviours. I've used the src and ports tree to the best of my knowledge, read the manual pages and additional commit notes and annotations. This is by no means a full audit and I could have easily missed something. The only favour I have is please don't ask me to write documentation, fixes or features for points mentioned here myself if I want to see them since I neither have control of nor current authors will share their plans and allow debate. I'm not a committer and I don't plan to be. I'm just a contributor with field experience on the matter and I've done my best to draw a current picture of the situation as it unfolds. I'll privately respond to technical inquiries if any come up but this is my last mailing list post on the subject. The trust bundle change and certctl are certainly fun endeavours and it was rewarding to look at the bigger picture. I thank everyone for being involved and working on and improving it over the years. And thanks for reading this far. :) Cheers, Franco > On 9. Oct 2023, at 10:39 AM, Franco Fichtner <fra...@lastsummer.de> wrote: > > Hi, > > Since discussing this appears to be difficult I'm starting a > new and brief thread about the recent changes based on > the verifiable facts: > > https://cgit.freebsd.org/ports/commit/security/ca_root_nss?id=574c939eccd322f546365bff8a68c7a5b7c3dc92 > > This commit changes the behaviour of the ETCSYMLINK > option without a revision bump or explanation. It forces > inconsistencies between the three files provided (two are > samples now, the other one is a direct link). Trying to contact > the author was unsuccessful. A few things were at least > discussed around the commit: > > https://lists.freebsd.org/archives/freebsd-ports/2023-September/004451.html > > Then we had another commit: > > https://cgit.freebsd.org/ports/commit/security/ca_root_nss?id=483e74f44b82f20bddd5608beef74b2a5ab38a88 > > This was approved by the author of the first commit not > responding for comments, but introduces other regressions > like merging the trust stores unconditionally and removing > other provided files in a default option. It was up for review > at least. It also modifies the port as indicated in the previous > mail thread that had no response: > > https://lists.freebsd.org/archives/freebsd-ports/2023-September/004459.html > > I've unprofessionally voiced my concerns about lack of discussion > out of frustration about lack of response, planning and care here: > > https://lists.freebsd.org/archives/freebsd-ports/2023-October/004612.html > > I deserve my fair share of criticism, but it doesn't change the fact > that both commits are of poor quality and they actually were actively > being discussed and concerns ignored. I will not yield on this fact. > > I see now that the latter one has been rolled back again: > > https://cgit.freebsd.org/ports/commit/security/ca_root_nss?id=52e0c40367d3ebd09ab7169e025c37fbf70b8dee > > Thanks for that. > > Now my main concern is what's the plan? Do we want to live with > the inconsistency untroduced in 574c939ecc that was actually > reported and fixed in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262755 > and introduced in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=228550 ? > > Why are we discarding prior work with documented use case and user > reports under the guise of cleanups? Why was there no planning and > proper review of the changes carried out? Why were the two committers > involved either not responding or criticising others harshly for > bringing it up while approving each others work anyway? > > As a side note I'd appreciate if asking for apologies is not a recurring > trend when ignoring technical discussion and concerns from non- > committers. This goes for on-list and off-list messages. It's highly > inappropriate. > > > Thanks for reading this far, > Franco
