Unclear Memory Leakage since OpenBSD 7.3 upgrade (nginx and MariaDB; Not consistent)
Moin, I have been dealing with memory-leaks on a host running an nginx reverse proxy for some time. The host had been running 7.2 with nginx 1.23.1 (self compiled as i need some features not in the package) until May, which was fine. After upgrading to 7.3 and nginx-1.24.0, i started to see heavy memory leakage over time. I initially attributed this to nginx, and solved the issue by ignoring it/throwing a bit more memory at the box for some time. However, I started debugging it now and could trace it to some commit between nginx 1.23.3 and 1.23.4; Currently, I am going through all commits to see with which version it first appears. However, clicking around this morning, i noticed that my primary NS shows a similar memory leakage for mysql/mariadb (runs powerdns with a mysql backend, both from packages) since the upgrade to 7.3 in May as well. One further host seems to show a significantly higher use of inactive memory since 7.3. I found one more host with mariadb that now shows a higher utilization of inactive memory (gitea+maria); However, other maria+X instances on 7.3 run fine. On the PowerDNS+Mysql host I also see gradually increasing CPU load along with the memory leakage. Restarting mysql and/or nginx resolves the issue for some time. I also saw the issue on a host running maria+nextcloud, where i fixed it by switching to postgres. Maria is still running there, but not receiving any requests, and the memory leak is gone. Images: https://rincewind.home.aperture-labs.org/~tfiebig/pdns-maria-cpu-year.png https://rincewind.home.aperture-labs.org/~tfiebig/pdns-maria-memory-year.png https://rincewind.home.aperture-labs.org/~tfiebig/nginx-rev-proxy-memory-year.png https://rincewind.home.aperture-labs.org/~tfiebig/gitea-maria-memory-year.png https://rincewind.home.aperture-labs.org/~tfiebig/nextcloud-maria-switched-to-postgres-memory-year.png All hosts are VMs on Linux/KVM hosts (proxmox), with a generic qemu-kvm processor as CPU type, running 7.3. Hence, I am no longer 100% convinced that this is an nginx problem. To help circling this in a bit closer (and maybe get it reproducible in a first step): Did anyone else experience memory leakage on openbsd with mariadb or a self-build >=nginx-1.23.4 (or other applications) since the upgrade to 7.3? With best regards, Tobias
Re: Unclear Memory Leakage since OpenBSD 7.3 upgrade (nginx and MariaDB; Not consistent)
Do the affected programs use the same libraries? On Sun, 2023-09-24 at 09:32 +0200, Tobias Fiebig wrote: > After upgrading to 7.3 and nginx-1.24.0, i started to see heavy > memory > leakage over time. I initially attributed this to nginx, and solved > the > issue by ignoring it/throwing a bit more memory at the box for some > time. However, I started debugging it now and could trace it to some > commit between nginx 1.23.3 and 1.23.4; Currently, I am going through > all commits to see with which version it first appears. > > However, clicking around this morning, i noticed that my primary NS > shows a similar memory leakage for mysql/mariadb (runs powerdns with > a > mysql backend, both from packages) since the upgrade to 7.3 in May as > well. One further host seems to show a significantly higher use of > inactive memory since 7.3. I found one more host with mariadb that > now > shows a higher utilization of inactive memory (gitea+maria); However, > other maria+X instances on 7.3 run fine.
Re: Unclear Memory Leakage since OpenBSD 7.3 upgrade (nginx and MariaDB; Not consistent)
They do, but nothing special. The common set between nginx and mysqld is: /usr/local/lib/libpcre2-8.so.0.6 /usr/lib/libssl.so.53.2 /usr/lib/libcrypto.so.50.2 /usr/lib/libz.so.7.0 /usr/lib/libc.so.97.0 /usr/libexec/ld.so However, an affected nginx (1.24.0) does not differ from an unaffected nginx (1.23.1). I somewhat hope that my testing through nginx commits will yield what calls are actually involved, and may allow me to see whether mysql is pushed into similar codepaths on affected systems (and not on unaffected ones), likely giving a better hint as to where the issue is. With best regards, Tobias On Sun, 2023-09-24 at 12:53 +0200, Rudolf Leitgeb wrote: > Do the affected programs use the same libraries? > > On Sun, 2023-09-24 at 09:32 +0200, Tobias Fiebig wrote: > > After upgrading to 7.3 and nginx-1.24.0, i started to see heavy > > memory > > leakage over time. I initially attributed this to nginx, and solved > > the > > issue by ignoring it/throwing a bit more memory at the box for some > > time. However, I started debugging it now and could trace it to > > some > > commit between nginx 1.23.3 and 1.23.4; Currently, I am going > > through > > all commits to see with which version it first appears. > > > > However, clicking around this morning, i noticed that my primary NS > > shows a similar memory leakage for mysql/mariadb (runs powerdns > > with > > a > > mysql backend, both from packages) since the upgrade to 7.3 in May > > as > > well. One further host seems to show a significantly higher use of > > inactive memory since 7.3. I found one more host with mariadb that > > now > > shows a higher utilization of inactive memory (gitea+maria); > > However, > > other maria+X instances on 7.3 run fine. >
Re: Unclear Memory Leakage since OpenBSD 7.3 upgrade (nginx and MariaDB; Not consistent)
This libpcre2 library seems to be the only one, which is not used all over the place. The library itself may not even be buggy, it may just return something, which the new versions of the caller can't handle, or it may be unhappy with something the new callers send. Still: if you can tie this memory/cpu leak to the interaction between application and that library, it should be much easier to identify the offending commit in nginx and all the other affected programs. I just saw, that the version 8 of libpcre2 seems to be quite a bit behind the current version: https://github.com/PCRE2Project/pcre2/releases Is this intentional? On Sun, 2023-09-24 at 12:59 +0200, Tobias Fiebig wrote: > They do, but nothing special. The common set between nginx and mysqld > is: > > /usr/local/lib/libpcre2-8.so.0.6 > /usr/lib/libssl.so.53.2 > /usr/lib/libcrypto.so.50.2 > /usr/lib/libz.so.7.0 > /usr/lib/libc.so.97.0 > /usr/libexec/ld.so > > However, an affected nginx (1.24.0) does not differ from an > unaffected > nginx (1.23.1). > > I somewhat hope that my testing through nginx commits will yield what > calls are actually involved, and may allow me to see whether mysql is > pushed into similar codepaths on affected systems (and not on > unaffected ones), likely giving a better hint as to where the issue > is. > > With best regards, > Tobias > > On Sun, 2023-09-24 at 12:53 +0200, Rudolf Leitgeb wrote: > > Do the affected programs use the same libraries? > > > > On Sun, 2023-09-24 at 09:32 +0200, Tobias Fiebig wrote: > > > After upgrading to 7.3 and nginx-1.24.0, i started to see heavy > > > memory > > > leakage over time. I initially attributed this to nginx, and > > > solved > > > the > > > issue by ignoring it/throwing a bit more memory at the box for > > > some > > > time. However, I started debugging it now and could trace it to > > > some > > > commit between nginx 1.23.3 and 1.23.4; Currently, I am going > > > through > > > all commits to see with which version it first appears. > > > > > > However, clicking around this morning, i noticed that my primary > > > NS > > > shows a similar memory leakage for mysql/mariadb (runs powerdns > > > with > > > a > > > mysql backend, both from packages) since the upgrade to 7.3 in > > > May > > > as > > > well. One further host seems to show a significantly higher use > > > of > > > inactive memory since 7.3. I found one more host with mariadb > > > that > > > now > > > shows a higher utilization of inactive memory (gitea+maria); > > > However, > > > other maria+X instances on 7.3 run fine. > > >
Re: Unclear Memory Leakage since OpenBSD 7.3 upgrade (nginx and MariaDB; Not consistent)
On Sun, 2023-09-24 at 13:07 +0200, Rudolf Leitgeb wrote: > This libpcre2 library seems to be the only one, which is not > used all over the place. The library itself may not even be buggy, it > may just return something, which the new versions of the caller can't > handle, or it may be unhappy with something the new callers send. > > Still: if you can tie this memory/cpu leak to the interaction between > application and that library, it should be much easier to identify > the offending commit in nginx and all the other affected programs. Yes, _if_ it is that library. I would not fully set my sights on that yet. Nginx has the benefit of having a cut somewhere that triggers this; Hopefully the tests are done soon (i went for brute force instead of binary search; takes four days but requires no interaction). But yes, getting a specific commit there will be helpful. > I just saw, that the version 8 of libpcre2 seems to be quite a bit > behind the current version: > https://github.com/PCRE2Project/pcre2/releases > > Is this intentional? I am using what comes from the packages/ports. So the intentions on that are with the maintainers there. With best regards, Tobias
porting snort3
Hello, Is there any plan for porting Snort3 into OpenBSD? Thanks. Best, Valdrin
Re: Unclear Memory Leakage since OpenBSD 7.3 upgrade (nginx and MariaDB; Not consistent)
My dev environment on 7.3 is completely different but a little more standard I share about my nginx: wiz$ pkg_info nginx Information for inst:nginx-1.22.0p0 Comment: robust and small HTTP server and mail proxy server Description: This is the stable branch of nginx, as distributed by nginx.org. nginx provides FastCGI and reverse HTTP proxying, with or without caching, including simple load balancing and fault tolerance. It has a modular architecture. Filters include gzipping, byte ranges, chunked responses, and SSI. Multiple SSI inclusions within a single page can be processed in parallel if they are handled by FastCGI or proxied servers. SSL and TLS SNI are supported. Maintainer: Robert Nagy WWW: https://nginx.org/ wiz$ ldd /usr/local/sbin/nginx /usr/local/sbin/nginx: StartEnd Type Open Ref GrpRef Name 0f5043481000 0f50435b4000 exe 10 0 /usr/local/sbin/nginx 0f52fcc29000 0f52fcc6f000 rlib 01 0 /usr/local/lib/libpcre.so.3.0 0f52dabca000 0f52dac3a000 rlib 01 0 /usr/lib/libssl.so.53.2 0f52b0c0e000 0f52b0e69000 rlib 02 0 /usr/lib/libcrypto.so.50.2 0f52e1cf9000 0f52e1d17000 rlib 01 0 /usr/lib/libz.so.7.0 0f531de9f000 0f531df95000 rlib 01 0 /usr/lib/libc.so.97.0 0f53165dc000 0f53165dc000 ld.so 01 0 /usr/libexec/ld.so I also own MariaDB, 10.9.4. I underline that on my system "pkg_info mysql" or "pkg_info mariadb" return blank and back to cursor. -- Daniele Bonini Tobias Fiebig wrote: > Moin, > > I have been dealing with memory-leaks on a host running an nginx > reverse proxy for some time. The host had been running 7.2 with nginx > 1.23.1 (self compiled as i need some features not in the package) > until May, which was fine. > > After upgrading to 7.3 and nginx-1.24.0, i started to see heavy memory > leakage over time. I initially attributed this to nginx, and solved > the issue by ignoring it/throwing a bit more memory at the box for > some time. However, I started debugging it now and could trace it to > some commit between nginx 1.23.3 and 1.23.4; Currently, I am going > through all commits to see with which version it first appears. > > However, clicking around this morning, i noticed that my primary NS > shows a similar memory leakage for mysql/mariadb (runs powerdns with a > mysql backend, both from packages) since the upgrade to 7.3 in May as > well. One further host seems to show a significantly higher use of > inactive memory since 7.3. I found one more host with mariadb that now > shows a higher utilization of inactive memory (gitea+maria); However, > other maria+X instances on 7.3 run fine. > > On the PowerDNS+Mysql host I also see gradually increasing CPU load > along with the memory leakage. > > Restarting mysql and/or nginx resolves the issue for some time. I also > saw the issue on a host running maria+nextcloud, where i fixed it by > switching to postgres. Maria is still running there, but not receiving > any requests, and the memory leak is gone. > > Images: > > https://rincewind.home.aperture-labs.org/~tfiebig/pdns-maria-cpu-year.png > https://rincewind.home.aperture-labs.org/~tfiebig/pdns-maria-memory-year.png > > https://rincewind.home.aperture-labs.org/~tfiebig/nginx-rev-proxy-memory-year.png > > https://rincewind.home.aperture-labs.org/~tfiebig/gitea-maria-memory-year.png > > https://rincewind.home.aperture-labs.org/~tfiebig/nextcloud-maria-switched-to-postgres-memory-year.png > > All hosts are VMs on Linux/KVM hosts (proxmox), with a generic > qemu-kvm processor as CPU type, running 7.3. > > Hence, I am no longer 100% convinced that this is an nginx problem. To > help circling this in a bit closer (and maybe get it reproducible in a > first step): > > Did anyone else experience memory leakage on openbsd with mariadb or a > self-build >=nginx-1.23.4 (or other applications) since the upgrade to > 7.3? > > With best regards, > Tobias >
OpenBSD FUD with Contributing
Just wanted to drop a note saying that for perhaps the entire length of the project I've heard how brutal it is to help contribute to the project of OpenBSD (think I've been using OpenBSD on and off since ~1998ish with obecian@ introducing me to it). But being back on the mailing lists I see how the feedback given is meant to be constructive, though can be a bit gruff at times, but is always helpful. I see developers and contributors pushing folks in the right direction like bumper lanes in bowling, and that is awesome. I know I helped spread the FUD before in my teens/early 20s, I was kind of a well... a teen and young 20-something loner smart guy with a big ego. Now I'm a 40-something not-so-loner smart guy that knows how much he doesn't know and can see kindness instead of attacks, without much of an ego at all. Anyway, so if anyone is reading this and you hear how OpenBSD is brutal to contribute and help with, realize that at best that information may be dated, but most likely it is simply incorrect. There seem to be very talented programmers and engineers on this project that truly care about quality and security, even if they have to lead the way and push/force upstream software to also care. If anything, it is incredible to see what lone individuals can contribute if they're willing to apply the effort and be open to feedback from those that know better. And the changes don't need to be huge to have a big impact. While the 90s were a bit more brutal of simply "RTFM, idiot" across the board, we seem to be in a more sensible time. BUT I love that OpenBSD is still keeping K.I.S.S. and RTFM a viable argument in the true Unix fashion, while others have chosen... poorly. What's wild to me-from-20-years-ago is that there's so much effort being put into porting Wayland and KDE Plasma (guess we're not all running Matrix-themed WindowMaker desktops anymore). Also OpenBSD gaming is now a thing, which blows my mind (see r/openbsd-gaming on Reddit and #openbsd-gaming on Libera.Chat IRC, also stuff on Mastodon, too). So, while I do personally apologize for spreading FUD about OpenBSD being a tough place to contribute to at times in my life, know that it certainly isn't the case. Be sure to stop the FUD about it if you hear it, too. (Theo still has some of the best quotes on the Internet.)
Re: OpenBSD FUD with Contributing
On 9/24/23 15:56, Christoff Humphries wrote: ... (Theo still has some of the best quotes on the Internet.) Used this one, for quite some time, as my email signature a few years ago: “You've been smoking something really mind altering, and I think you should share it.” (Theo de Raadt)
Re: Unclear Memory Leakage since OpenBSD 7.3 upgrade (nginx and MariaDB; Not consistent)
> But yes, getting a specific commit there will be helpful. Sadly it turns out that it is the commit i feared it would be: > commit 7b24b93d67daa9c16d665129fd5d3e7dbc583e4f > Author: Maxim Dounin > Date: Fri Mar 24 02:57:43 2023 +0300 > > SSL: enabled TLSv1.3 by default. Feared, because it basically puts me back to start w.r.t. what the root cause might be; Could be anything that happened to TLSv1.3 code in either LibreSSL or Nginx. I guess the next step is going through all commits of libressl between what is in 7.2 and 7.3. With best regards, Tobias
Re: Does openBSD come with a web browser?
Hi Eric, You'll find how to install OpenBSD following FAQ pretty easily. After install, you'll be able to add packages (install software) with a simple internet connection. You'd have to install for example XFCE, Thunderbird, Firefox, Chromium. OpenBSD base install does includes a set of GUI and packages, but not a full fledged OS, but that's easy to do and above recommended packages should do well. Forget about searches, at this point you can easily start install base OS, packages, if needed get help on mail list or IRC, first go to the man and FAQ on website, they provide a path to get you up and running no difficulty. Regards Jean-François On 9/12/23 08:21, Eric Demer wrote: (I am considering getting a laptop with openBSD, but have not yet done so, which is why I can't easily check on my own.) Does openBSD come with a web browser? The "the FAQ and" parts of https://www.openbsd.org/mail.html suggest that it does, but I haven't found any more detail regarding this at https://www.openbsd.org/ . Quite frankly, if you're incapable of using one, I'd steer clear. The answer to this is the result of a very basic web search. Cheers! Perhaps I should steer clear anyway, but what's probably the reason I didn't find that answer may change things. Specifically, do you find that information with a basic web search while using none ofStackexchange , Reddit , Youtube , Google ? For the reasons explained in the following paragraphs, I am not willing to use those four sites. I still got into results saying that one _can easily install_ Firefox on openBSD, and remember at least one result saying that some people _use_ Lynx _on_ it, but those didn't address whether there's one that comes _already_ installed. I did go into results saying that one _can easily install_ Firefox on openBSD, and remember at least one result saying that some people _use_ Lynx _on_ it, but those didn't address whether there's one that comes _already_ installed. The other search results (from using duckduckgo) I found that mentioned openBSD - as opposed to just freeBSD - were all from stackexchange and reddit and youtube. I left Stackexchange when it adopted Terms according to which, them changing those terms other than the arbitration clause as I am scrolling a page on their site would result in me being bound by whatever they changed the Terms to. Since the trigger for those Terms was something like, using their Network in any way, I have never intentionally gone back there, and have left immediately when I've accidentally when I've accidentally gone back there. (In particular, if they no longer have such Terms then I don't know that.) My brief search for Reddit's Terms brought up Reddit result previews suggesting that Reddit's Terms are also such that according to them, using their site to view their terms would constitute acceptance of those terms. Furthermore, according to https://github.com/OpenTermsArchive/contrib-versions /blob/main/Reddit/Terms%20of%20Service.md , the changes provision in Reddit's Terms manages to be even worse than that of Stackexchange's Terms: Its change-acceptance is from access to or use of "the Services on or after the Effective Date of the revised Terms", and it does not say the Effective Date can't be _before_ the revised Terms were posted. Youtube's Terms are better, but (0) it's Google, and (1) the "launch a new product or feature" exception is merely a timing restriction: It's not limited to changes that have anything else to do with the new product or feature. Google's Terms seem to have the same changes provision. Eric Demer
Re: Does openBSD come with a web browser?
The FAQ is nice, but there are also folks out there that have written some additional handy resources, such as: - https://www.k58.uk/openbsd.html (on installing and getting XFCE and Firefox working, including changes to staff group to increase allowed resource limits, etc) - https://www.openbsdhandbook.com/ (howtos on many things) -- including https://www.openbsdhandbook.com/services/webserver/ssl/ (how to setup httpd with acme-client with multiple domains) Note that after you install packages via pkg_add, there may be a note displayed telling you to read a file. Within that file is important information you should know. They're usually in the /usr/local/share/doc/pkg-readmes directory and you should read them. For Firefox in particular it will tell you things you may want to do to get the behavior you're used to. See: https://github.com/openbsd/ports/blob/master/www/mozilla-firefox/pkg/README And for XFCE: https://github.com/openbsd/ports/blob/master/meta/xfce/pkg/README-main --- Original Message --- On Sunday, September 24th, 2023 at 11:34 PM, Jean-François Simon wrote: > > > Hi Eric, > > You'll find how to install OpenBSD following FAQ pretty easily. > > After install, you'll be able to add packages (install software) with a > simple internet connection. > > You'd have to install for example XFCE, Thunderbird, Firefox, Chromium. > > OpenBSD base install does includes a set of GUI and packages, but not a > full fledged OS, but that's easy to do and above recommended packages > should do well. > > Forget about searches, at this point you can easily start install base > OS, packages, if needed get help on mail list or IRC, first go to the > man and FAQ on website, they provide a path to get you up and running no > difficulty. > > Regards > > Jean-François > > > On 9/12/23 08:21, Eric Demer wrote: > > > > > (I am considering getting a laptop with openBSD, but have > > > > not yet done so, which is why I can't easily check on my own.) > > > > > > > > Does openBSD come with a web browser? > > > > The "the FAQ and" parts of https://www.openbsd.org/mail.html > > > > suggest that it does, but I haven't found any more > > > > detail regarding this at https://www.openbsd.org/ . > > > > Quite frankly, if you're incapable of using one, I'd steer clear. > > > > The answer to this is the result of a very basic web search. > > > > Cheers! > > > > Perhaps I should steer clear anyway, but what's probably > > the reason I didn't find that answer may change things. > > > > Specifically, do you find that information with a basic web search > > while using none of Stackexchange , Reddit , Youtube , Google ? > > > > For the reasons explained in the following paragraphs, I am > > not willing to use those four sites. I still got into results saying > > that one can easily install Firefox on openBSD, and remember at > > least one result saying that some people use Lynx on it, but those > > didn't address whether there's one that comes already installed. > > > > I did go into results saying that one can easily install > > Firefox on openBSD, and remember at least one result saying > > that some people use Lynx on it, but those didn't > > address whether there's one that comes already installed. > > The other search results (from using duckduckgo) I found > > that mentioned openBSD - as opposed to just freeBSD - > > were all from stackexchange and reddit and youtube. > > > > I left Stackexchange when it adopted Terms according to which, > > them changing those terms other than the arbitration clause > > as I am scrolling a page on their site would result in > > me being bound by whatever they changed the Terms to. > > Since the trigger for those Terms was something like, > > using their Network in any way, I have never intentionally > > gone back there, and have left immediately when I've > > accidentally when I've accidentally gone back there. > > (In particular, if they no longer have > > such Terms then I don't know that.) > > > > My brief search for Reddit's Terms brought up Reddit > > result previews suggesting that Reddit's Terms are also > > such that according to them, using their site to view > > their terms would constitute acceptance of those terms. > > Furthermore, according to > > https://github.com/OpenTermsArchive/contrib-versions > > /blob/main/Reddit/Terms%20of%20Service.md > > , the changes provision in Reddit's Terms manages > > to be even worse than that of Stackexchange's Terms: > > Its change-acceptance is from access to or use of "the Services on or > > after the Effective Date of the revised Terms", and it does not say > > the Effective Date can't be before the revised Terms were posted. > > > > Youtube's Terms are better, but (0) it's Google, and > > (1) the "launch a new product or feature" exception is > > merely a timing restriction: It's not limited to changes > > that have anything else to do with the new product or feature. > > Google's Terms seem t
Re: Does openBSD come with a web browser?
Neat On 9/25/23 02:03, Christoff Humphries wrote: The FAQ is nice, but there are also folks out there that have written some additional handy resources, such as: - https://www.k58.uk/openbsd.html (on installing and getting XFCE and Firefox working, including changes to staff group to increase allowed resource limits, etc) - https://www.openbsdhandbook.com/ (howtos on many things) -- including https://www.openbsdhandbook.com/services/webserver/ssl/ (how to setup httpd with acme-client with multiple domains) Note that after you install packages via pkg_add, there may be a note displayed telling you to read a file. Within that file is important information you should know. They're usually in the /usr/local/share/doc/pkg-readmes directory and you should read them. For Firefox in particular it will tell you things you may want to do to get the behavior you're used to. See: https://github.com/openbsd/ports/blob/master/www/mozilla-firefox/pkg/README And for XFCE: https://github.com/openbsd/ports/blob/master/meta/xfce/pkg/README-main --- Original Message --- On Sunday, September 24th, 2023 at 11:34 PM, Jean-François Simon wrote: Hi Eric, You'll find how to install OpenBSD following FAQ pretty easily. After install, you'll be able to add packages (install software) with a simple internet connection. You'd have to install for example XFCE, Thunderbird, Firefox, Chromium. OpenBSD base install does includes a set of GUI and packages, but not a full fledged OS, but that's easy to do and above recommended packages should do well. Forget about searches, at this point you can easily start install base OS, packages, if needed get help on mail list or IRC, first go to the man and FAQ on website, they provide a path to get you up and running no difficulty. Regards Jean-François On 9/12/23 08:21, Eric Demer wrote: (I am considering getting a laptop with openBSD, but have not yet done so, which is why I can't easily check on my own.) Does openBSD come with a web browser? The "the FAQ and" parts of https://www.openbsd.org/mail.html suggest that it does, but I haven't found any more detail regarding this at https://www.openbsd.org/ . Quite frankly, if you're incapable of using one, I'd steer clear. The answer to this is the result of a very basic web search. Cheers! Perhaps I should steer clear anyway, but what's probably the reason I didn't find that answer may change things. Specifically, do you find that information with a basic web search while using none of Stackexchange , Reddit , Youtube , Google ? For the reasons explained in the following paragraphs, I am not willing to use those four sites. I still got into results saying that one can easily install Firefox on openBSD, and remember at least one result saying that some people use Lynx on it, but those didn't address whether there's one that comes already installed. I did go into results saying that one can easily install Firefox on openBSD, and remember at least one result saying that some people use Lynx on it, but those didn't address whether there's one that comes already installed. The other search results (from using duckduckgo) I found that mentioned openBSD - as opposed to just freeBSD - were all from stackexchange and reddit and youtube. I left Stackexchange when it adopted Terms according to which, them changing those terms other than the arbitration clause as I am scrolling a page on their site would result in me being bound by whatever they changed the Terms to. Since the trigger for those Terms was something like, using their Network in any way, I have never intentionally gone back there, and have left immediately when I've accidentally when I've accidentally gone back there. (In particular, if they no longer have such Terms then I don't know that.) My brief search for Reddit's Terms brought up Reddit result previews suggesting that Reddit's Terms are also such that according to them, using their site to view their terms would constitute acceptance of those terms. Furthermore, according to https://github.com/OpenTermsArchive/contrib-versions /blob/main/Reddit/Terms%20of%20Service.md , the changes provision in Reddit's Terms manages to be even worse than that of Stackexchange's Terms: Its change-acceptance is from access to or use of "the Services on or after the Effective Date of the revised Terms", and it does not say the Effective Date can't be before the revised Terms were posted. Youtube's Terms are better, but (0) it's Google, and (1) the "launch a new product or feature" exception is merely a timing restriction: It's not limited to changes that have anything else to do with the new product or feature. Google's Terms seem to have the same changes provision. Eric Demer
