On Tue, Jan 24, 2023 at 11:59:16PM -0600, Marc West wrote: > On 2023-01-24 23:04:14, Olivier Houchard wrote: > > On Tue, Jan 24, 2023 at 11:05:37PM +0100, Willy Tarreau wrote: > > > On Tue, Jan 24, 2023 at 02:15:08PM -0600, Marc West wrote: > > > > > Stupid question but I prefer to ask in order to be certain, are all of > > > > > these 32 threads located on the same physical CPU ? I just want to be > > > > > sure that locks (kernel or user) are not traveling between multiple > > > > > CPU > > > > > sockets, as that ruins scalability. > > > > > > > > Very good observation. This is a 2 socket system, 20 cores per socket, > > > > and since there is no affinity in OpenBSD unfortunately the 32 threads > > > > are not on the same physical CPU. > > > > > > Ah, then that's particularly bad. When you're saying "there is no > > > affinity", do you mean the OS doesn't implement anything or only that > > > haproxy doesn't support it ? Could you please check if you have man > > > pages about "cpuset_setaffinity" or "sched_setaffinity" ? The former > > > is the FreeBSD version and the second the Linux version. > > > > Unfortunately, having a quick look at OpenBSD's sources, I don't think > > they provide anything that could look like that. In fact, I'm not sure > > there is any way to restrict a process to a specific CPU. > > Unfortunately that is what I found also, no references to affinity in > manpages nor finding any tools like taskset or NUMA related knobs. > It seems to be a design decision. > > https://marc.info/?l=openbsd-misc&m=152507006602422&w=2 > https://marc.info/?l=openbsd-tech&m=133909957708933&w=2 > https://marc.info/?l=openbsd-misc&m=135884346431916&w=2
Then it's really pathetic, because implementing multi-CPU support without being able to pin tasks to CPUs it counter-productive. I know that their focus is primarily on code simplicity and auditability but they already lost half of it by supporting SMP, and not doing it right only makes the situation worse, and even opens the way to trivial DoS attacks on running processes. Sadly the response to that message from Ilya in the last comment you pointed above says it all: "The scheduler will know what CPUs are busy and which ones are not and will make apprpriate decisions." That's of course an indication of total ignorance of how hardware works, since the right CPU to use is not directly related to the ones that are busy or not, but first and foremost to the cost of the communications with other threads/processes/devices that is related to the distance between them, and the impacts of cache line migration. That's precisely why operating systems supporting multi-processing all offer the application the ability to set its affinity based on what it knows about its workload. And seeing that it was already required 10 years ago and still not addressed just shows a total lack of interest for multicore machines. > I thought there might be a way to enable just the CPUs that are on the > first package via the kernel config mechanism but attempting that > resulted in an immediate reboot. It looks like some kernel hacking would > be needed to do this. There are no options in the BIOS to disable one of > the sockets unfortunately and I don't have any single-socket machines to > test with (aside from a laptop). Yeah that's annoying. And you're not necessarily interested in running that OS inside a virtual machine managed by a more capable operating system. > Gven that I think we are going to have to plan to switch OS to FreeBSD > for now. Yeah in your case I think that's the best plan. In addition, there's a large community using haproxy on FreeBSD, and it's generally agreed that it's the second operating system working the best with haproxy. Like Linux it remains a modern and featureful OS with a powerful network stack, and I think you shouldn't have any issues seeking help about it if needed. > OpenBSD is a great OS and a joy to work with but for this > particular use case we do need to handle much more HTTPS traffic in > production. I agree. I've used it for a long time and really loved its light weight, simplicity and reliability. It has been working flawlessly for something like 10 years on my VAX with 24 MB RAM. I had to stop due to disks failing repeatedly. But while I think that running it on a Geode-LX based firewall or VPN can still make sense nowadays for an ADSL line, I also think it's about time to abandon designs stuck in the 90s that try too hard to run correctly on hardware designed in 2020+ with totally different concepts. As a rule of thumb it should probably not be installed on a machine which possesses a fan. > Thanks again for all the replies and I will have this hardware available > for the foreseeable future in case there is any more testing that would > be helpful. You're welcome, and thanks for sharing your experience, it's always useful to many others! Willy
