Steve,
I checked with strace and top -p (press H to see threads).
It seems that Go can create few more threads than cores are given by taskset.
For example, on an 8 core ARM machine in the cloud, I got these
results on a program launching 100 goroutines utilizing a lot of CPU
each.
$ strace --follow-forks -o /tmp/m2.log taskset --cpu-list 0 ./main
1
^C
$ strace --follow-forks -o /tmp/m2.log taskset --cpu-list 0-7 ./main
8
^C
$ grep -c clone /tmp/m.log
3
$ grep -c clone /tmp/m2.log
13
So for 8 cores it launched 13 threads, for 1 core - 3 threads.
You can try running your tool under "strace --follow-forks" and count
how many "clone" calls there are in the log.
Try to set just 1 core in taskset (taskset --cpu-list 0) and see if it
works and how many threads it launches. If it works, try to increase
the number of cores in taskset until it breaks.
Here is my program:
package main
import (
"sync/atomic"
"fmt"
"runtime"
"time"
)
func main() {
fmt.Println(runtime.NumCPU())
var b atomic.Int64
for i := 0; i < 100; i++ {
go func() {
for {
b.Add(1)
time.Sleep(1000)
}
}()
}
time.Sleep(1000*time.Second)
}
On Thu, Feb 1, 2024 at 1:08 PM Steve Roth <st...@rothskeller.net> wrote:
>
> Thanks to the people who suggested how to limit the number of apparent cores.
> Unfortunately, doing that didn't solve the problem. I have the number of
> cores now limited to two — confirmed by checking runtime.NumCPU() — but the
> program is still trying to allocate more than 25 threads, and crashing when
> cgroups won't let it.
>
> Surely there must be some way to get a Go program to run successfully in an
> environment with process limits? Any further suggestions would be greatly
> appreciated.
>
> Regards,
> Steve
>
>
> On Wed, Jan 31, 2024 at 6:43 PM Steve Roth <st...@rothskeller.net> wrote:
>>
>> I am running Go code on a shared web hosting server from a major hosting
>> company. Using cgroups, they limit the number of threads any user can
>> create to 25. Up until a week ago, they had me running on a server with 24
>> cores, and everything worked fine. Now they've moved me to a new server
>> with 128 cores. The Go runtime thinks it should be able to create that many
>> threads, and it can't, and all of my Go programs crash with, "runtime:
>> failed to create new OS thread".
>>
>> Setting GOMAXPROCS doesn't help this, since it only controls the number of
>> active threads, not the total number of threads. The Go runtime still
>> thinks it can create as many threads as there are cores, and crashes the
>> program when it's blocked from doing so.
>>
>> Is there any way around this? Or is Go just completely unusable in an
>> environment where the per-user process limit is less than the number of
>> cores?
>>
>> Many thanks,
>> Steve
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CAAnpqKEzZcFxDkG2Qtzf25hPwkZq2EstbDbYtBz4CtTaBQxqWA%40mail.gmail.com.
--
Best regards,
Boris Nagaev
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/CAFC_Vt4JfNa_wXF8s8qf%2BxpogQsVeKORoq0870g%3DU8wHGu2eww%40mail.gmail.com.
