Re: [go-nuts] stamped lock in Go?

2025-01-31 Thread 'Michael Knyszek' via golang-nuts
On Friday, January 31, 2025 at 1:58:39 PM UTC-5 Michael Knyszek wrote: I think I got a little overzealous with my opinions, it wasn't my intent to be particularly prescriptive. I also did not get the impression that you were suggesting the memory model should be different. Bottom line

Re: [go-nuts] stamped lock in Go?

2025-01-31 Thread 'Michael Knyszek' via golang-nuts
applicable). > > I think a more viable performant replacement in Go is to use copy on write > with an atomic object reference - which in most cases is going to be more > performant than using a StampedLock. > > On Jan 31, 2025, at 11:55 AM, 'Michael Knyszek' via gola

[go-nuts] Re: stamped lock in Go?

2025-01-31 Thread 'Michael Knyszek' via golang-nuts
This is a well-known pattern (use an atomic counter to check if anyone else modified some data in a critical section and retry or fall back if it happened), but as the extensive Java docs imply, any such abstraction is going be fairly leaky. Java has a very thoroughly defined memory model that

[go-nuts] Re: How does Go GC set target heap size in kubernetes?

2025-01-17 Thread 'Michael Knyszek' via golang-nuts
The Go runtime isn't aware if it's running in a pod/container/cgroup/etc. or not, not out of the box anyway. Also, GOMEMLIMIT=infinity (effectively) by default. Are you using something like https://github.com/KimMachineGun/automemlimit? At least, that would also explain why you saw things chang

[go-nuts] [security] Go 1.24 Release Candidate 2 is released

2025-01-16 Thread 'Michael Knyszek' via golang-nuts
Hello gophers, We have just released go1.24rc2, a release candidate version of Go 1.24. This release candidate includes 4 security fixes following the security policy : - crypto/x509: usage of IPv6 zone IDs can bypass URI name constraints A certificate with

[go-nuts] [security] Go 1.23.5, Go 1.22.11, and Go 1.24 RC2 pre-announcement

2025-01-13 Thread 'Michael Knyszek' via golang-nuts
Hello gophers, We plan to issue Go 1.23.5, Go 1.22.11, and Go 1.24 RC2 during US business hours on Thursday, January 16. These releases include PRIVATE security fixes to the standard library, covering the following CVEs: - CVE-2024-45336 - CVE-2024-45341 The Go 1.24 RC2 release

[go-nuts] Re: Performance of unique package

2024-06-25 Thread 'Michael Knyszek' via golang-nuts
The unique package itself has a few benchmarks you can run if you'd like, though they're somewhat limited. They exist primarily to validate that the implementation has

Re: [go-nuts] Memory operations hung in semacquire state, GC routine parked while holding semaphore?

2024-05-29 Thread 'Michael Knyszek' via golang-nuts
The example goroutine in the original post is parked waiting for some other goroutine to finish up the GC cycle. Somewhere, a goroutine is getting stuck trying to finish it up, which could possibly be a deadlock. (I am especially suspicious of a deadlock bug because most threads are stopped the

[go-nuts] Re: discrepancy between goal heap size and NextGC

2024-01-30 Thread 'Michael Knyszek' via golang-nuts
838103616 bytes is 799 MiB (Mebibytes, the power-of-two-based prefix), which matches GC 13. (Note that the computation for the heap goal just divides by 1<<20, or 1 MiB.) The discrepancy in GC count is probably due to the fact that the GC count value reported by the GC trace is *after* the incr

[go-nuts] Re: array index not starting at 0

2024-01-30 Thread &#x27;Michael Knyszek' via golang-nuts
I see that you're on Windows. The clock granularity on Windows is really coarse (15 ms) because AFAIK there are no fast fine-grained OS-level clocks available. There are fine-grained clocks, but they're heavyweight. They're too heavyweight for many situations in the runtime, but honestly, they'

Re: [go-nuts] Re: Passing 2d arrays from Go to C using runtime.Pinner

2024-01-29 Thread &#x27;Michael Knyszek' via golang-nuts
hm is for debugging > purposes only. The use of runtime.Pinner is not required. A single > implicitly pinned C wrapper function argument pointing to contiguous > underlying slice data arrays would suffice. > > Peter > > > On Friday, January 26, 2024 at 12:05:38 PM UTC-5 Michael Knyszek

[go-nuts] Re: Passing 2d arrays from Go to C using runtime.Pinner

2024-01-26 Thread &#x27;Michael Knyszek' via golang-nuts
Ignoring more efficient ways to pass memory to C for a moment, superficially I do think your code using Pinner should work. Do you have a full reproducer? It's hard to tell from just looking at your code if your code is the problem, or its just enough to trigger some other cgo issue elsewhere i

[go-nuts] Re: garbage collections on Windows and Mac

2024-01-09 Thread &#x27;Michael Knyszek' via golang-nuts
On Tuesday, January 9, 2024 at 10:21:26 PM UTC-5 Michael Mitchell wrote: Hi, I have noticed on my macbook pro that HeapInUse is always slightly above NextGC when the garbage collection occurs. According to mstats.go, the "garbage collector's goal is to keep HeapAlloc less than NextGC" and

[go-nuts] Re: Low memory utilization when using soft memory limit with GOGC=off

2023-12-06 Thread &#x27;Michael Knyszek' via golang-nuts
On Tuesday, December 5, 2023 at 1:06:19 AM UTC-5 Zhihui Jiang wrote: Hi there, We are running a large scale recommendation system using Golang and we are working on some GC related improvement recently. One of the changes we are trying to apply is to use soft memory limit with GOGC=off as su

Re: [go-nuts] Go/Python interop

2023-11-27 Thread &#x27;Michael Knyszek' via golang-nuts
On Thu, Nov 23, 2023 at 11:52 AM Sebastien Binet wrote: > On Thu Nov 23, 2023 at 15:23 CET, 'Michael Knyszek' via golang-nuts wrote: > > Also, I'd like to clarify: > > > > - [David]: is it a good idea to use cgo for Go-Python interop? > > - [Michael

Re: [go-nuts] Go/Python interop

2023-11-23 Thread &#x27;Michael Knyszek' via golang-nuts
Also, I'd like to clarify: - [David]: is it a good idea to use cgo for Go-Python interop? - [Michael]: no. better with pipe or RPC I'm wrong about this. A conversation after the meeting clarified a misunderstanding I had about Go->Python calls specifically. Both Go->Python and Python->Go with c

[go-nuts] Re: Why golang's garbage collector didn't free obj when an finalizer is associate with it?

2023-11-07 Thread &#x27;Michael Knyszek' via golang-nuts
t one finalizer, one could argue such an ordering problem doesn't exist (unless I'm missing something of course). Nonetheless, the cycle still isn't collected. On Tuesday, November 7, 2023 at 3:51:58 AM UTC+1 Michael Knyszek wrote: Yes, cycles containing a finalizer aren't guaran

[go-nuts] Re: Why golang's garbage collector didn't free obj when an finalizer is associate with it?

2023-11-06 Thread &#x27;Michael Knyszek' via golang-nuts
Yes, cycles containing a finalizer aren't guaranteed to be freed. As others have pointed out, this is documented. SetFinalizer is really designed for a narrow set of use-cases, so concessions were made for overall GC performance. This case is one of them. IIUC, the core problem is a combination

Re: [go-nuts] Suggestions on optimizing Go GC

2023-10-30 Thread &#x27;Michael Knyszek' via golang-nuts
I second Jason's message, and +1 to off-heap memory as a last resort. Here are a few more details: For a starting point on how to reduce memory allocations directly, see https://go.dev/doc/gc-guide#Optimization_guide. Note that this may require restructuring your program in places. (e.g. passin

Re: [go-nuts] Question about runtime.Pinner

2023-10-12 Thread &#x27;Michael Knyszek' via golang-nuts
On Thursday, October 12, 2023 at 9:55:12 AM UTC-4 Ian Lance Taylor wrote: On Thu, Oct 12, 2023 at 5:06 AM Pascal Costanza wrote: > > I have a question about runtime.Pinner: The documentation states that you can only pin an object when it’s the result of calling new, taking the address of a

[go-nuts] Re: Question about runtime.Pinner

2023-10-12 Thread &#x27;Michael Knyszek' via golang-nuts
On Thursday, October 12, 2023 at 8:07:02 AM UTC-4 Pascal Costanza wrote: Hi, I have a question about runtime.Pinner: The documentation states that you can only pin an object when it’s the result of calling new, taking the address of a composite literal, or taking the address of a local variab

[go-nuts] Re: go 1.21 tuned gc: what's it about?

2023-08-07 Thread &#x27;Michael Knyszek' via golang-nuts
I saw this post a while back and forgot to reply; sorry about that. It's about this issue but also a little bit about this issue . Both of those problems can be summarized as the GC pacing itself for the exac

[go-nuts] Re: Maybe a Bug? The Go compiler stores a stack pointer into a global object

2023-08-03 Thread &#x27;Michael Knyszek' via golang-nuts
That line (the full sentence is "The garbage collector now includes non-heap sources of garbage collector work (e.g., stack scanning) when determining how frequently to run.") is unrelated. It only refers to a change in accounting for what gets included in the GOGC calculation, not a change in

[go-nuts] Re: about "hybrid barrier"

2023-07-30 Thread &#x27;Michael Knyszek' via golang-nuts
On Friday, July 28, 2023 at 8:05:03 AM UTC-4 metronome wrote: Hi, I came across two questions regarding the hybrid barrier that I am hoping someone can help with. 1. Chang https://go-review.googlesource.com/c/go/+/31765 says: *"It's unconditional for now because barriers on channel operati

[go-nuts] Re: Possible Go Compiler or Runtime bug?

2023-07-27 Thread &#x27;Michael Knyszek' via golang-nuts
On Thursday, July 27, 2023 at 6:16:57 PM UTC-4 Michael Knyszek wrote: I believe this is working as intended, because I don't think the spec <https://go.dev/ref/spec#Conversions> makes any guarantees about the capacity of the slice you get back from a conversion. (Other than the fa

[go-nuts] Re: Possible Go Compiler or Runtime bug?

2023-07-27 Thread &#x27;Michael Knyszek' via golang-nuts
The number 32 for the capacity comes from this constant , and the returned byte slice in that case is likely coming from a compiler-inserted call to stringtoslicebyte

[go-nuts] Go 1.20 is released

2023-02-01 Thread &#x27;Michael Knyszek' via golang-nuts
Hello gophers, We have just released Go 1.20. To find out what has changed in Go 1.20, read the release notes: https://go.dev/doc/go1.20 You can download binary and source distributions from our download page: https://go.dev/dl/#go1.20 If you have Go installed already, an easy way to try go1.20

Re: [go-nuts] Create a 1GB ballast but it takes up RSS and pages are Dirty?

2022-11-16 Thread &#x27;Michael Knyszek' via golang-nuts
On Wed, Nov 16, 2022 at 10:19 AM Jie Zhang wrote: > Actually, we used go1.16.5 before go1.19 released. Now we're consider > using go1.19 GOMEMLIMIT instead. > > But GOMEMLIMIT is a soft limit, if you deploy multiple services in the > same host (not per container per service), there will be proble

Re: [go-nuts] Create a 1GB ballast but it takes up RSS and pages are Dirty?

2022-11-10 Thread &#x27;Michael Knyszek' via golang-nuts
That's correct. The runtime has a simple heuristic for avoiding zeroing but it's far from perfect. As a result, a ballast is inherently always going to be a little risky. This is especially true on some platforms, such as Windows, since there's no way to avoid marking the memory as committed (W

[go-nuts] Re: Go allocator: allocates arenas, reclaims pages?

2022-06-20 Thread &#x27;Michael Knyszek' via golang-nuts
une 20, 2022 at 5:46:36 PM UTC-4 Michael Knyszek wrote: > Thanks for the question. The scavenger isn't as publicly visible as other > parts of the runtime. You've got it mostly right, but I'm going to repeat > some things you've already said to make it cl

[go-nuts] Re: Go allocator: allocates arenas, reclaims pages?

2022-06-20 Thread &#x27;Michael Knyszek' via golang-nuts
Thanks for the question. The scavenger isn't as publicly visible as other parts of the runtime. You've got it mostly right, but I'm going to repeat some things you've already said to make it clear what's different. The Go runtime maps new heap memory (specifically: a new virtual memory mapping

[go-nuts] Go 1.17.4 and Go 1.16.11 are released

2021-12-02 Thread Michael Knyszek
Hello gophers, We have just released Go versions 1.17.4 and 1.16.11, minor point releases. View the release notes for more information: https://go.dev/doc/devel/release#go1.17.minor You can download binary and source distributions from the Go web site: https://go.dev/dl/ To compile from

[go-nuts] [security] Go 1.17.2 and Go 1.16.9 are released

2021-10-07 Thread Michael Knyszek
Hello gophers, We have just released Go versions 1.17.2 and 1.16.9, minor point releases. These minor releases include a security fix according to the new security policy (#44918 ). When invoking functions from WASM modules, built using GOARCH=wasm GOOS=js, passin

[go-nuts] Go 1.17 is released

2021-08-16 Thread Michael Knyszek
Hello gophers, We just released Go 1.17 To find out what has changed in Go 1.17, read the release notes: https://golang.org/doc/go1.17 You can download binary and source distributions from our download page: https://golang.org/dl/ If you have Go installed already, an easy way to try go1.17 is b

Re: [go-nuts] TotalAlloc dropped in runtime/metrics

2021-04-29 Thread &#x27;Michael Knyszek' via golang-nuts
Hi Marco, https://go-review.googlesource.com/c/go/+/312431 just landed and should resolve the issue for Go 1.17. On Tue, Apr 20, 2021 at 3:28 PM Michael Knyszek wrote: > Oh, actually, you can compute Mallocs and Frees from allocs-by-size and > frees-by-size (summing the total # of samples

Re: [go-nuts] TotalAlloc dropped in runtime/metrics

2021-04-20 Thread &#x27;Michael Knyszek' via golang-nuts
Oh, actually, you can compute Mallocs and Frees from allocs-by-size and frees-by-size (summing the total # of samples). You can only estimate TotalAlloc though, which is genuinely missing. On Tue, Apr 20, 2021 at 3:14 PM Michael Knyszek wrote: > Oh gosh, I think TotalAlloc, Mallocs, and Fr

Re: [go-nuts] TotalAlloc dropped in runtime/metrics

2021-04-20 Thread &#x27;Michael Knyszek' via golang-nuts
Oh gosh, I think TotalAlloc, Mallocs, and Frees are actually an oversight on my part. Sorry about that. They're very easy to add and I can probably even add them for this release. Please do use the new runtime/metrics package! Most of the other metrics should be there in some form (e.g. the divis

[go-nuts] Re: super large virtual memory allocation for simplest golang apps, why?

2021-01-14 Thread &#x27;Michael Knyszek' via golang-nuts
orld http server which takes 3MB VSS, I can run > hundreds of them in parallel without issues. > > and yes ulimit -v does not really work for go apps, can't limit its VSS at > all. > > On Thursday, January 14, 2021 at 10:18:03 AM UTC-6 Michael Knyszek wrote: > >> This

[go-nuts] Re: super large virtual memory allocation for simplest golang apps, why?

2021-01-14 Thread &#x27;Michael Knyszek' via golang-nuts
This change came in Go 1.14 as part of an allocator scalability improvement (a specific data structure benefits from having a large memory reservation), but with it also came an effort to mitigate the (potential) negative effects you mention. I think all of the possible problematic cases you li

Re: [go-nuts] Does recent bug in debug.FreeOSMemory also affect automatic background scavenging?

2020-03-11 Thread &#x27;Michael Knyszek' via golang-nuts
Hi John, If you're referring to scavenging that kicked in every 2.5 minutes and returned spans which had been unused for 5 minutes, that's been replaced with an asynchronous, continuous mechanism as of Go 1.13 ( https://github.com/golang/go/issues/30333). This policy is generally more aggressive t

Re: [go-nuts] Change in virtual memory patterns in Go 1.12

2019-05-08 Thread &#x27;Michael Knyszek' via golang-nuts
he problem is now gone, and I > am pretty much confident that the situation is even better than Go 1.11, > which is consistent with the fact that the bug was already there, but > milder because it only applied to large allocations. > > Thanks a lot, > > Rémy > > Le mar.

Re: [go-nuts] Change in virtual memory patterns in Go 1.12

2019-04-16 Thread &#x27;Michael Knyszek' via golang-nuts
y used again. >> Unfortunately, each time we do this counts as a separate "mapping" just to >> track that one flag. These "mappings" are always at least 2MB, but you have >> a large enough virtual address space to reach the max_map_count even then. >> You