Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

Mon, 26 Aug 2019 13:42:12 -0700

You might want to try 'perf mem' to report the access delays - it may be contention on the memory controller as well.

Thinking about it again, I wouldn't expect a large jump if things were fair - for example, if at 100 they all fit in the cache, at 110, some are still in the cache, but some operations are slower, etc. so I would expect a jump but not as large as you see.

Still, most linux context switches are 3-4 us, and you are talking about 300 ns, so you're still doing pretty good, and at approx 40 ns, there are so many aspects that come into play, i'm not sure you or anyone has the time to figure out - maybe the HFT guys are interested...

Like I said, on my OSX machine the times are very similar with both approaches, so it is OS dependent, and probably OS and hardware configuration dependent - so I think I've probably reached the end of being able to help.

And finally, it probably doesn't matter at all - if the Go routine is doing anything of value, 300 ns is probably an insignificant cost.


-----Original Message-----
From: changkun
Sent: Aug 26, 2019 3:15 PM
To: golang-nuts
Subject: Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

Did I do anything wrong, the cache hint ratio decrease linearly, is it an expected result? I thought the cache hint ratio would have a significant drop:

chart.png

Raw data:

#goroutinescache-referencescache-misseshint/(hint+miss)
2400697103572176416860.9753175194
3200798160789541697840.9364451004
336013879724731484156780.9033996208
360018245410622721663550.8701934506
400020537794013935865010.8391795437
480018856222754618728990.8032486268

On Monday, August 26, 2019 at 9:26:05 PM UTC+2, Robert Engels wrote:
You can run the process under 'perf' and monitor the cpu cache hit/miss ratio.

-----Original Message-----
From: changkun
Sent: Aug 26, 2019 2:23 PM
To: golang-nuts
Subject: Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

Your cache theory is very reasonable, but this was clear in the beginning post:  "before or after the massive increase, performance drops linearly".
Your hypothesis is reasonable, but how can you prove your hypothesis? By host machine cache usage monitoring? 
Matching of a concept is still not persuasive.

On Monday, August 26, 2019 at 8:08:27 PM UTC+2, Robert Engels wrote:
Which is what I would expect - once the number of routines exhaust the cache, it will take the next level (or never since its main memory) to see an massive increase in time. 4800 is 30% slower than 3600 - so it is increasing linearly with the number of Go routines.


-----Original Message-----
From: changkun
Sent: Aug 26, 2019 11:49 AM
To: golang-nuts
Subject: Re: [go-nuts] sync.Mutex encounter large performance drop when goroutine contention more than 3400

According to your formula let's sample three points:

2400 goroutines: 2.508s/(50000000*2400) = 2.09 × 10^-11 s
3600 goroutines: 12.219s/(50000000*3600) = 6.78833333 × 10-11 seconds
4800 goroutines: 16.020s/(50000000*4800) = 6.67500 × 10^-11 s

One can observe that 3600 and 4800 mostly equal to each other, but they both three times slower than 2400.

goos: linux
goarch: amd64
BenchmarkMutexWrite/goroutines-2400-8           50000000                46.5 ns/op
PASS
ok      _/home/changkun/dev/tests       2.508s

goos: linux
goarch: amd64
BenchmarkMutexWrite/goroutines-3600-8           50000000               240 ns/op
PASS
ok      _/home/changkun/dev/tests       12.219s

goos: linux
goarch: amd64
BenchmarkMutexWrite/goroutines-4800-8           50000000               317 ns/op
PASS
ok      _/home/changkun/dev/tests       16.020s


--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/6dd6ec66-b0cc-4c8e-a341-94bff187214f%40googlegroups.com.









-- 

You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/495e22e8-4a5f-4a1d-88f8-59ff2b0a4006%40googlegroups.com.












-- 

You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/0c2b4dc1-3774-4fe6-9cd6-a841dd7265c4%40googlegroups.com.










-- 

You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/726802398.7022.1566852068570%40wamui-albus.atl.sa.earthlink.net.























					Reply via email to