Can you post the svg versions of those profiles? Also, I recommend the execution trace profiler for this job, it'll show you a lot of detail about how the runtime is interacting with your program.
On Wed, 17 May 2017, 17:29 Manish Rai Jain <manishrj...@gmail.com> wrote: > > libaio sounds good on paper, but at least on GNU/Linux it's all in user > space. > > I see. That makes sense. Reading a bit more, Linux native I/O sounds like > it does exactly what we expect, i.e. save OS threads, and push this to > kernel: http://man7.org/linux/man-pages/man2/io_submit.2.html > But, I suppose this can't be part of Go, because it's not portable. Is my > understanding correct? > > Also, any explanations about why GOMAXPROCS causes throughput to increase, > if new OS threads are being spawned by blocked goroutines anyway? I thought > I understood it before but now I don't. > > Dave, profiler doesn't show any issues with the code itself. It's just > blocked waiting on syscalls. > > $ go tool pprof randread /tmp/profile398062565/cpu.pprof > ~/go/src/github.com/dgraph-io/badger-bench/randread > Entering interactive mode (type "help" for commands) > (pprof) top > 19.48s of 19.76s total (98.58%) > Dropped 27 nodes (cum <= 0.10s) > flat flat% sum% cum cum% > 19.34s 97.87% 97.87% 19.52s 98.79% syscall.Syscall6 > 0.07s 0.35% 98.23% 0.11s 0.56% runtime.exitsyscall > 0.03s 0.15% 98.38% 19.56s 98.99% os.(*File).ReadAt > 0.02s 0.1% 98.48% 0.10s 0.51% math/rand.(*Rand).Intn > 0.01s 0.051% 98.53% 19.70s 99.70% main.Conc2.func1 > 0.01s 0.051% 98.58% 19.53s 98.84% syscall.Pread > 0 0% 98.58% 0.13s 0.66% main.getIndices > 0 0% 98.58% 19.53s 98.84% os.(*File).pread > 0 0% 98.58% 19.70s 99.70% runtime.goexit > (pprof) > > > $ go tool pprof randread /tmp/profile192709852/block.pprof > ~/go/src/github.com/dgraph-io/badger-bench/randread > Entering interactive mode (type "help" for commands) > (pprof) top > 58.48s of 58.48s total ( 100%) > Dropped 8 nodes (cum <= 0.29s) > flat flat% sum% cum cum% > 58.48s 100% 100% 58.48s 100% sync.(*WaitGroup).Wait > 0 0% 100% 58.48s 100% main.Conc2 > 0 0% 100% 58.48s 100% main.main > 0 0% 100% 58.48s 100% runtime.goexit > 0 0% 100% 58.48s 100% runtime.main > (pprof) > > > On Wed, May 17, 2017 at 3:25 PM, Dave Cheney <d...@cheney.net> wrote: > >> Rather than guessing what is going on, I think it's time to break out the >> profiling tools Manish. >> >> On Wed, 17 May 2017, 15:23 David Klempner <klemp...@google.com> wrote: >> >>> >>> On May 16, 2017 22:03, "Ian Lance Taylor" <i...@golang.org> wrote: >>> >>> On Tue, May 16, 2017 at 9:26 PM, Manish Rai Jain <manishrj...@gmail.com> >>> wrote: >>> >> The runtime will spawn a new thread to replace the one that is >>> blocked. >>> > >>> > Realized that after writing my last mail. And that actually explains >>> some of >>> > the other crashes we saw, about "too many threads", if we run tens of >>> > thousands of goroutines to do these reads, one goroutine per read. >>> > >>> > It is obviously lot more expensive to spawn a new OS thread. It seems >>> like >>> > this exact same problem was already solved for network via netpoller >>> > (https://morsmachine.dk/netpoller). Blocking OS threads for disk >>> reads made >>> > sense for HDDs, which could only do 200 IOPS; for SSDs we'd need a >>> solution >>> > based on async I/O. >>> >>> Note that in the upcoming Go 1.9 release we now use the netpoller for >>> the os package as well. However, it's not as effective as one would >>> hope, because on GNU/Linux you can't use epoll for disk files. >>> >>> >>> There's a not very well documented API to make AIO completions kick an >>> eventfd. >>> >>> It >>> mainly helps with pipes. >>> >>> >>> Ian >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "golang-nuts" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to golang-nuts+unsubscr...@googlegroups.com. >>> >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >>> > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
