Hey folks,
Just wanted to update the status of this.
During Gophercon, I happened to meet Russ Cox and asked him the same
question. If File::Read blocks goroutines, which then spawn new OS threads,
in a long running job, there should be plenty of OS threads created
already, so the random read thr
On Fri, May 19, 2017 at 3:26 AM, Manish Rai Jain wrote:
>
>> It's not obvious to me that io_submit would be a win for normal
> programs, but if anybody wants to try it out and see that would be
> great.
>
> Yeah, my hunch is that the cost of threads context switching is going to be
> a hindrance t
Sorry for the delay in replying. Got busy with a presentation at Go meetup.
> I agree with Dave that looking at the execution tracer is likely to help.
I tried to run it, but nothing renders on my chrome (running on Arch
Linux). Typical about:tracing works, but this doesn't. And there isn't much
On Wed, May 17, 2017 at 12:29 AM, Manish Rai Jain wrote:
>
>> libaio sounds good on paper, but at least on GNU/Linux it's all in user
>> space.
>
> I see. That makes sense. Reading a bit more, Linux native I/O sounds like it
> does exactly what we expect, i.e. save OS threads, and push this to ker
On May 16, 2017 22:03, "Ian Lance Taylor" wrote:
On Tue, May 16, 2017 at 9:26 PM, Manish Rai Jain
wrote:
>> The runtime will spawn a new thread to replace the one that is blocked.
>
> Realized that after writing my last mail. And that actually explains some
of
> the other crashes we saw, about "
Can you post the svg versions of those profiles?
Also, I recommend the execution trace profiler for this job, it'll show you
a lot of detail about how the runtime is interacting with your program.
On Wed, 17 May 2017, 17:29 Manish Rai Jain wrote:
> > libaio sounds good on paper, but at least on
> libaio sounds good on paper, but at least on GNU/Linux it's all in user
space.
I see. That makes sense. Reading a bit more, Linux native I/O sounds like
it does exactly what we expect, i.e. save OS threads, and push this to
kernel: http://man7.org/linux/man-pages/man2/io_submit.2.html
But, I sup
Rather than guessing what is going on, I think it's time to break out the
profiling tools Manish.
On Wed, 17 May 2017, 15:23 David Klempner wrote:
>
> On May 16, 2017 22:03, "Ian Lance Taylor" wrote:
>
> On Tue, May 16, 2017 at 9:26 PM, Manish Rai Jain
> wrote:
> >> The runtime will spawn a ne
On Tue, May 16, 2017 at 9:26 PM, Manish Rai Jain wrote:
>> The runtime will spawn a new thread to replace the one that is blocked.
>
> Realized that after writing my last mail. And that actually explains some of
> the other crashes we saw, about "too many threads", if we run tens of
> thousands of
On Tue, May 16, 2017 at 8:04 PM, Manish Rai Jain wrote:
>
> Ideally, the disk reads could be happening via libaio, causing the OS
> threads to not block, so all goroutines can make progress, increasing the
> number of read requests that can be made concurrently. This would then also
> ensure that
> The runtime will spawn a new thread to replace the one that is blocked.
Realized that after writing my last mail. And that actually explains some
of the other crashes we saw, about "too many threads", if we run tens of
thousands of goroutines to do these reads, one goroutine per read.
It is obv
> So, if an OS thread is blocked, no goroutines can be scheduled on this
thread, therefore even pure CPU operations can't be run.
The runtime will spawn a new thread to replace the one that is blocked.
On Wednesday, 17 May 2017 13:05:49 UTC+10, Manish Rai Jain wrote:
>
> On further thought about
On further thought about GOMAXPROCS, and its impact on throughput:
A file::pread would block the OS thread. Go runs one OS thread per core.
So, if an OS thread is blocked, no goroutines can be scheduled on this
thread, therefore even pure CPU operations can't be run. This would lead to
core wastag
So, I fixed the rand and removed the atomics usage (link in my original
post).
Setting GOMAXPROCS definitely helped a lot. And now it seems to make sense,
because (the following command in) fio spawns 16 threads; and GOMAXPROCS
would do the same thing. However, the numbers are still quite a bit of
On Tue, May 16, 2017 at 4:59 AM, Manish Rai Jain wrote:
>
> 3 is slower than 2 (of course). But, 2 is never able to achieve the IOPS
> that Fio can achieve. I've tried other things, to no luck. What I notice is
> that Go and Fio are close to each other as long as number of Goroutines is
> <= numbe
15 matches
Mail list logo
