What (i think) is happening is that the GC runs concurrently with your
program. After the call f(8) you have 2.2 gigabyte memory allocated. When
you start the call to f(9) the GC starts a mark phase, but concurrently to
that your program allocates 2.2 gigabyte more space. And before it manages
to complete that allocation, you need around 4.5 gigabyte on heap space.
What confounds is that the program terminates after the call to f(9), so we
don't get to see the heap in its final state. We can, however, alter your
main function to manually collect before exiting since this will block the
exit as well:
func main() {
debug.SetGCPercent(10)
f(8)
f(9)
runtime.GC()
}
And now the GODEBUG=gctrace=1 output is:
[jlouis@lady-of-pain z]$ GODEBUG=gctrace=1 ./z
gc 1 @0.000s 17%: 0.054+0+0.28 ms clock, 0.16+0/0/0+0.86 ms cpu, 0->0->0
MB, 0 MB goal, 8 P (forced)
gc 2 @0.313s 0%: 0.060+247+0.050 ms clock, 0.24+0/0.013/0.11+0.20 ms cpu,
2288->2288->2288 MB, 2289 MB goal, 8 P
8
gc 3 @0.855s 0%: 0.009+247+0.047 ms clock, 0.074+0/0.030/247+0.37 ms cpu,
4577->4577->2288 MB, 4578 MB goal, 8 P
9
gc 4 @1.103s 0%: 0.010+0+0.25 ms clock, 0.086+0/0.030/247+2.0 ms cpu,
2288->2288->0 MB, 2288 MB goal, 8 P (forced)
which shows gc #3 has a heap of 4.6 gigabyte it manages to shrink down one
step to 2.28 gigabyte. Our own forced gc is #4 which then eliminates the
heap entirely.
The "problem" is that when you allocate big chunks like these, the GC
doesn't really stand a chance and can't work on your program quickly enough
to free up space before your program requests more. The GC is simply being
overloaded because there are only two allocations, and both are very large.
To verify, you can try inserting a runtime.GC() between the f(8) and f(9)
calls which block until the GC is done. This should make your program able
to run in 2.2 gigs of memory space.
The solution is to allocate less data, get more RAM, or allocate what you
need up front where it is less likely to get you into trouble.
On Mon, May 1, 2017 at 4:08 PM Serhat Sevki Dincer <jfcga...@gmail.com>
wrote:
> <jesper.louis.ander...@gmail.com> yazdı:
>
> Your program completes without any trouble on my machine, but do note if
> we enter the following in an emacs scratch buffer and hit C-j to use it as
> a glorified calculator:
>
> (/ (* 3e8 8) (* 1024 1024 1024))
> 2.2351741790771484
>
> If each element in your array takes up 8 bytes, which is the case for a
> 64bit integer, then you need at least 2.2 gigabytes of memory to allocate
> the array.
>
> Thanks for the arithmetic help, yes I am allocating 2+ gb ram
>
> Looking at your output, it is clear your operating system is not allowing
> you to malloc some contiguous piece of space for this.
>
> My os IS allowing me to allocate the ram. Note that the "2nd" call fails.
>
> There are a couple of possible things to look out for:
>
> * 32bit systems can often allocate a constrained amount of memory in one
> go.
> * You may be limited articifically by the operating system, either through
> cgroups or the 'ulimit' facility, though the latter has a rather peculiar
> way of working on linux.
> * You may not have enough memory in your system.
>
> In practice, a garbage collector will usually seek to allocate more space
> than what is needed. If we run your program with
>
> Mine is a laptop with 4 gb ram & Ubuntu 64-bit 16.04
>
> [jlouis@lady-of-pain z]$ GODEBUG=gctrace=1 ./z
> gc 1 @0.000s 16%: 0.003+0+0.34 ms clock, 0.014+0/0/0+1.3 ms cpu, 0->0->0
> MB, 0 MB goal, 8 P (forced)
> gc 2 @0.020s 0%: 0.066+256+0.048 ms clock, 0.26+0/0.038/0.097+0.19 ms cpu,
> 2288->2288->2288 MB, 2289 MB goal, 8 P
> 8
> 9
>
> we can see the garbage collector collects twice. The first one is probably
> forced by the SetGCPercentage call. The second because of the rather large
> allocation that is happening. And the amount of needed memory more or less
> coincides with the 3e8 * 8 calculation above.
>
> If I have just the 1st call, there is no crash. After the 1st call
> returns, 2nd call should be able to succeed.
> So what's happening?
>
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
