GC Warning: Repeated allocation of very large block
hello, on an intensive computation of symbolic calculus running since 10 days approx. on a desktop, i get now this message since yesterday,and in the computation of C12 (still waiting): GC Warning: Repeated allocation of very large block (appr. size 84377600): May lead to memory leak and poor performance it is now at : size 121749504, i suppose one of my symbolic expression get big! is this message appearing when a single scheme variable reach a given size? i wanted to debug and trace because i know this algorithm expand a lot expressions (perheaps too much) sometimes (but it is an NP-problem and exponential, so perheaps stop the program is the only solution and run it on more little data) i do not think there could be a memory leak in a recursive scheme program, at least not like in C ,i'm not allocating memory and i do not to set-car! or set-cdr! that perhaps could loose the GC. Damien C1 = #t C1 = T C2 = T C3 = (B1 ∨ B2) C4 = (B2 ∨ (B1 ∧ B3)) C5 = ((B1 ∧ B3) ∨ (B2 ∧ B3) ∨ (B2 ∧ B4) ∨ (B3 ∧ B4)) C6 = ((B1 ∧ B3 ∧ B5) ∨ (B2 ∧ B3 ∧ B5) ∨ (B2 ∧ B4) ∨ (B3 ∧ B4) ∨ (B4 ∧ B5)) C7 = ((B1 ∧ B3 ∧ B5) ∨ (B2 ∧ B3 ∧ B5) ∨ (B2 ∧ B4 ∧ B6) ∨ (B3 ∧ B4 ∧ B6) ∨ (B4 ∧ B5) ∨ (B5 ∧ B6)) C8 = ((B1 ∧ B3 ∧ B5 ∧ B7) ∨ (B2 ∧ B3 ∧ B5 ∧ B7) ∨ (B2 ∧ B4 ∧ B6) ∨ (B3 ∧ B4 ∧ B6) ∨ (B4 ∧ B5 ∧ B7) ∨ (B5 ∧ B6) ∨ (B6 ∧ B7)) C9 = ((B1 ∧ B3 ∧ B5 ∧ B7) ∨ (B2 ∧ B3 ∧ B5 ∧ B7) ∨ (B2 ∧ B4 ∧ B6 ∧ B8) ∨ (B3 ∧ B4 ∧ B6 ∧ B8) ∨ (B4 ∧ B5 ∧ B7) ∨ (B5 ∧ B6 ∧ B8) ∨ (B6 ∧ B7) ∨ (B7 ∧ B8)) C10 = ((B1 ∧ B3 ∧ B5 ∧ B7 ∧ B9) ∨ (B2 ∧ B3 ∧ B5 ∧ B7 ∧ B9) ∨ (B2 ∧ B4 ∧ B6 ∧ B8) ∨ (B3 ∧ B4 ∧ B6 ∧ B8) ∨ (B4 ∧ B5 ∧ B7 ∧ B9) ∨ (B5 ∧ B6 ∧ B8) ∨ (B6 ∧ B7 ∧ B9) ∨ (B7 ∧ B8) ∨ (B8 ∧ B9)) C11 = ((B1 ∧ B3 ∧ B5 ∧ B7 ∧ B9) ∨ (B10 ∧ B2 ∧ B4 ∧ B6 ∧ B8) ∨ (B10 ∧ B3 ∧ B4 ∧ B6 ∧ B8) ∨ (B10 ∧ B5 ∧ B6 ∧ B8) ∨ (B10 ∧ B7 ∧ B8) ∨ (B10 ∧ B9) ∨ (B2 ∧ B3 ∧ B5 ∧ B7 ∧ B9) ∨ (B4 ∧ B5 ∧ B7 ∧ B9) ∨ (B6 ∧ B7 ∧ B9) ∨ (B8 ∧ B9)) GC Warning: Repeated allocation of very large block (appr. size 84377600): May lead to memory leak and poor performance GC Warning: Repeated allocation of very large block (appr. size 102641664): May lead to memory leak and poor performance GC Warning: Repeated allocation of very large block (appr. size 102653952): May lead to memory leak and poor performance GC Warning: Repeated allocation of very large block (appr. size 102678528): May lead to memory leak and poor performance GC Warning: Repeated allocation of very large block (appr. size 102690816): May lead to memory leak and poor performance GC Warning: Repeated allocation of very large block (appr. size 102739968): May lead to memory leak and poor performance GC Warning: Repeated allocation of very large block (appr. size 102764544): May lead to memory leak and poor performance GC Warning: Repeated allocation of very large block (appr. size 102789120): May lead to memory leak and poor performance GC Warning: Repeated allocation of very large block (appr. size 102825984):
Re: GC Warning: Repeated allocation of very large block
On Mon, 19 Sep 2022, Damien Mattei wrote:
> is this message appearing when a single scheme variable reach a given
> size?
This message is from the bdwgc and not from Guile itself. From their
documentation:
--8<---cut here---start->8---
The garbage collector generates warning messages of the form:
Repeated allocation of very large block ...
May lead to memory leak and poor performance
when it needs to allocate a block at a location that it knows to be referenced
by a false pointer. These false pointers can be either permanent (e.g.
a static integer variable that never changes) or temporary. In the latter
case, the warning is largely spurious, and the block will eventually
be reclaimed normally. In the former case, the program will still run
correctly, but the block will never be reclaimed. Unless the block is intended
to be permanent, the warning indicates a memory leak.
1. Ignore these warnings while you are using GC_DEBUG. Some of the routines
mentioned below don't have debugging equivalents. (Alternatively, write the
missing routines and send them to me.)
2. Replace allocator calls that request large blocks with calls to
`GC_malloc_ignore_off_page` or `GC_malloc_atomic_ignore_off_page`. You may
want to set a breakpoint in `GC_default_warn_proc` to help you identify such
calls. Make sure that a pointer to somewhere near the beginning of the
resulting block is maintained in a (preferably volatile) variable as long
as the block is needed.
3. If the large blocks are allocated with realloc, we suggest instead
allocating them with something like the following. Note that the realloc
size increment should be fairly large (e.g. a factor of 3/2) for this to
exhibit reasonable performance. But we all know we should do that anyway.
void * big_realloc(void *p, size_t new_size) {
size_t old_size = GC_size(p);
void * result;
if (new_size <= 1) return(GC_realloc(p, new_size));
if (new_size <= old_size) return(p);
result = GC_malloc_ignore_off_page(new_size);
if (result == 0) return(0);
memcpy(result,p,old_size);
GC_free(p);
return(result);
}
4. In the unlikely case that even relatively small object (<20 KB)
allocations are triggering these warnings, then your address space contains
lots of "bogus pointers", i.e. values that appear to be pointers but aren't.
Usually this can be solved by using `GC_malloc_atomic` or the routines
in `gc_typed.h` to allocate large pointer-free regions of bitmaps, etc.
Sometimes the problem can be solved with trivial changes of encoding
in certain values. It is possible, to identify the source of the bogus
pointers by building the collector with `-DPRINT_BLACK_LIST`, which will
cause it to print the "bogus pointers", along with their location.
5. If you get only a fixed number of these warnings, you are probably only
introducing a bounded leak by ignoring them. If the data structures being
allocated are intended to be permanent, then it is also safe to ignore them.
The warnings can be turned off by calling `GC_set_warn_proc` with
a procedure that ignores these warnings (e.g. by doing absolutely
nothing).
--8<---cut here---end--->8---
It's my understanding that either A) there's a leak or B) its a false
positive.
> i wanted to debug and trace because i know this algorithm expand a lot
> expressions (perheaps too much) sometimes (but it is an NP-problem and
> exponential, so perheaps stop the program is the only solution and run it
> on more little data)
> i do not think there could be a memory leak in a recursive scheme
> program,
Gotta be careful with a conservative GC I guess. Your memory will get
reclaim at some point, but it's not guaranteed when since any value on
the C stack -- and global variables -- could reference your allocation
by accident. So if by accident there's a static constant value in C
that has the value of an allocation by the GC, it will never reclaim it.
Since it's the Scheme stack and I suppose it's tail call optimized, that
should not impact the GC.
The most important thing would be to check the memory usage of the
program with a tool like `htop'. If there's leak, you will see that the
memory usage percentage keep increasing.
--
Olivier Dion
oldiob.dev
Re: GC Warning: Repeated allocation of very large block
thank you, i think my algorithm is right but can be improved, i have to
work on it,activate tracing to see where is the big expression list and if
it is possible to correct that. There is a main loop in code and the
warning only appear when data is bigger so i do not think of any memory
leak (only have alittle doubt on an hastable which is global variable and
reused:
(when debug-mode (display-nl "Quine-Mc-Cluskey:"))
{minterms-ht <- (make-hash-table)} ;; need to be cleared at each run
(init-hash-table-with-set-and-value minterms-ht minterms #f)
this :
;; the hash table for minterms, better to be a top-level definition,it's
nightmare otherwise...
(declare minterms-ht)
{minterms-ht <- (make-hash-table)} ;; Scheme+ syntax
expression create a new hash table with pointer minterms-ht
this is done at each loop in the for each Cn
and i should clear it before but there is no instruction to do that in the
SRFI 69:
https://srfi.schemers.org/srfi-69/srfi-69.html
???
I will also check the still running memory used by program.
Damien
On Mon, Sep 19, 2022 at 1:44 PM Olivier Dion
wrote:
> On Mon, 19 Sep 2022, Damien Mattei wrote:
> > is this message appearing when a single scheme variable reach a given
> > size?
>
> This message is from the bdwgc and not from Guile itself. From their
> documentation:
> --8<---cut here---start->8---
> The garbage collector generates warning messages of the form:
>
>
> Repeated allocation of very large block ...
> May lead to memory leak and poor performance
>
>
> when it needs to allocate a block at a location that it knows to be
> referenced
> by a false pointer. These false pointers can be either permanent (e.g.
> a static integer variable that never changes) or temporary. In the latter
> case, the warning is largely spurious, and the block will eventually
> be reclaimed normally. In the former case, the program will still run
> correctly, but the block will never be reclaimed. Unless the block is
> intended
> to be permanent, the warning indicates a memory leak.
>
> 1. Ignore these warnings while you are using GC_DEBUG. Some of the
> routines
> mentioned below don't have debugging equivalents. (Alternatively, write
> the
> missing routines and send them to me.)
> 2. Replace allocator calls that request large blocks with calls to
> `GC_malloc_ignore_off_page` or `GC_malloc_atomic_ignore_off_page`. You
> may
> want to set a breakpoint in `GC_default_warn_proc` to help you identify
> such
> calls. Make sure that a pointer to somewhere near the beginning of the
> resulting block is maintained in a (preferably volatile) variable as long
> as the block is needed.
> 3. If the large blocks are allocated with realloc, we suggest instead
> allocating them with something like the following. Note that the realloc
> size increment should be fairly large (e.g. a factor of 3/2) for this to
> exhibit reasonable performance. But we all know we should do that anyway.
>
>
> void * big_realloc(void *p, size_t new_size) {
> size_t old_size = GC_size(p);
> void * result;
> if (new_size <= 1) return(GC_realloc(p, new_size));
> if (new_size <= old_size) return(p);
> result = GC_malloc_ignore_off_page(new_size);
> if (result == 0) return(0);
> memcpy(result,p,old_size);
> GC_free(p);
> return(result);
> }
>
>
> 4. In the unlikely case that even relatively small object (<20 KB)
> allocations are triggering these warnings, then your address space
> contains
> lots of "bogus pointers", i.e. values that appear to be pointers but
> aren't.
> Usually this can be solved by using `GC_malloc_atomic` or the routines
> in `gc_typed.h` to allocate large pointer-free regions of bitmaps, etc.
> Sometimes the problem can be solved with trivial changes of encoding
> in certain values. It is possible, to identify the source of the bogus
> pointers by building the collector with `-DPRINT_BLACK_LIST`, which will
> cause it to print the "bogus pointers", along with their location.
> 5. If you get only a fixed number of these warnings, you are probably
> only
> introducing a bounded leak by ignoring them. If the data structures being
> allocated are intended to be permanent, then it is also safe to ignore
> them.
> The warnings can be turned off by calling `GC_set_warn_proc` with
> a procedure that ignores these warnings (e.g. by doing absolutely
> nothing).
> --8<---cut here---end--->8---
>
> It's my understanding that either A) there's a leak or B) its a false
> positive.
>
> > i wanted to debug and trace because i know this algorithm expand a lot
> > expressions (perheaps too much) sometimes (but it is an NP-problem and
> > exponential, so perheaps stop the program is the only solution and run it
> > on more little data)
> > i do not think there could be a memor
