I checked the implementation of bytecectors and SRFI-4 in Guile and they are definitely not scanned for pointers. But I would say hacking them is not a good general solution for this problem. They are good and natural data structures for large arrays of numerical data that are standard signed/unsigned integers of various fixed sizes and IEEE floating point numbers, or structures/unions of these types. Using them for things other than that or strings/byte-arrays could be error prone, messy, and performance poor.
Freja Nordsiek On July 16, 2017 11:18:18 AM GMT+02:00, Marko Rauhamaa <ma...@pacujo.net> wrote: >Freja Nordsiek <fnord...@gmail.com>: > >> If I was to hazard a reason for why Guile gets very slow when loading >> 20 GB or more (may or may not be related to it being buggy and >> crashy), my guesses would be a lot of the data when loaded into Guile >> was allocated such that the GC scans it for pointers (using >> scm_gc_malloc instead of scm_gc_malloc_pointerless) which would >vastly >> increase the amount of memory the GC needs to scan every time it >runs. > >Good point! > >If you didn't to any C programming, what kind of native Guile data >structures are good for such large random-access storage? At least >arrays haven't specifically been documented for such GC optimization: ><URL: https://www.gnu.org/software/guile/manual/html_node/Arrays.htm >l#Arrays>. > >Maybe bytevectors would do: <URL: https://www.gnu.org/software/guile/m >anual/html_node/Bytevectors.html#Bytevectors>. > >Of course constantly encoding to and decoding from a bytevector using >scheme code might be very slow without the help of some binary bulk >formatting facilities for the data records. > > >Marko
