pymalloc manages only small blocks of memory. Large (more than 512 byte) memory blocks are managed by malloc/free.
glibc malloc doesn't return much freed memory to OS. You can try jemalloc instead of glibc. On Ubuntu 20.04, you can try it by: LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so python your-script.py On Fri, May 29, 2020 at 11:07 AM <rmli...@riseup.net> wrote: > > > We just ran into this problem when running our aiootp package's memory > hard password hashing function (https://github.com/rmlibre/aiootp/). The > memory was not being cleared after the function finished running but the > script was still live. We tried making sure everything went out of scope > and deleting things explicitly inside the function, but that didn't > help. We even tried forcing the garbage collector to free up > unreferenced memory with import gc; gc.collect(). But that only had a > small dent in the memory that was being built up. > > The most useful answer online had to do with Python's free lists being > created automatically when very large datasets were being processed > (https://stackoverflow.com/questions/23937189/how-do-i-use-subprocesses-to-force-python-to-release-memory/24126616#24126616). > > > After putting the memory intensive work into a separate process, as the > answer suggested, the memory was finally freed after execution. > > In our case, we wound up passing the result back in a > ``multiprocessing.Manager().List()``. Though, passing your whole numpy > dataset back doesn't seem feasible. I'd recommend doing the necessary > memory intensive work in a separate process and passing only the > necessary conclusions back to the main process. > > Or you could set up a multiprocessing queue to pass control messages to > the spawned (daemon) process so it returns desired results on demand. > > > Here's an excerpt from our code after the fix in case it's helpful. > > > @classmethod > > def passcrypt(cls, password, salt, kb=1024, cpu=1024, hardness=256): > > """ > > The ``passcrypt`` function can be highly memory intensive. > > These resources may not be freed up, & often are not, because of > > python quirks around memory management. This is a huge problem. > > So to force the release of those resources, we run the function > > in another process which is guaranteed to release them. > > """ > > cls._validate_passcrypt_args(kb, cpu, hardness) > > state = Manager().list() > > process = Process( > > target=cls._passcrypt, > > args=(password, salt), > > kwargs=dict(kb=kb, cpu=cpu, hardness=hardness, state=state), > > ) > > process.start() > > process.join() > > return state.pop() > > > > > On 2020-05-28 16:00, python-list-requ...@python.org wrote: > > Send Python-list mailing list submissions to > > python-list@python.org > > > > To subscribe or unsubscribe via the World Wide Web, visit > > https://mail.python.org/mailman/listinfo/python-list > > or, via email, send a message with subject or body 'help' to > > python-list-requ...@python.org > > > > You can reach the person managing the list at > > python-list-ow...@python.org > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of Python-list digest..." > > > > Today's Topics: > > > > 1. Re: Custom logging function (zljubi...@gmail.com) > > 2. Ram memory not freed after executing python script on ubuntu > > system (Rahul Gupta) > > 3. Re: Ram memory not freed after executing python script on > > ubuntu system (Chris Angelico) > > 4. Re: Custom logging function (Peter Otten) > > 5. Re: Elegant hack or gross hack? TextWrapper and escape codes > > (Peter Otten) > > 6. Re: Elegant hack or gross hack? TextWrapper and escape codes > > (Chris Angelico) > > 7. Re: Elegant hack or gross hack? TextWrapper and escape codes > > (Peter Otten) > > 8. Re: Ram memory not freed after executing python script on > > ubuntu system (Rahul Gupta) > > 9. Re: Ram memory not freed after executing python script on > > ubuntu system (Chris Angelico) > > 10. Re: Behaviour of os.path.join (BlindAnagram) > > 11. Constructing mime image attachment (Joseph L. Casale) > > 12. Re: Behaviour of os.path.join (Eryk Sun) > > 13. Re: Behaviour of os.path.join (Eryk Sun) > > 14. Re: Behaviour of os.path.join (BlindAnagram) > > 15. Re: Behaviour of os.path.join (Eryk Sun) > -- > https://mail.python.org/mailman/listinfo/python-list -- Inada Naoki <songofaca...@gmail.com> -- https://mail.python.org/mailman/listinfo/python-list
