Alexey wrote at 2021-3-29 06:26 -0700: >понедельник, 29 марта 2021 г. в 15:57:43 UTC+3, Julio Oña: >> It looks like the problem is on celery. >> The mentioned issue is still open, so not sure if it was corrected. >> >> https://manhtai.github.io/posts/memory-leak-in-celery/ > >As I mentioned in my first message, I tried to run >this task(class) via Flask API calls, without Celery. >And results are the same. Flask worker receives the API call and >executes MyClass().run() inside of view. After a few calls >worker size increases to 1Gb of RAM. In production I have 8 workers, > so in idle they will hold 8Gb.
Depending on your system (this works for `glibc` systems), you can instruct the memory management via the envvar `MALLOC_ARENA_MAX` to use a common memory pool (called "arena") for all threads. It is known that this can drastically reduce memory consumption in multi thread systems. -- https://mail.python.org/mailman/listinfo/python-list
