On Mon, Mar 29, 2021 at 2:32 PM Alexey <zen.supag...@gmail.com> wrote:
> понедельник, 29 марта 2021 г. в 15:57:43 UTC+3, Julio Oña: > > It looks like the problem is on celery. > > The mentioned issue is still open, so not sure if it was corrected. > > > > https://manhtai.github.io/posts/memory-leak-in-celery/ > > As I mentioned in my first message, I tried to run > this task(class) via Flask API calls, without Celery. > And results are the same. Flask worker receives the API call and > executes MyClass().run() inside of view. After a few calls > worker size increases to 1Gb of RAM. In production I have 8 workers, > so in idle they will hold 8Gb. > Memory statistics in modern OSs are incredibly complex to reason about. It's /possible/ that, while it certainly looks bad in the monitoring tools, there actually isn't a problem here. Some questions here to help understand more: 1. Do you have any actual problems caused by running 8 celery workers (beyond high memory reports)? What are they? 2. Can you try a test with 16 or 32 active workers (i.e. number of workers=2x available memory in GB), do they all still end up with 1gb usage? or do you get any other memory-related issues running this? Thanks Steve > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
