On 04/15/2016 05:25 AM, cshin...@gmail.com wrote:
I have written an application with flask and uses celery for a long running
task. While load testing I noticed that the celery tasks are not releasing
memory even after completing the task. So I googled and found this group
discussion..
https://groups.google.com/forum/#!topic/celery-users/jVc3I3kPtlw
In that discussion it says, thats how python works.
Also the article at
https://hbfs.wordpress.com/2013/01/08/python-memory-management-part-ii/ says
"But from the OS's perspective, your program's size is the total (maximum) memory
allocated to Python. Since Python returns memory to the OS on the heap (that allocates
other objects than small objects) only on Windows, if you run on Linux, you can only see
the total memory used by your program increase."
And I use Linux. So I wrote the below script to verify it.
import gc
def memory_usage_psutil():
# return the memory usage in MB
import resource
print 'Memory usage: %s (MB)' %
(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1000.0)
def fileopen(fname):
memory_usage_psutil()# 10 MB
f = open(fname)
memory_usage_psutil()# 10 MB
content = f.read()
memory_usage_psutil()# 14 MB
def fun(fname):
memory_usage_psutil() # 10 MB
fileopen(fname)
gc.collect()
memory_usage_psutil() # 14 MB
import sys
from time import sleep
if __name__ == '__main__':
fun(sys.argv[1])
for _ in range(60):
gc.collect()
memory_usage_psutil()#14 MB ...
sleep(1)
The input was a 4MB file. Even after returning from the 'fileopen' function the
4MB memory was not released. I checked htop output while the loop was running,
the resident memory stays at 14MB. So unless the process is stopped the memory
stays with it.
So if the celery worker is not killed after its task is finished it is going to
keep the memory for itself. I know I can use **max_tasks_per_child** config
value to kill the process and spawn a new one. **Is there any other way to
return the memory to OS from a python process?.**
With situations like this, I normally just fork and do the mem intensive
work in the child and then kill it off when done. Might be able to use
a thread instead of a fork. But not sure how well all that would work
with celery.
--Sam
--
https://mail.python.org/mailman/listinfo/python-list
