Hi all,
I realized that when working with large input arguments in a task, this
causes problems of high memory consumption. Let me give you more details.
I developed a custom module, called get_artifacts, which receives as an
input argument (among others), a list of artifacts to be downloaded in each
host.
This list of artifacts is list of json objects containing a lot of
properties for each artifact (like size, url, permissions, path, etc.)
which means it can weigh around 60MB.
When I execute my playbook, I have noticed that memory is increased every
time that a host finish the custom task.
Detailed examples:
- My set up:
>>ansible --version
ansible [core 2.11.12]
config file = memory_consumption/ansible/ansible.cfg
configured module search path =
[u'memory_consumption/ansible/role_deploy/library']
ansible python module location =
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ansible
ansible collection location =
/Users/user/.ansible/collections:/usr/share/ansible/collections
executable location =
/Library/Frameworks/Python.framework/Versions/2.7/bin/ansible
python version = 2.7.16 (v2.7.16:413a49145e, Mar 2 2019, 14:32:10) [GCC
4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
jinja version = 2.10.1
libyaml = False
>>cat ansible.cfg
[defaults]
library = ./role_deploy/library
module_utils = ./role_deploy/library/module_utils
callback_whitelist = profile_memory
forks = 1
- When the task starts:
TASK [role_deploy : Download the artifacts]
***********************************************************************************
---------->[started task TASK: role_deploy : Download the artifacts]
Filename:
memory_consumption/ansible/role_deploy/callback_plugins/profile_memory.py
Line # Mem usage Increment Line Contents
================================================
18 343.113 MiB 343.113 MiB @profile(precision=3)
19 def v2_playbook_on_task_start(self,
task, is_conditional):
20 343.113 MiB 0.000 MiB
self._display.display("---------->[started task %s]" % (task),
color=C.COLOR_WARN)
- When the first host starts and finishes the task:
---------------------------------->[started task TASK: role_deploy :
Download the artifacts on host 18.203.162.10]
Filename:
memory_consumption/ansible/role_deploy/callback_plugins/profile_memory.py
Line # Mem usage Increment Line Contents
================================================
22 343.113 MiB 343.113 MiB @profile(precision=3)
23 def v2_runner_on_start(self, host,
task):
24 343.113 MiB 0.000 MiB
self._display.display("---------------------------------->[started task %s
on host %s]" % (task, host), color=C.COLOR_WARN)
---------------------------------->[finished result {u'total_failed': 0,
u'total_ok': 3238, '_ansible_no_log': False, u'deleted': False, u'changed':
True, u'response_action': {u'deleted': [], u'failed': [], u'changed': 6,
u'ok': 3238}, u'total_changed': 6, u'total_deleted': 0, u'invocation': {},
u'total': 3244}]
Filename:
memory_consumption/ansible/role_deploy/callback_plugins/profile_memory.py
Line # Mem usage Increment Line Contents
================================================
26 420.051 MiB 420.051 MiB @profile(precision=3)
27 def v2_runner_on_ok(self, result):
28 420.051 MiB 0.000 MiB
self._display.display("---------------------------------->[finished result
%s]" % (result._result), color=C.COLOR_WARN)
- When the second host starts and finishes the task:
---------------------------------->[started task TASK: role_deploy :
Download the artifacts on host 54.216.154.16]
Filename:
memory_consumption/ansible/role_deploy/callback_plugins/profile_memory.py
Line # Mem usage Increment Line Contents
================================================
22 420.051 MiB 420.051 MiB @profile(precision=3)
23 def v2_runner_on_start(self, host,
task):
24 420.051 MiB 0.000 MiB
self._display.display("---------------------------------->[started
task %s on host %s]" % (task, host), color=C.COLOR_WARN)
---------------------------------->[finished result {u'total_failed': 0,
u'total_ok': 3238, '_ansible_no_log': False, u'deleted': False, u'changed':
True, u'response_action': {u'deleted': [], u'failed': [], u'changed': 6,
u'ok': 3238}, u'total_changed': 6, u'total_deleted': 0, u'invocation': {},
u'total': 3244}]
Filename:
memory_consumption/ansible/role_deploy/callback_plugins/profile_memory.py
Line # Mem usage Increment Line Contents
================================================
26 480.516 MiB 480.516 MiB @profile(precision=3)
27 def v2_runner_on_ok(self, result):
28 480.516 MiB 0.000 MiB
self._display.display("---------------------------------->[finished result
%s]" % (result._result), color=C.COLOR_WARN)
- When the third host starts and finishes the task:
---------------------------------->[started task TASK: role_deploy :
Download the artifacts on host 18.202.233.9]
Filename:
memory_consumption/ansible/role_deploy/callback_plugins/profile_memory.py
Line # Mem usage Increment Line Contents
================================================
22 480.516 MiB 480.516 MiB @profile(precision=3)
23 def v2_runner_on_start(self, host,
task):
24 480.516 MiB 0.000 MiB
self._display.display("---------------------------------->[started
task %s on host %s]" % (task, host), color=C.COLOR_WARN)
---------------------------------->[finished result {u'total_failed': 0,
u'total_ok': 3238, '_ansible_no_log': False, u'deleted': False, u'changed':
True, u'response_action': {u'deleted': [], u'failed': [], u'changed': 6,
u'ok': 3238}, u'total_changed': 6, u'total_deleted': 0, u'invocation': {},
u'total': 3244}]
Filename:
memory_consumption/ansible/role_deploy/callback_plugins/profile_memory.py
Line # Mem usage Increment Line Contents
================================================
26 537.219 MiB 537.219 MiB @profile(precision=3)
27 def v2_runner_on_ok(self, result):
28 537.219 MiB 0.000 MiB
self._display.display("---------------------------------->[finished result
%s]" % (result._result), color=C.COLOR_WARN)
As you can see, It can be a real problem when managing a lot of nodes
because the master is going into OOM exception... just as it is my case.
Going into detail, I realized that ansible is keeping/storing the input
args of the task in two places in the each host result: in the
*_result.invocation.module_args* key and in the *_task_fields.args*.
I found that the invocation data can be forced to be empty in my custom
module using something like:
...
res_args.update({"invocation": dict()})
module.exit_json(**res_args)
But I haven't found any solution to force ansible not to storing the input
args under the *_task_fields.args *key.
Is there any way to do it? How can ansible work more efficiently when
working with large input parameters?
Thank you in advance!!
PD: Is my first message in this community so please be benevolent. :)
--
You received this message because you are subscribed to the Google Groups
"Ansible Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to ansible-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-devel/623711b8-9d8a-4520-b2cb-ecc62825a508n%40googlegroups.com.
