Let me share some example and what I wanted to achieve:
*raw json data *
(one sample for each ipData notation)
{
"protVersion": 1,
"policyObject": {
"networkPolicyObject": [
{
"gid": "00000000-0000-0000-0000-180388630238",
"name": "Network Object dummy1",
"lastUpdateTime": "2013-09-10T09:55:32.743Z",
"parentGID": "00000000-0000-0000-0000-000000000000",
"type": "NetworkPolicyObject",
"comment": "dns",
"nodeGID": "00000000-0000-0000-0000-000000000001",
"isProperty": false,
"isGroup": false,
*"ipData": "131.117.1.125"*
},
{
"gid": "00000000-0000-0000-0012-206297055671",
"name": "Network Object dummy2",
"lastUpdateTime": "2020-06-24T06:08:08.82Z",
"parentGID": "00000000-0000-0000-0000-000000000000",
"type": "NetworkPolicyObject",
"comment": "sample comment",
"nodeGID": "00000000-0000-0000-0000-000000000001",
"isProperty": false,
"isGroup": false,
* "ipData": [ "10.7.190.8", "10.7.48.65",
"10.7.49.41", "10.7.50.42", "10.7.190.73",
"10.7.45.188", "10.7.45.189", "10.7.190.144" ]*
},
{
"gid": "00000000-0000-0000-0000-180388630208",
"name": "Network Object dummy3",
"lastUpdateTime": "2013-06-10T07:34:40.56Z",
"parentGID": "00000000-0000-0000-0000-000000000000",
"type": "NetworkPolicyObject",
"comment": "logging",
"nodeGID": "00000000-0000-0000-0000-000000000001",
"isProperty": false,
"isGroup": false,
* "ipData": "10.1.57.52"*
},
{
"gid": "00000000-0000-0000-0001-765231559040",
"name": "Network Object dummy4",
"lastUpdateTime": "2010-10-29T11:58:06.496Z",
"parentGID": "00000000-0000-0000-0000-000000000000",
"type": "NetworkPolicyObject",
"comment": "nms",
"nodeGID": "00000000-0000-0000-0000-000000000001",
"isProperty": false,
"isGroup": false,
*"ipData": "10.194.8.0/255.255.255.0"*
},
{
"gid": "00000000-0000-0000-0010-758893077356",
"name": "Network Object dummy5",
"lastUpdateTime": "2016-10-12T12:21:40.51Z",
"parentGID": "00000000-0000-0000-0000-000000000000",
"type": "NetworkPolicyObject",
"comment": "abc range",
"nodeGID": "00000000-0000-0000-0000-000000000001",
"isProperty": false,
"subType": "NR",
"isGroup": false,
*"ipData": "131.117.80.0-131.117.127.255"*
}
]
}
}
target structure
now, as a first step I build a map like this
{
"131.117.1.125": {
"gid": "00000000-0000-0000-0000-180388630238",
"name": "Network Object dummy1",
"lastUpdateTime": "2013-09-10T09:55:32.743Z",
"parentGID": "00000000-0000-0000-0000-000000000000",
"type": "NetworkPolicyObject",
"comment": "dns",
"nodeGID": "00000000-0000-0000-0000-000000000001",
"isProperty": false,
"isGroup": false,
"ipData": "131.117.1.125"
}
},
"131.117.1.126": ...
or better said in memory not using dictionary:
{
"131.117.1.125" : networkobject_class_ref1,
"131.117.1.126" : networkobject_class_ref2
}
...
{
"ranges": [
{
"10.194.8.0/255.255.255.0": {
"gid": "00000000-0000-0000-0001-765231559040",
"name": "Network Object dummy4",
"lastUpdateTime": "2010-10-29T11:58:06.496Z",
"parentGID": "00000000-0000-0000-0000-000000000000",
"type": "NetworkPolicyObject",
"comment": "nms",
"nodeGID": "00000000-0000-0000-0000-000000000001",
"isProperty": false,
"isGroup": false,
"ipData": "10.194.8.0/255.255.255.0"
}
},
{
"131.117.80.0-131.117.127.255": {
"gid": "00000000-0000-0000-0010-758893077356",
"name": "Network Object dummy5",
"lastUpdateTime": "2016-10-12T12:21:40.51Z",
"parentGID": "00000000-0000-0000-0000-000000000000",
"type": "NetworkPolicyObject",
"comment": "abc range",
"nodeGID": "00000000-0000-0000-0000-000000000001",
"isProperty": false,
"subType": "NR",
"isGroup": false,
"ipData": "131.117.80.0-131.117.127.255"
}
}
]
}
and of course a lookup- Method to check wheather there is an entry that
matches this given ip-address:
lookup.yml (using custom module/plugin 'csm_network_objects')
- name: load data in memory
csm_network_objects:
loadJSONfull: "{{ content_json_networkobjects }}"
- name: do ip lookup first time
csm_network_objects:
lookupIP: "131.117.1.125"
- name: do ip lookup second time
csm_network_objects:
lookupIP: "131.117.1.125"
thlanzer schrieb am Montag, 28. September 2020 um 13:24:35 UTC+2:
> Hello Abhijit
>
> Thank you for the reply! And sorry if my original post was not clear
> enough.
> I thought that it doesn't matter to put all the code or examples in the
> post because I would to solve an issue in general that custom python
> classes get constructed on every task.
>
> I assumed that there should be a possibility as a plugin not a module
> because they are used on the ansible controller not the target host..
> During the execution of play that should be one process and I really do
> not understand why I class should not live for the time the playbook runs
> but maybe my ansible knowledge is not deep enough.
>
> Yes, the usecase is bit special. Is is single target playbook with CIsco
> Security Manager (firewall mgmg) application as target using xml files in
> request and response.
> So basically I do:
> 1.) parsexml2json using Filter
> 2.) load the a huge json string into class (as there is only a GET-ALL
> method)
> 2.1.) parse the json file into nice python classes to easily use with an
> IDE (i.e. Visual Studio Code)
> 2.2.) take from class one attribute (ip-address) to build a
> hashmap/dictionary { key = ipAdress, value = classObject }
> 3.) lookup one specific ipAddress according to given variables and
> retreive one single entry..
> (this alreay works using a custom ansible modul, If I pass the fullJSON
> string arguement and a lookup(ip) arguement it works like a charm)
>
> More or less I had the same ideas as you suggested. Best for performance
> would be database but then I have another dependency.
> As we are planing to use a tower cluster it will not be single machine so
> local files , local databases, local caching or seperate process etc. can't
> be used.
> Anyway, this makes it just a bit too complicated and I would use a shared
> database only as last option if performance is not good enough.
>
> If there is really no other way I will go with your suggestion to store
> one fact for all parsed items and to lookup on that dictionary.
> But this is kinda very limited and works not for complex use cases where
> we have not an ip-adress as key but a netmask instead.
> As you might can imagine some firewall rules do not use a specific
> ip-adress but a whole range (netmask).
>
> Anyway, I am still learning Ansible from day to day and I am happy to get
> further insights how such problems can be solved!
>
> Best regards,
> Thomas
>
> a...@2ndquadrant.com schrieb am Montag, 28. September 2020 um 07:03:45
> UTC+2:
>
>> On Sun, 27 Sep 2020 at 05:15, thlanzer <th....@gmx.ch> wrote:
>> >
>> > I would like to parse specifc JSON files, create objects from them and
>> store them in memory for further lookup
>>
>> Hi Thomas.
>>
>> You obviously have some very specific use-case in mind, but I was not
>> able to follow your thought process based on the description in your
>> post. So my advice is a bit generic:
>>
>> 1. I don't know of anything in Ansible that will help you to keep a
>> single Python object in memory and use it from different tasks. Even
>> if there is some trickery by which you could accomplish this goal
>> directly, it doesn't seem to fit how Ansible works, so I wouldn't
>> recommend it.
>>
>> 2. I assume the reason you want to do this is that the JSON input
>> file(s) are large, and you don't want to keep re-parsing everything.
>>
>> 3. Here are some ideas you could consider instead of the singleton
>> approach:
>>
>> (a) Store the JSON input as an Ansible fact and pass it to
>> json_query/a custom lookup plugin (may use a lot of memory, but will
>> give you convenient access to everything).
>>
>> (b) Write a module that parses the input and returns the result of not
>> just one but many key lookups at once, and store the results as an
>> Ansible fact. This would use less memory, but would work only if you
>> have—or can build—a list of all the keys you want to retrieve.
>>
>> (c) Write a separate process that parses the input and responds to
>> questions over a network socket, and make your lookup plugin (or
>> module) ask it questions and return the results. This could be just a
>> few-line program, or you could potentially even load your data into
>> something like Postgres or redis.
>>
>> It's up to you to decide, based on the size and nature of the input
>> data and the pattern of lookups, which of these approaches would work
>> best for you.
>>
>> -- Abhijit
>>
>
--
You received this message because you are subscribed to the Google Groups
"Ansible Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to ansible-devel+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-devel/f84047f4-0556-424d-ab69-1767e3e1ff4dn%40googlegroups.com.
