This is somewhat unrelated to the original email, but more of a question of an
issue I have recently come across that seems tangentially related.
When run_one_command is executed and the message is sent to rootwrap via that
pipe, if rootwrap errors for some reason, I get " Unserializable message:
('#ERROR', ValueError('I/O operation on closed file',))" returned with a stack
trace.
In this case it appears the problem was that conntrack wasn't installed. In my
case it was that the rootwrap filters denied the request to kill the process.
I only tracked it down because I configured syslog on the rootwrap daemon and
killed the process (so l3-agent would respawn it, retry the command, and error
again). The error then appeared very clearly in /var/log/messages, so rootwrap
does understand what is going on but it doesn't seem to feed it back to the
caller.
Is this a known listed bug? It would make diagnosis a lot easier if an easy to
understand error message came back instead of the above.
Cheers,
Dan.
-----Original Message-----
From: Brian Haley [mailto:haleyb....@gmail.com]
Sent: Tuesday, 11 April 2017 6:24 AM
To: openstack@lists.openstack.org
Subject: Re: [Openstack] LinuxBridge Agent Error
On 04/10/2017 04:15 PM, Georgios Dimitrakakis wrote:
> Installing the "conntrack-tools" package from the
> "centos-openstack-ocata" repository seems to have fixed the problem.
>
> My question is if this should have happened automatically during
> installation and if yes why it didn't happen?
From your comment it was a manual install. Looking at the
neutron-sanity-check tool it doesn't seem to check for conntrack. Fix proposed
at https://review.openstack.org/455450
Thanks!
-Brian
> On Fri, 07 Apr 2017 19:26:02 +0300, Georgios Dimitrakakis wrote:
>> Hello,
>>
>> I am on CentOS 7 and this is a manual Ocata installation as described
>> on the deployment guide.
>>
>> My cloud seems to be operating normally even with these messages. The
>> thing is that I don't like all these (there are a whole bunch of them
>> every second) in my log files.
>>
>> I will try to see if I am missing the "conntrack" and try to install
>> the patch that will provide more info..
>>
>>
>> Best,
>>
>> G.
>>
>>> Is this a packstack installation?
>>>
>>> Inviato da iPhone
>>>
>>> Il giorno 07 apr 2017, alle ore di 07:19, Kevin Benton ha scritto:
>>>
>>>> You may be missing whatever package in your distro provides the
>>>> 'conntrack' util.
>>>>
>>>> On Apr 7, 2017 03:52, "Georgios Dimitrakakis" wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I am having an Ocata installation and on my Compute Node the
>>>>> "linuxbridge-agent.log" file is constantly filling out with
>>>>> messages like this:
>>>>>
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [1]ent._common_agent Traceback (most recent call last):
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [2]ent._common_agent File
>>>>>
>>>>
>>>
>>> "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/
>>> _common_agent.py",
>>>
>>>>> line 453, in daemon_loop
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [3]ent._common_agent sync =
>>>>> self.process_network_devices(device_info)
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [4]ent._common_agent File
>>>>> "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line
>>>>> 153, in wrapper
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [5]ent._common_agent return f(*args, **kwargs)
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [6]ent._common_agent File
>>>>>
>>>>
>>>
>>> "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/
>>> _common_agent.py",
>>>
>>>>> line 213, in process_network_devices
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [7]ent._common_agent resync_b =
>>>>> self.treat_devices_removed(device_info['removed'])
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [8]ent._common_agent File
>>>>> "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line
>>>>> 153, in wrapper
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [9]ent._common_agent return f(*args, **kwargs)
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [10]ent._common_agent File
>>>>>
>>>>
>>>
>>> "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/
>>> _common_agent.py",
>>>
>>>>> line 331, in treat_devices_removed
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [11]ent._common_agent self.sg_agent.remove_devices_filter(devices)
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [12]ent._common_agent File
>>>>>
>>>>
>>>
>>> "/usr/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.p
>>> y",
>>>>> line 238, in remove_devices_filter
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [13]ent._common_agent self.firewall.remove_port_filter(device)
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [14]ent._common_agent File
>>>>>
>>>>
>>>
>>> "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_firew
>>> all.py",
>>>
>>>>> line 222, in remove_port_filter
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [15]ent._common_agent
>>>>> self._remove_conntrack_entries_from_port_deleted(port)
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [16]ent._common_agent File
>>>>>
>>>>
>>>
>>> "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_firew
>>> all.py",
>>>
>>>>> line 194, in _remove_conntrack_entries_from_port_deleted
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [17]ent._common_agent [device_info], ethertype, set())
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [18]ent._common_agent File
>>>>>
>>>>
>>>
>>> "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_conntrack.p
>>> y",
>>>>> line 121, in delete_conntrack_state_by_remote_ips
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [19]ent._common_agent
>>>>> self._delete_conntrack_state(device_info_list, rule)
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [20]ent._common_agent File
>>>>>
>>>>
>>>
>>> "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_conntrack.p
>>> y",
>>>>> line 103, in _delete_conntrack_state
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [21]ent._common_agent extra_ok_codes=[1])
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [22]ent._common_agent File
>>>>> "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py",
>>>>> line 128, in execute
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [23]ent._common_agent execute_rootwrap_daemon(cmd, process_input,
>>>>> addl_env))
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [24]ent._common_agent File
>>>>> "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py",
>>>>> line 115, in execute_rootwrap_daemon
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [25]ent._common_agent return client.execute(cmd, process_input)
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [26]ent._common_agent File
>>>>> "/usr/lib/python2.7/site-packages/oslo_rootwrap/client.py", line
>>>>> 128, in execute
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [27]ent._common_agent res = proxy.run_one_command(cmd, stdin)
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [28]ent._common_agent File "", line 2, in run_one_command
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [29]ent._common_agent File
>>>>> "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
>>>>> _callmethod
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [30]ent._common_agent raise convert_to_error(kind, result)
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [31]ent._common_agent RemoteError:
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [32]ent._common_agent
>>>>>
>>>>
>>>
>>> --------------------------------------------------------------------
>>> -------
>>>
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [33]ent._common_agent Unserializable message: ('#ERROR',
>>>>> ValueError('I/O operation on closed file',))
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [34]ent._common_agent
>>>>>
>>>>
>>>
>>> --------------------------------------------------------------------
>>> -------
>>>
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>> [35]ent._common_agent
>>>>> 2017-04-07 13:37:47.710 1397 INFO neutron.plugins.ml2.drivers.ag
>>>>> [36]ent._common_agent [req-93157f5e-9f2d-4547-9002-c473211df719 -
>>>>> - - - -] Linux bridge agent Agent out of sync with plugin!
>>>>>
>>>>> Any idea what might be the problem???
>>>>>
>>>>> The full log of a total circle of messages can be found here:
>>>>> https://pastebin.com/6Yzn0FuY [37]
>>>>>
>>>>> Looking forward for an answer!
>>>>>
>>>>> Best regards,
>>>>>
>>>>> G.
>>>>>
>>>>> _______________________________________________
>>>>> Mailing list:
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [38]
>>>>> Post to : openstack@lists.openstack.org [39] Unsubscribe :
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [40]
>>>> !DSPAM:1,58e777a158574020391806!
>>>
>>>> _______________________________________________
>>>> Mailing list:
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [42]
>>>> Post to : openstack@lists.openstack.org [43] Unsubscribe :
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [44]
>>>>
>>>> !DSPAM:1,58e777a158574020391806!
>>>
>>>
>>> Links:
>>> ------
>>> [1] http://neutron.plugins.ml2.drivers.ag
>>> [2] http://neutron.plugins.ml2.drivers.ag
>>> [3] http://neutron.plugins.ml2.drivers.ag
>>> [4] http://neutron.plugins.ml2.drivers.ag
>>> [5] http://neutron.plugins.ml2.drivers.ag
>>> [6] http://neutron.plugins.ml2.drivers.ag
>>> [7] http://neutron.plugins.ml2.drivers.ag
>>> [8] http://neutron.plugins.ml2.drivers.ag
>>> [9] http://neutron.plugins.ml2.drivers.ag
>>> [10] http://neutron.plugins.ml2.drivers.ag
>>> [11] http://neutron.plugins.ml2.drivers.ag
>>> [12] http://neutron.plugins.ml2.drivers.ag
>>> [13] http://neutron.plugins.ml2.drivers.ag
>>> [14] http://neutron.plugins.ml2.drivers.ag
>>> [15] http://neutron.plugins.ml2.drivers.ag
>>> [16] http://neutron.plugins.ml2.drivers.ag
>>> [17] http://neutron.plugins.ml2.drivers.ag
>>> [18] http://neutron.plugins.ml2.drivers.ag
>>> [19] http://neutron.plugins.ml2.drivers.ag
>>> [20] http://neutron.plugins.ml2.drivers.ag
>>> [21] http://neutron.plugins.ml2.drivers.ag
>>> [22] http://neutron.plugins.ml2.drivers.ag
>>> [23] http://neutron.plugins.ml2.drivers.ag
>>> [24] http://neutron.plugins.ml2.drivers.ag
>>> [25] http://neutron.plugins.ml2.drivers.ag
>>> [26] http://neutron.plugins.ml2.drivers.ag
>>> [27] http://neutron.plugins.ml2.drivers.ag
>>> [28] http://neutron.plugins.ml2.drivers.ag
>>> [29] http://neutron.plugins.ml2.drivers.ag
>>> [30] http://neutron.plugins.ml2.drivers.ag
>>> [31] http://neutron.plugins.ml2.drivers.ag
>>> [32] http://neutron.plugins.ml2.drivers.ag
>>> [33] http://neutron.plugins.ml2.drivers.ag
>>> [34] http://neutron.plugins.ml2.drivers.ag
>>> [35] http://neutron.plugins.ml2.drivers.ag
>>> [36] http://neutron.plugins.ml2.drivers.ag
>>> [37] https://pastebin.com/6Yzn0FuY
>>> [38] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>> [39] mailto:openstack@lists.openstack.org
>>> [40] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>> [41] mailto:gior...@acmac.uoc.gr
>>> [42] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>> [43] mailto:openstack@lists.openstack.org
>>> [44] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>> [45] mailto:ke...@benton.pub
>>
>>
>> _______________________________________________
>> Mailing list:
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to : openstack@lists.openstack.org
>> Unsubscribe :
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
