That's definitely a helpful improvement, but isn't there a way to get the
full error as well rather than an error about serializing the error?
On Wed, Apr 12, 2017 at 8:19 AM, Brian Haley <haleyb....@gmail.com> wrote:
> On 04/11/2017 06:35 PM, Daniel Russell wrote:
>
>> This is somewhat unrelated to the original email, but more of a question
>> of an issue I have recently come across that seems tangentially related.
>>
>> When run_one_command is executed and the message is sent to rootwrap via
>> that pipe, if rootwrap errors for some reason, I get " Unserializable
>> message: ('#ERROR', ValueError('I/O operation on closed file',))" returned
>> with a stack trace.
>>
>> In this case it appears the problem was that conntrack wasn't installed.
>> In my case it was that the rootwrap filters denied the request to kill the
>> process. I only tracked it down because I configured syslog on the
>> rootwrap daemon and killed the process (so l3-agent would respawn it, retry
>> the command, and error again). The error then appeared very clearly in
>> /var/log/messages, so rootwrap does understand what is going on but it
>> doesn't seem to feed it back to the caller.
>>
>> Is this a known listed bug? It would make diagnosis a lot easier if an
>> easy to understand error message came back instead of the above.
>>
>
> Hi Dan,
>
> https://bugs.launchpad.net/neutron/+bug/1677742 was the bug I opened for
> the original rootwrap error, and the change for that merged upstream. That
> will catch the error and print the command being tried, which is at least a
> bread crumb and finding the failure. Could be there is more info we can
> gather, if you can try that change and add info to the bug it would be
> great.
>
> -Brian
>
>
>
> -----Original Message-----
>> From: Brian Haley [mailto:haleyb....@gmail.com]
>> Sent: Tuesday, 11 April 2017 6:24 AM
>> To: openstack@lists.openstack.org
>> Subject: Re: [Openstack] LinuxBridge Agent Error
>>
>> On 04/10/2017 04:15 PM, Georgios Dimitrakakis wrote:
>>
>>> Installing the "conntrack-tools" package from the
>>> "centos-openstack-ocata" repository seems to have fixed the problem.
>>>
>>> My question is if this should have happened automatically during
>>> installation and if yes why it didn't happen?
>>>
>>
>> From your comment it was a manual install. Looking at the
>> neutron-sanity-check tool it doesn't seem to check for conntrack. Fix
>> proposed at https://review.openstack.org/455450
>>
>> Thanks!
>>
>> -Brian
>>
>>
>> On Fri, 07 Apr 2017 19:26:02 +0300, Georgios Dimitrakakis wrote:
>>>
>>>> Hello,
>>>>
>>>> I am on CentOS 7 and this is a manual Ocata installation as described
>>>> on the deployment guide.
>>>>
>>>> My cloud seems to be operating normally even with these messages. The
>>>> thing is that I don't like all these (there are a whole bunch of them
>>>> every second) in my log files.
>>>>
>>>> I will try to see if I am missing the "conntrack" and try to install
>>>> the patch that will provide more info..
>>>>
>>>>
>>>> Best,
>>>>
>>>> G.
>>>>
>>>> Is this a packstack installation?
>>>>>
>>>>> Inviato da iPhone
>>>>>
>>>>> Il giorno 07 apr 2017, alle ore di 07:19, Kevin Benton ha scritto:
>>>>>
>>>>> You may be missing whatever package in your distro provides the
>>>>>> 'conntrack' util.
>>>>>>
>>>>>> On Apr 7, 2017 03:52, "Georgios Dimitrakakis" wrote:
>>>>>>
>>>>>> Hello,
>>>>>>>
>>>>>>> I am having an Ocata installation and on my Compute Node the
>>>>>>> "linuxbridge-agent.log" file is constantly filling out with
>>>>>>> messages like this:
>>>>>>>
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [1]ent._common_agent Traceback (most recent call last):
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [2]ent._common_agent File
>>>>>>>
>>>>>>>
>>>>>>
>>>>> "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/
>>>>> _common_agent.py",
>>>>>
>>>>> line 453, in daemon_loop
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [3]ent._common_agent sync =
>>>>>>> self.process_network_devices(device_info)
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [4]ent._common_agent File
>>>>>>> "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line
>>>>>>> 153, in wrapper
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [5]ent._common_agent return f(*args, **kwargs)
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [6]ent._common_agent File
>>>>>>>
>>>>>>>
>>>>>>
>>>>> "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/
>>>>> _common_agent.py",
>>>>>
>>>>> line 213, in process_network_devices
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [7]ent._common_agent resync_b =
>>>>>>> self.treat_devices_removed(device_info['removed'])
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [8]ent._common_agent File
>>>>>>> "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line
>>>>>>> 153, in wrapper
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [9]ent._common_agent return f(*args, **kwargs)
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [10]ent._common_agent File
>>>>>>>
>>>>>>>
>>>>>>
>>>>> "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/
>>>>> _common_agent.py",
>>>>>
>>>>> line 331, in treat_devices_removed
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [11]ent._common_agent self.sg_agent.remove_devices_filter(devices)
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [12]ent._common_agent File
>>>>>>>
>>>>>>>
>>>>>>
>>>>> "/usr/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.p
>>>>> y",
>>>>>
>>>>>> line 238, in remove_devices_filter
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [13]ent._common_agent self.firewall.remove_port_filter(device)
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [14]ent._common_agent File
>>>>>>>
>>>>>>>
>>>>>>
>>>>> "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_firew
>>>>> all.py",
>>>>>
>>>>> line 222, in remove_port_filter
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [15]ent._common_agent
>>>>>>> self._remove_conntrack_entries_from_port_deleted(port)
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [16]ent._common_agent File
>>>>>>>
>>>>>>>
>>>>>>
>>>>> "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_firew
>>>>> all.py",
>>>>>
>>>>> line 194, in _remove_conntrack_entries_from_port_deleted
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [17]ent._common_agent [device_info], ethertype, set())
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [18]ent._common_agent File
>>>>>>>
>>>>>>>
>>>>>>
>>>>> "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_conntrack.p
>>>>> y",
>>>>>
>>>>>> line 121, in delete_conntrack_state_by_remote_ips
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [19]ent._common_agent
>>>>>>> self._delete_conntrack_state(device_info_list, rule)
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [20]ent._common_agent File
>>>>>>>
>>>>>>>
>>>>>>
>>>>> "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_conntrack.p
>>>>> y",
>>>>>
>>>>>> line 103, in _delete_conntrack_state
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [21]ent._common_agent extra_ok_codes=[1])
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [22]ent._common_agent File
>>>>>>> "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py",
>>>>>>> line 128, in execute
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [23]ent._common_agent execute_rootwrap_daemon(cmd, process_input,
>>>>>>> addl_env))
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [24]ent._common_agent File
>>>>>>> "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py",
>>>>>>> line 115, in execute_rootwrap_daemon
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [25]ent._common_agent return client.execute(cmd, process_input)
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [26]ent._common_agent File
>>>>>>> "/usr/lib/python2.7/site-packages/oslo_rootwrap/client.py", line
>>>>>>> 128, in execute
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [27]ent._common_agent res = proxy.run_one_command(cmd, stdin)
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [28]ent._common_agent File "", line 2, in run_one_command
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [29]ent._common_agent File
>>>>>>> "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
>>>>>>> _callmethod
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [30]ent._common_agent raise convert_to_error(kind, result)
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [31]ent._common_agent RemoteError:
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [32]ent._common_agent
>>>>>>>
>>>>>>>
>>>>>>
>>>>> --------------------------------------------------------------------
>>>>> -------
>>>>>
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [33]ent._common_agent Unserializable message: ('#ERROR',
>>>>>>> ValueError('I/O operation on closed file',))
>>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [34]ent._common_agent
>>>>>>>
>>>>>>>
>>>>>>
>>>>> --------------------------------------------------------------------
>>>>> -------
>>>>>
>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>>> [35]ent._common_agent
>>>>>>> 2017-04-07 13:37:47.710 1397 INFO neutron.plugins.ml2.drivers.ag
>>>>>>> [36]ent._common_agent [req-93157f5e-9f2d-4547-9002-c473211df719 -
>>>>>>> - - - -] Linux bridge agent Agent out of sync with plugin!
>>>>>>>
>>>>>>> Any idea what might be the problem???
>>>>>>>
>>>>>>> The full log of a total circle of messages can be found here:
>>>>>>> https://pastebin.com/6Yzn0FuY [37]
>>>>>>>
>>>>>>> Looking forward for an answer!
>>>>>>>
>>>>>>> Best regards,
>>>>>>>
>>>>>>> G.
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Mailing list:
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [38]
>>>>>>> Post to : openstack@lists.openstack.org [39] Unsubscribe :
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [40]
>>>>>>>
>>>>>> !DSPAM:1,58e777a158574020391806!
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>>> Mailing list:
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [42]
>>>>>> Post to : openstack@lists.openstack.org [43] Unsubscribe :
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [44]
>>>>>>
>>>>>> !DSPAM:1,58e777a158574020391806!
>>>>>>
>>>>>
>>>>>
>>>>> Links:
>>>>> ------
>>>>> [1] http://neutron.plugins.ml2.drivers.ag
>>>>> [2] http://neutron.plugins.ml2.drivers.ag
>>>>> [3] http://neutron.plugins.ml2.drivers.ag
>>>>> [4] http://neutron.plugins.ml2.drivers.ag
>>>>> [5] http://neutron.plugins.ml2.drivers.ag
>>>>> [6] http://neutron.plugins.ml2.drivers.ag
>>>>> [7] http://neutron.plugins.ml2.drivers.ag
>>>>> [8] http://neutron.plugins.ml2.drivers.ag
>>>>> [9] http://neutron.plugins.ml2.drivers.ag
>>>>> [10] http://neutron.plugins.ml2.drivers.ag
>>>>> [11] http://neutron.plugins.ml2.drivers.ag
>>>>> [12] http://neutron.plugins.ml2.drivers.ag
>>>>> [13] http://neutron.plugins.ml2.drivers.ag
>>>>> [14] http://neutron.plugins.ml2.drivers.ag
>>>>> [15] http://neutron.plugins.ml2.drivers.ag
>>>>> [16] http://neutron.plugins.ml2.drivers.ag
>>>>> [17] http://neutron.plugins.ml2.drivers.ag
>>>>> [18] http://neutron.plugins.ml2.drivers.ag
>>>>> [19] http://neutron.plugins.ml2.drivers.ag
>>>>> [20] http://neutron.plugins.ml2.drivers.ag
>>>>> [21] http://neutron.plugins.ml2.drivers.ag
>>>>> [22] http://neutron.plugins.ml2.drivers.ag
>>>>> [23] http://neutron.plugins.ml2.drivers.ag
>>>>> [24] http://neutron.plugins.ml2.drivers.ag
>>>>> [25] http://neutron.plugins.ml2.drivers.ag
>>>>> [26] http://neutron.plugins.ml2.drivers.ag
>>>>> [27] http://neutron.plugins.ml2.drivers.ag
>>>>> [28] http://neutron.plugins.ml2.drivers.ag
>>>>> [29] http://neutron.plugins.ml2.drivers.ag
>>>>> [30] http://neutron.plugins.ml2.drivers.ag
>>>>> [31] http://neutron.plugins.ml2.drivers.ag
>>>>> [32] http://neutron.plugins.ml2.drivers.ag
>>>>> [33] http://neutron.plugins.ml2.drivers.ag
>>>>> [34] http://neutron.plugins.ml2.drivers.ag
>>>>> [35] http://neutron.plugins.ml2.drivers.ag
>>>>> [36] http://neutron.plugins.ml2.drivers.ag
>>>>> [37] https://pastebin.com/6Yzn0FuY
>>>>> [38] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>> [39] mailto:openstack@lists.openstack.org
>>>>> [40] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>> [41] mailto:gior...@acmac.uoc.gr
>>>>> [42] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>> [43] mailto:openstack@lists.openstack.org
>>>>> [44] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>> [45] mailto:ke...@benton.pub
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list:
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>> Post to : openstack@lists.openstack.org
>>>> Unsubscribe :
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>
>>>
>>>
>>> _______________________________________________
>>> Mailing list: http://lists.openstack.org/cgi
>>> -bin/mailman/listinfo/openstack
>>> Post to : openstack@lists.openstack.org
>>> Unsubscribe :
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>
>>
>>
>> _______________________________________________
>> Mailing list: http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>> Post to : openstack@lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>>
>>
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac
> k
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac
> k
>
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
