don;t see this issue on kilo setup , so close it
** Changed in: neutron
Status: Incomplete => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1498472
Title:
lbaas:after 871 namespace was created for 1v1 mapping, the new LB
pool with pending_create status
Status in neutron:
Invalid
Bug description:
after 817 tenants was create for namespace 1v1 maping, the new LB pool
with pending_creat status
test step
1 create a tenant
2 create a pool network with subnet 10.1.1.0
create a vip network with subnet 20.1.1.0
3 create a router and add 2 interface on it for subnet 10 and subnet20
4 create a LB pool with subnet 10
5 add a VIP with vip address 20.1.1.100 on this pool with protocol HTTP and
port 80
6(option ) add one or two member
7 repeat above step to continuous add 1v1 map( 1 tenant map 1 lb namespce)
for scale test
add to following tenant
80 tenants with 2 backend and 1 client (LB monitor enable )with simple
traffic
100 tenants with 1 backend and 1 client (LB monitor enable) with simple
traffic
1067 tenants without member and monitor
total of 1247 pools , total of 817 pools with active , total of 376 pools
with error or pending-create status ) log in attachment
the issue :after add 817 namespace , created new lb pool with pending-status
or error status
after check log, AMQP can't connect to server ,
and then reset rabbitmq service on controller node " service rabbitmq-server
restart"
the setup can't recover , the new lb pool still with pending-create status
for recover setup, start to delete tenants and lb-pool and vip ( 1v1 maping)
after delete tenant to 328(remain), the setup recover , new lb pool can be
create with active status , so it was a scale issue , and file this bug for
trace this issue
log
[root@nsj17 ~]# tail -n 200 /var/log/neutron/lbaas-agent.log
2015-08-11 20:35:51.939 2581 TRACE oslo.messaging._drivers.impl_rabbit
2015-08-11 20:35:51.944 2581 INFO oslo.messaging._drivers.impl_rabbit [-]
Delaying reconnect for 1.0 seconds...
2015-08-11 20:35:51.970 2581 ERROR oslo.messaging._drivers.impl_rabbit [-]
Failed to consume message from queue: (0, 0): (320) CONNECTION_FORCED - broker
forced connection closure with reason 'shutdown'
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit
Traceback (most recent call last):
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line
655, in ensure
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit
return method()
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line
735, in _consume
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit
return self.connection.drain_events(timeout=timeout)
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 281, in
drain_events
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit
return self.transport.drain_events(self.connection, **kwargs)
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 94, in
drain_events
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit
return connection.drain_events(**kwargs)
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/connection.py", line 320, in drain_events
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit
return amqp_method(channel, args)
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/connection.py", line 526, in _close
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit
(class_id, method_id), ConnectionError)
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit
ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection
closure with reason 'shutdown'
2015-08-11 20:35:51.970 2581 TRACE oslo.messaging._drivers.impl_rabbit
2015-08-11 20:35:51.972 2581 INFO oslo.messaging._drivers.impl_rabbit [-]
Delaying reconnect for 1.0 seconds...
2015-08-11 20:35:52.946 2581 INFO oslo.messaging._drivers.impl_rabbit [-]
Connecting to AMQP server on 10.53.87.192:5672
2015-08-11 20:35:52.971 2581 ERROR oslo.messaging._drivers.impl_rabbit [-]
AMQP server on 10.53.87.192:5672 is unreachable: [Errno 111] ECONNREFUSED.
Trying again in 1 seconds.
2015-08-11 20:35:52.973 2581 INFO oslo.messaging._drivers.impl_rabbit [-]
Connecting to AMQP server on 10.53.87.192:5672
2015-08-11 20:35:52.993 2581 ERROR oslo.messaging._drivers.impl_rabbit [-]
AMQP server on 10.53.87.192:5672 is unreachable: [Errno 111] ECONNREFUSED.
Trying again in 1 seconds.
2015-08-11 20:35:53.990 2581 INFO oslo.messaging._drivers.impl_rabbit [-]
Delaying reconnect for 1.0 seconds...
2015-08-11 20:35:53.999 2581 INFO oslo.messaging._drivers.impl_rabbit [-]
Delaying reconnect for 1.0 seconds...
2015-08-11 20:35:55.000 2581 INFO oslo.messaging._drivers.impl_rabbit [-]
Connecting to AMQP server on 10.53.87.192:5672
2015-08-11 20:35:55.023 2581 INFO oslo.messaging._drivers.impl_rabbit [-]
Connecting to AMQP server on 10.53.87.192:5672
2015-08-11 20:35:55.060 2581 INFO oslo.messaging._drivers.impl_rabbit [-]
Connected to AMQP server on 10.53.87.192:5672
2015-08-11 20:35:55.083 2581 INFO oslo.messaging._drivers.impl_rabbit [-]
Connected to AMQP server on 10.53.87.192:5672
2015-08-11 20:35:57.681 2581 WARNING oslo.messaging._drivers.amqpdriver [-]
No calling threads waiting for msg_id : ed3ca0b705914e4b9e0542fd1cdc3eda,
message : {u'_unique_id': u'4b941437471a4de99f0871e34216a70a', u'failure':
None, u'result': None}
2015-08-11 20:35:57.683 2581 WARNING oslo.messaging._drivers.amqpdriver [-]
_queues: {'060c2e4d42f04e31be500647207d3a47': <Queue at 0x3a6e290 maxsize=None
getters[1]>, '78db36a255ae4b20b090554216923a20': <Queue at 0x3c2e5d0
maxsize=None>}
2015-08-11 20:35:57.737 2581 WARNING oslo.messaging._drivers.amqpdriver [-]
No calling threads waiting for msg_id : ed3ca0b705914e4b9e0542fd1cdc3eda,
message : {u'_unique_id': u'e9bfc396a78c49a2993f6db149711c17', u'failure':
None, u'result': None, u'ending': True}
2015-08-11 20:35:57.738 2581 WARNING oslo.messaging._drivers.amqpdriver [-]
_queues: {'060c2e4d42f04e31be500647207d3a47': <Queue at 0x3a6e290 maxsize=None
getters[1] tasks=1 _cond=<Event at 0x3b09050 result=NOT_USED _exc=None
_waiters[0]>>, '78db36a255ae4b20b090554216923a20': <Queue at 0x3c2e5d0
maxsize=None>}
2015-08-11 20:36:14.044 2581 ERROR oslo.messaging._drivers.impl_rabbit [-]
Failed to publish message to topic 'n-lbaas-plugin': [Errno 32] Broken pipe
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
Traceback (most recent call last):
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line
655, in ensure
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
return method()
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line
752, in _publish
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
publisher = cls(self.conf, self.channel, topic=topic, **kwargs)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line
378, in __init__
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
**options)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line
326, in __init__
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
self.reconnect(channel)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line
334, in reconnect
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
routing_key=self.routing_key)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/kombu/messaging.py", line 83, in __init__
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
self.revive(self._channel)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/kombu/messaging.py", line 214, in revive
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
self.declare()
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/kombu/messaging.py", line 100, in declare
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
self.exchange.declare()
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/kombu/entity.py", line 163, in declare
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
nowait=nowait, passive=passive,
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/channel.py", line 620, in
exchange_declare
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
(40, 11), # Channel.exchange_declare_ok
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
self.channel_id, allowed_methods)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/connection.py", line 270, in _wait_method
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
self.wait()
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
return self.dispatch_method(method_sig, args, content)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in
dispatch_method
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
return amqp_method(self, args)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/connection.py", line 523, in _close
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
self._x_close_ok()
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/connection.py", line 551, in _x_close_ok
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
self._send_method((10, 51))
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 56, in
_send_method
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
self.channel_id, method_sig, args, content,
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 221, in
write_method
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
write_frame(1, channel, payload)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/transport.py", line 177, in write_frame
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
frame_type, channel, size, payload, 0xce,
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 359, in sendall
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
tail = self.send(data, flags)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 342, in send
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
total_sent += fd.send(data[total_sent:], flags)
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit error:
[Errno 32] Broken pipe
2015-08-11 20:36:14.044 2581 TRACE oslo.messaging._drivers.impl_rabbit
2015-08-11 20:36:14.050 2581 INFO oslo.messaging._drivers.impl_rabbit [-]
Delaying reconnect for 1.0 seconds...
2015-08-11 20:36:15.052 2581 INFO oslo.messaging._drivers.impl_rabbit [-]
Connecting to AMQP server on 10.53.87.192:5672
2015-08-11 20:36:15.080 2581 INFO oslo.messaging._drivers.impl_rabbit [-]
Connected to AMQP server on 10.53.87.192:5672
2015-08-11 20:36:57.747 2581 ERROR
neutron.services.loadbalancer.agent.agent_manager [-] Unable to deploy instance
for pool: 92fb078d-f203-4972-a555-e2ba033d50d8
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager Traceback (most recent call
last):
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager File
"/usr/lib/python2.7/site-packages/neutron/services/loadbalancer/agent/agent_manager.py",
line 169, in _reload_pool
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager logical_config =
self.plugin_rpc.get_logical_device(pool_id)
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager File
"/usr/lib/python2.7/site-packages/neutron/services/loadbalancer/agent/agent_api.py",
line 57, in get_logical_device
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager host=self.host
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager File
"/usr/lib/python2.7/site-packages/neutron/common/log.py", line 34, in wrapper
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager return method(*args,
**kwargs)
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager File
"/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 161, in call
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager context, msg,
rpc_method='call', **kwargs)
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager File
"/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 187, in
__call_rpc_method
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager return func(context,
msg['method'], **msg['args'])
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager File
"/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 389, in
call
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager return
self.prepare().call(ctxt, method, **kwargs)
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager File
"/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 152, in
call
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager retry=self.retry)
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager File
"/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in
_send
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager timeout=timeout,
retry=retry)
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager File
"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line
408, in send
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager retry=retry)
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager File
"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line
397, in _send
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager result =
self._waiter.wait(msg_id, timeout)
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager File
"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line
285, in wait
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager reply, ending =
self._poll_connection(msg_id, timeout)
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager File
"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line
235, in _poll_connection
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager % msg_id)
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager MessagingTimeout: Timed out
waiting for a reply to message ID 78db36a255ae4b20b090554216923a20
2015-08-11 20:36:57.747 2581 TRACE
neutron.services.loadbalancer.agent.agent_manager
2015-08-11 20:37:15.791 2581 ERROR oslo.messaging._drivers.impl_rabbit [-]
Failed to consume message from queue: (0, 0): (320) CONNECTION_FORCED - broker
forced connection closure with reason 'shutdown'
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit
Traceback (most recent call last):
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line
655, in ensure
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit
return method()
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line
735, in _consume
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit
return self.connection.drain_events(timeout=timeout)
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 281, in
drain_events
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit
return self.transport.drain_events(self.connection, **kwargs)
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 94, in
drain_events
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit
return connection.drain_events(**kwargs)
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/connection.py", line 320, in drain_events
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit
return amqp_method(channel, args)
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/connection.py", line 526, in _close
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit
(class_id, method_id), ConnectionError)
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit
ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection
closure with reason 'shutdown'
2015-08-11 20:37:15.791 2581 TRACE oslo.messaging._drivers.impl_rabbit
2015-08-11 20:37:15.795 2581 INFO oslo.messaging._drivers.impl_rabbit [-]
Delaying reconnect for 1.0 seconds...
2015-08-11 20:37:16.796 2581 INFO oslo.messaging._drivers.impl_rabbit [-]
Connecting to AMQP server on 10.53.87.192:5672
2015-08-11 20:37:16.815 2581 ERROR oslo.messaging._drivers.impl_rabbit [-]
AMQP server on 10.53.87.192:5672 is unreachable: [Errno 111] ECONNREFUSED.
Trying again in 1 seconds.
2015-08-11 20:37:17.832 2581 INFO oslo.messaging._drivers.impl_rabbit [-]
Delaying reconnect for 1.0 seconds...
2015-08-11 20:37:18.855 2581 INFO oslo.messaging._drivers.impl_rabbit [-]
Connecting to AMQP server on 10.53.87.192:5672
2015-08-11 20:37:18.903 2581 INFO oslo.messaging._drivers.impl_rabbit [-]
Connected to AMQP server on 10.53.87.192:5672
2015-08-11 20:37:23.258 2581 ERROR oslo.messaging._drivers.impl_rabbit
[req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Failed to publish message to topic
'n-lbaas-plugin': [Errno 32] Broken pipe
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
Traceback (most recent call last):
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line
655, in ensure
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
return method()
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line
752, in _publish
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
publisher = cls(self.conf, self.channel, topic=topic, **kwargs)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line
378, in __init__
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
**options)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line
326, in __init__
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
self.reconnect(channel)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line
334, in reconnect
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
routing_key=self.routing_key)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/kombu/messaging.py", line 83, in __init__
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
self.revive(self._channel)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/kombu/messaging.py", line 214, in revive
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
self.declare()
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/kombu/messaging.py", line 100, in declare
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
self.exchange.declare()
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/kombu/entity.py", line 163, in declare
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
nowait=nowait, passive=passive,
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/channel.py", line 620, in
exchange_declare
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
(40, 11), # Channel.exchange_declare_ok
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
self.channel_id, allowed_methods)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/connection.py", line 270, in _wait_method
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
self.wait()
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
return self.dispatch_method(method_sig, args, content)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in
dispatch_method
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
return amqp_method(self, args)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/connection.py", line 523, in _close
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
self._x_close_ok()
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/connection.py", line 551, in _x_close_ok
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
self._send_method((10, 51))
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 56, in
_send_method
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
self.channel_id, method_sig, args, content,
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 221, in
write_method
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
write_frame(1, channel, payload)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/transport.py", line 177, in write_frame
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
frame_type, channel, size, payload, 0xce,
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 359, in sendall
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
tail = self.send(data, flags)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 342, in send
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
total_sent += fd.send(data[total_sent:], flags)
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit error:
[Errno 32] Broken pipe
2015-08-11 20:37:23.258 2581 TRACE oslo.messaging._drivers.impl_rabbit
2015-08-11 20:37:23.266 2581 INFO oslo.messaging._drivers.impl_rabbit
[req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Delaying reconnect for 1.0
seconds...
2015-08-11 20:37:24.277 2581 INFO oslo.messaging._drivers.impl_rabbit
[req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Connecting to AMQP server on
10.53.87.192:5672
2015-08-11 20:37:24.337 2581 INFO oslo.messaging._drivers.impl_rabbit
[req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Connected to AMQP server on
10.53.87.192:5672
2015-08-11 20:37:24.344 2581 ERROR oslo.messaging._drivers.impl_rabbit
[req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Failed to consume message from
queue: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with
reason 'shutdown'
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit
Traceback (most recent call last):
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line
655, in ensure
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit
return method()
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line
735, in _consume
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit
return self.connection.drain_events(timeout=timeout)
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 281, in
drain_events
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit
return self.transport.drain_events(self.connection, **kwargs)
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 94, in
drain_events
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit
return connection.drain_events(**kwargs)
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/connection.py", line 320, in drain_events
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit
return amqp_method(channel, args)
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit File
"/usr/lib/python2.7/site-packages/amqp/connection.py", line 526, in _close
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit
(class_id, method_id), ConnectionError)
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit
ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection
closure with reason 'shutdown'
2015-08-11 20:37:24.344 2581 TRACE oslo.messaging._drivers.impl_rabbit
2015-08-11 20:37:24.346 2581 INFO oslo.messaging._drivers.impl_rabbit
[req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Delaying reconnect for 1.0
seconds...
2015-08-11 20:37:25.347 2581 INFO oslo.messaging._drivers.impl_rabbit
[req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Connecting to AMQP server on
10.53.87.192:5672
2015-08-11 20:37:25.387 2581 INFO oslo.messaging._drivers.impl_rabbit
[req-ec66e3e6-7afb-43a0-8c72-52205fc99b3f ] Connected to AMQP server on
10.53.87.192:5672
[root@nsj17 ~]#
Notes Log:
IBM Licensed Material - Property of IBM ? Cop
correct the number
180 with traffic
690 without traffic
Total Active LB Namespace = 870
Total LB Namespace = 1247
Inactive = 377
[root@nsj17 ~]# sar -P ALL 60 1
Linux 3.10.0-229.7.2.el7.x86_64 (nsj17) 08/12/2015 _x86_64_
(80 CPU)
07:28:50 PM CPU %user %nice %system %iowait %steal
%idle
07:29:50 PM all 2.45 0.00 72.87 0.00 0.00
24.68
07:29:50 PM 0 3.09 0.00 86.10 0.00 0.00
10.81
07:29:50 PM 1 3.24 0.00 83.92 0.00 0.00
12.84
07:29:50 PM 2 2.88 0.00 77.57 0.00 0.00
19.54
07:29:50 PM 3 2.16 0.00 75.61 0.00 0.00
22.23
07:29:50 PM 4 2.37 0.00 76.48 0.00 0.00
21.15
07:29:50 PM 5 2.44 0.00 75.88 0.00 0.00
21.69
07:29:50 PM 6 2.23 0.00 78.28 0.00 0.00
19.48
Average: CPU %user %nice %system %iowait %steal
%idle
Average: all 2.45 0.00 72.87 0.00 0.00
24.68
Average: 0 3.09 0.00 86.10 0.00 0.00
10.81
Average: 1 3.24 0.00 83.92 0.00 0.00
12.84
Average: 2 2.88 0.00 77.57 0.00 0.00
19.54
Average: 3 2.16 0.00 75.61 0.00 0.00
22.23
Average: 4 2.37 0.00 76.48 0.00 0.00
21.15
Average: 5 2.44 0.00 75.88 0.00 0.00
21.69
Average: 6 2.23 0.00 78.28 0.00 0.00
19.48
Average: 7 2.49 0.00 72.97 0.00 0.00
24.54
Average: 8 2.63 0.00 75.52 0.00 0.00
21.84
Average: 9 2.95 0.00 71.80 0.00 0.00
25.25
Average: 10 2.23 0.00 76.04 0.00 0.00
21.73
[root@nsj17 ~]# free
total used free shared buff/cache
available
Mem: 263961496 19596608 220503484 4236472 23861404
231497040
Swap: 1048572 0 1048572
[root@nsj17 ~]#
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1498472/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
