We also believe the change from auto-delete queues to 10min expiration queues was the cause of our rabbit whoes a month or so ago. Where we had rabbitmq servers filling their stats DB and consuming 20+ GB of ram before hitting the rabbitmq mem high watermark. We were running for 6+ months without issue under kilo and when we moved to Liberty rabbit consistently started falling on its face. We eventually turned down the stats collection interval, but I would imagine keeping stats around for queue’s for 10 minutes that were used for a single RPC message when we are passing 1500+ messages per second wasn’t helping anything. We haven’t tried changing the timeout values to be lower, to see if that made things better. But we did identify this change as something that could contribute to our rabbitmq issues.
___________________________________________________________________ Kris Lindgren Senior Linux Systems Engineer GoDaddy From: Dmitry Mescheryakov <dmescherya...@mirantis.com> Date: Thursday, July 28, 2016 at 6:17 AM To: Sam Morrison <sorri...@gmail.com> Cc: OpenStack Operators <openstack-operators@lists.openstack.org> Subject: Re: [Openstack-operators] [oslo] RabbitMQ queue TTL issues moving to Liberty 2016-07-27 2:20 GMT+03:00 Sam Morrison <sorri...@gmail.com<mailto:sorri...@gmail.com>>: On 27 Jul 2016, at 4:05 AM, Dmitry Mescheryakov <dmescherya...@mirantis.com<mailto:dmescherya...@mirantis.com>> wrote: 2016-07-26 2:15 GMT+03:00 Sam Morrison <sorri...@gmail.com<mailto:sorri...@gmail.com>>: The queue TTL happens on reply queues and fanout queues. I don’t think it should happen on fanout queues. They should auto delete. I can understand the reason for having them on reply queues though so maybe that would be a way to forward? Or am I missing something and it is needed on fanout queues too? I would say we do need fanout queues to expire for the very same reason we want reply queues to expire instead of auto delete. In case of broken connection, the expiration provides client time to reconnect and continue consuming from the queue. In case of auto-delete queues, it was a frequent case that RabbitMQ deleted the queue before client reconnects ... along with all non-consumed messages in it. But in the case of fanout queues, if there is a broken connection can’t the service just recreate the queue if it doesn’t exist? I guess that means it needs to store the state of what the queue name is though? Yes they could loose messages directed at them but all the services I know that consume on fanout queues have a re sync functionality for this very case. If the connection is broken will oslo messaging know how to connect to the same queue again anyway? I would’ve thought it would handle the disconnect and then reconnect, either with the same queue name or a new queue all together? oslo.messaging handles reconnect perfectly - on connect it just unconditionally declares the queue and starts consuming from it. If queue already existed, the declaration operation will just be ignored by RabbitMQ. For your earlier point that services re sync and hence messages lost in fanout are not that important, I can't comment on that. But after some thinking I do agree that having big expiration time for fanouts is non-adequate for big deployments anyway. How about we split rabbit_transient_queues_ttl into two parameters - one for reply queue and one for fanout ones? In that case people concerned with messages piling up in fanouts might set it to 1, which will virtually make these queues behave like auto-delete ones (though I strongly recommend to leave it at least at 20 seconds, to give service a chance to reconnect). Thanks, Dmitry Sam
_______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
