Mike,

For a sufficiently large number of volumes, the thin provisioning stats 
gathering could break things already
before the referenced patch:
https://bugs.launchpad.net/cinder/+bug/1704106 
<https://bugs.launchpad.net/cinder/+bug/1704106>

It seems, however, that the attempt to gather at least the correct data (used 
instead of allocated) lowers that
threshold even further.

In order to allow our c-vol to start (and as we don’t use over-provisioning), 
we’ve for now commented out the
usage stats gathering.  

Cheers,
 Arne




> On 03 Aug 2017, at 20:47, Mike Lowe <joml...@iu.edu> wrote:
> 
> I did the minor point release update from 10.0.2 to 10.0.4 and found my 
> cinder volume services would go out to lunch during startup. They would do 
> their initial heartbeat then get marked as dead never sending another 
> heartbeat.  The process was running and there were constant logs about ceph 
> connections but what was missing was the follow up to "Initializing RPC 
> dependent components of volume driver RBDDriver (1.2.0)”. It never finished 
> the rpc init "Driver post RPC initialization completed successfully.”  
> Digging in a little bit with my limited knowledge of the python librbd it 
> seems that this commit landed in 10.0.4 
> https://github.com/openstack/cinder/commit/e72dead5ce085a6ba66f7aad2ff58061842f43d2
>   Instead of looping over the volume size for every volume it looped over all 
> the volumes calling diff_iterate from offset 0 to the end.   Near as I can 
> tell this actually calls whatever you pass in as iterate_cb for every used 
> extent of the volume. So a handful of empty volumes no problem, but in 
> production by my count I would have to call iterate_cb 12.6M times just to 
> add up the bytes used from each extent.   I’ve filed a bug 
> https://bugs.launchpad.net/cinder/+bug/1708507 and downgrading to 10.0.2 
> seems to be an ok workaround.
> 
> TLDR; if you have ceph don’t upgrade past 10.0.2, for the time being
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Reply via email to