Hello Wido, Thanks for the reply.
I used the RBD command line tool directly without specifying any features. It’s the default since Ceph Jewel. The features that I require are for these use cases: 1.Using 'rbd du' the get the real disk usage of a volume without fast-diff is very slow. We need to schedule this after hours on large images. 2.We are starting to use rbd mirroring across to our secondary DC and this requires exclusive-lock. 3.Deep flatten can not be turned on after image creation. I am not sure if I require this feature at this time. Kind regards, Glen Baars -----Original Message----- From: Wido den Hollander <w...@widodh.nl> Sent: Thursday, 17 May 2018 10:18 PM To: dev@cloudstack.apache.org; Glen Baars <g...@onsitecomputers.com.au> Subject: Re: Ceph RBD issues in 4.11 On 05/17/2018 01:50 PM, Glen Baars wrote: > Hello Dev, > > I have recently upgraded our cloudstack environment to 4.11. Mostly > all has been smooth. ( this environment is legacy from cloud.com days! > ) > > There are some issues that I have run into: > > 1.Can't install any VMs from ISO ( I have seen this in the list previously > but can't find a bug report for it ) If further reports or debug will help I > can assist. It is easy to reproduce. > 2.When a VM is created from a template, the RBD features are lost. More info > below. > > Example of VM volume from template: - > > user@NAS-AUBUN-RK3-CEPH01:~# rbd info > AUBUN-KVM-CLUSTER01-SSD/feeb52ec-f111-4a0d-9785-23aadd7650a5 > > rbd image 'feeb52ec-f111-4a0d-9785-23aadd7650a5': > size 150 GB in 38400 objects > order 22 (4096 kB objects) > block_name_prefix: rbd_data.142926a5ee64 > format: 2 > features: layering > flags: > create_timestamp: Fri Apr 27 12:46:21 2018 > parent: > AUBUN-KVM-CLUSTER01-SSD/d7dcd9e4-ed55-44ae-9a71-52c9307e53b4@cloudstack-base-snap > overlap: 150 GB > > Note the features are not the same as the parent : - > > user@NAS-AUBUN-RK3-CEPH01:~# rbd info > AUBUN-KVM-CLUSTER01-SSD/d7dcd9e4-ed55-44ae-9a71-52c9307e53b4 > rbd image 'd7dcd9e4-ed55-44ae-9a71-52c9307e53b4': > size 150 GB in 38400 objects > order 22 (4096 kB objects) > block_name_prefix: rbd_data.141d274b0dc51 > format: 2 > features: layering, exclusive-lock, object-map, fast-diff, > deep-flatten > flags: > create_timestamp: Fri Apr 27 12:37:05 2018 > > > If you manually clone the volume the expected features are retained. We are > running the latest Ceph version, KVM hosts on Ubuntu 16.04 with the latest > Luminous qemu-img. > How do you clone the volume manually? I assume with the rbd tool? Because this is where Java/CloudStack clones the image: https://github.com/apache/cloudstack/blob/master/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java#L957 private int rbdFeatures = (1 << 0); /* Feature 1<<0 means layering in RBD format 2 */ rbd.clone(template.getName(), rbdTemplateSnapName, io, disk.getName(), rbdFeatures, rbdOrder); So it's on purpose and this has a historical reason which I can't thin of anymore. We can probably update this to include exclusive-lock, object-map, fast-diff and deep-flatten. Or completely skip it and have RBD figure it out. Don't know anymore why this code is in there. But if this a real probably that you don't have those features? Wido > Kind regards, > Glen Baars > > This e-mail is intended solely for the benefit of the addressee(s) and any > other named recipient. It is confidential and may contain legally privileged > or confidential information. If you are not the recipient, any use, > distribution, disclosure or copying of this e-mail is prohibited. The > confidentiality and legal privilege attached to this communication is not > waived or lost by reason of the mistaken transmission or delivery to you. If > you have received this e-mail in error, please notify us immediately. > This e-mail is intended solely for the benefit of the addressee(s) and any other named recipient. It is confidential and may contain legally privileged or confidential information. If you are not the recipient, any use, distribution, disclosure or copying of this e-mail is prohibited. The confidentiality and legal privilege attached to this communication is not waived or lost by reason of the mistaken transmission or delivery to you. If you have received this e-mail in error, please notify us immediately.
