On 5/28/19 1:48 PM, li jerry wrote:
> Hi Wido
>
>
>
> I filled in the CLOUDSTACK is the following KEY
>
>
>
> [root@cn01-nodeb ~]# ceph auth get client.cloudstack
>
> exported keyring for client.cloudstack
>
> [client.cloudstack]
>
> key = AQDTh7pcIJjNIhAAwk8jtxilJWXQR7osJRFMLw==
>
> caps mon = "allow r"
>
> caps osd = "allow rwx pool=rbd"
>
>
That's the problem :-) Your user needs to be updated.
The caps should be:
[client.cloudstack]
key = AQDTh7pcIJjNIhAAwk8jtxilJWXQR7osJRFMLw==
caps mon = "profile rbd"
caps osd = "profile rbd pool=rbd"
See: http://docs.ceph.com/docs/master/rbd/rbd-cloudstack/
This will allow the client to blacklist the other and take over the
exclusive-lock.
Wido
>
> *发件人: *Wido den Hollander <mailto:w...@widodh.nl>
> *发送时间: *2019年5月28日19:42
> *收件人: *dev@cloudstack.apache.org <mailto:dev@cloudstack.apache.org>;
> li jerry <mailto:div...@hotmail.com>; us...@cloudstack.apache.org
> <mailto:us...@cloudstack.apache.org>
> *主题: *Re: RBD primary storage VM encounters Exclusive Lock after
> triggering HA
>
>
>
>
>
> On 5/28/19 6:16 AM, li jerry wrote:
>> Hello guys
>>
>> we’ve deployed an environment with CloudStack 4.11.2 and KVM(CentOS7.6), and
>> Ceph 13.2.5 is deployed as the primary storage.
>> We found some issues with the HA solution, and we are here to ask for you
>> suggestions.
>>
>> We’ve both enabled VM HA and Host HA feature in CloudStack, and the compute
>> offering is tagged as ha.
>> When we try to perform a power failure test (unplug 1 node of 4), the
>> running VMs on the removed node is automatically rescheduled to the other
>> living nodes after 5 minutes, but all of them can not boot into the OS. We
>> found the booting procedure is stuck by the IO read/write failure.
>>
>>
>>
>> The following information is prompted after VM starts:
>>
>> Generating "/run/initramfs/rdsosreport.txt"
>>
>> Entering emergency mode. Exit the shell to continue.
>> Type "journalctl" to view system logs.
>> You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or
>> /boot
>> after mounting them and attach it to a bug report
>>
>> :/#
>>
>>
>>
>> We found this is caused by the lock on the image:
>> [root@cn01-nodea ~]# rbd lock list a93010b0-2be2-49bd-b25e-ec89b3a98b4b
>> There is 1 exclusive lock on this image.
>> Locker ID Address
>> client.1164351 auto 94464726847232 10.226.16.128:0/3002249644
>>
>> If we remove the lock from the image, and restart the VM under CloudStack,
>> this VM will boot successfully.
>>
>> We know that if we disable the Exclusive Lock feature (by setting
>> rbd_default_features = 3) for Ceph would solve this problem. But we don’t
>> think it’s the best solution for the HA, so could you please give us some
>> ideas about how you are doing and what is the best practice for this feature?
>>
>
> exclusive-lock is something to prevent a split-brain and having two
> clients write to it at the same time.
>
> The lock should be released to the other client if this is requested,
> but I have the feeling that you might have a cephx problem there.
>
> Can you post the output of:
>
> $ ceph auth get client.X
>
> Where you replace X by the user you are using for CloudStack? Also
> remove they 'key', I don't need that.
>
> I want to look at the caps of the user.
>
> Wido
>
>> Thanks.
>>
>>
>
>
>
