On Mon, Jan 6, 2020 at 4:59 PM <miguel.casti...@centro.net> wrote:
>
> Happy New Year Ceph Community!
>
> I'm in the process of figuring out RBD mirroring with Ceph and having a
> really tough time with it. I'm trying to set up just one way mirroring right
> now on some test systems (baremetal servers, all Debian 9). The first cluster
> is 3 nodes, and the 2nd cluster is 2 nodes (not worried about a properly
> performing setup, just the functionality of RBD mirroring right now). The
> purpose is to have a passive failover ceph cluster in a separate DC.
> Mirroring seems like the best solution, but if we can't get it working, we'll
> end up resorting to a scheduled rsync which is less than ideal. I've followed
> several guides, read through a lot of documentation, and nothing has worked
> for me thus far. If anyone can offer some troubleshooting help or insight
> into what I might have missed in this setup, I'd greatly appreciate it! I
> also don't fully understand the relationship between images and pools and how
> you're supposed to configure statically sized images for a pool that has a
> variable amount of data, but that's a question for afterwards, I think :)
>
> Once RBD mirroring is set up, the mirror test image status shows as
> down+unknown:
>
> On ceph1-dc2:
> rbd --cluster dc1ceph mirror pool status fs_data --verbose
> health: WARNING
> images: 1 total
> 1 unknown
>
> mirror_test:
> global_id: c335017c-9b8f-49ee-9bc1-888789537c47
> state: down+unknown
> description: status not found
> last_update:
>
> Here are the commands I run using ceph-deploy on both clusters to get
> everything up and running (run from a deploy directory on the first node of
> each cluster). The clusters are created at the same time, and rbd setup
> commands are only run after the clusters are up and healthy, and the fs_data
> pool is created.
>
> -----------------------------------------------------------
>
> Cluster 1 (dc1ceph):
>
> ceph-deploy new ceph1-dc1 ceph2-dc1 ceph3-dc1
> sed -i '$ s,.*,public_network = *.*.*.0/24\n,g' ceph.conf
> ceph-deploy install ceph1-dc1 ceph2-dc1 ceph3-dc1 --release luminous
> ceph-deploy mon create-initial
> ceph-deploy admin ceph1-dc1 ceph2-dc1 ceph3-dc1
> ceph-deploy mgr create ceph1-dc1 ceph2-dc1 ceph3-dc1
> for x in b c d e f g h i j k; do ceph-deploy osd create --data /dev/sd${x}1
> ceph1-dc1 ; done
> for x in b c d e f g h i j k; do ceph-deploy osd create --data /dev/sd${x}1
> ceph2-dc1 ; done
> for x in b c d e f g h i j k; do ceph-deploy osd create --data /dev/sd${x}1
> ceph3-dc1 ; done
> ceph-deploy mds create ceph1-dc1 ceph2-dc1 ceph3-dc1
> ceph-deploy rgw create ceph1-dc1 ceph2-dc1 ceph3-dc1
> for f in 1 2 ; do scp ceph.client.admin.keyring
> ceph$f-dc2:/etc/ceph/dc1ceph.client.admin.keyring ; done
> for f in 1 2 ; do scp ceph.conf ceph$f-dc2:/etc/ceph/dc1ceph.conf ; done
> for f in 1 2 ; do ssh ceph$f-dc2 "chown ceph.ceph /etc/ceph/dc1ceph*" ; done
> ceph osd pool create fs_data 512 512 replicated
> rbd --cluster ceph mirror pool enable fs_data image
> rbd --cluster dc2ceph mirror pool enable fs_data image
> rbd --cluster ceph mirror pool peer add fs_data client.admin@dc2ceph
> (generated id: b5e347b3-0515-4142-bc49-921a07636865)
> rbd create fs_data/mirror_test --size=1G
> rbd feature enable fs_data/mirror_test journaling
> rbd mirror image enable fs_data/mirror_test
> chown ceph.ceph ceph.client.admin.keyring
>
> Cluster 2 (dc2ceph):
>
> ceph-deploy new ceph1-dc2 ceph2-dc2
> sed -i '$ s,.*,public_network = *.*.*.0/24\n,g' ceph.conf
> ceph-deploy install ceph1-dc2 ceph2-dc2 --release luminous
> ceph-deploy mon create-initial
> ceph-deploy admin ceph1-dc2 ceph2-dc2
> ceph-deploy mgr create ceph1-dc2 ceph2-dc2
> for x in b c d e f g h i j k; do ceph-deploy osd create --data /dev/sd${x}1
> ceph1-dc2 ; done
> for x in b c d e f g h i j k; do ceph-deploy osd create --data /dev/sd${x}1
> ceph2-dc2 ; done
> ceph-deploy mds create ceph1-dc2 ceph2-dc2
> ceph-deploy rgw create ceph1-dc2 ceph2-dc2
> apt install rbd-mirror
> for f in 1 2 3 ; do scp ceph.conf ceph$f-dc1:/etc/ceph/dc2ceph.conf ; done
> for f in 1 2 3 ; do scp ceph.client.admin.keyring
> ceph$f-dc1:/etc/ceph/dc2ceph.client.admin.keyring ; done
> for f in 1 2 3 ; do ssh ceph$f-dc1 "chown ceph.ceph /etc/ceph/dc2ceph*" ; done
> ceph osd pool create fs_data 512 512 replicated
> rbd --cluster ceph mirror pool peer add fs_data client.admin@dc1ceph
> (generated id: e486c401-e24d-49bc-9800-759760822282)
> systemctl enable ceph-rbd-mirror@admin
> systemctl start ceph-rbd-mirror@admin
> rbd --cluster dc1ceph mirror pool status fs_data --verbose
>
>
> Cluster 1:
>
> ls /etc/ceph:
> ceph.client.admin.keyring
> ceph.conf
> dc2ceph.client.admin.keyring
> dc2ceph.conf
> rbdmap
> tmpG36OYs
>
> cat /etc/ceph/ceph.conf:
> [global]
> fsid = 8fede407-50e1-4487-8356-3dc98b30c500
> mon_initial_members = ceph1-dc1, ceph2-dc1, ceph3-dc1
> mon_host = *.*.*.1,*.*.*.27,*.*.*.41
> auth_cluster_required = cephx
> auth_service_required = cephx
> auth_client_required = cephx
> public_network = *.*.*.0/24
>
> cat /etc/ceph/dc2ceph.conf
> [global]
> fsid = 813ff410-02dc-47bd-b678-38add38495bb
> mon_initial_members = ceph1-dc2, ceph2-dc2
> mon_host = *.*.*.56,*.*.*.0
> auth_cluster_required = cephx
> auth_service_required = cephx
> auth_client_required = cephx
> public_network = *.*.*.0/24
>
>
> Cluster 2:
>
> ls /etc/ceph:
> ceph.client.admin.keyring
> ceph.conf
> dc1ceph.client.admin.keyring
> dc1ceph.conf
> rbdmap
> tmp_yxkPs
>
> cat /etc/ceph/ceph.conf
> [global]
> fsid = 813ff410-02dc-47bd-b678-38add38495bb
> mon_initial_members = ceph1-dc2, ceph2-dc2
> mon_host = *.*.*.56,*.*.*.70
> auth_cluster_required = cephx
> auth_service_required = cephx
> auth_client_required = cephx
> public_network = *.*.*.0/24
>
> cat /etc/ceph/dc1ceph.conf
> [global]
> fsid = 8fede407-50e1-4487-8356-3dc98b30c500
> mon_initial_members = ceph1-dc1, ceph2-dc1, ceph3-dc1
> mon_host = *.*.*.1,*.*.*.27,*.*.*.41
> auth_cluster_required = cephx
> auth_service_required = cephx
> auth_client_required = cephx
> public_network = *.*.*.0/24
>
>
> RBD Mirror daemon status:
>
> ceph-rbd-mirror@admin.service - Ceph rbd mirror daemon
> Loaded: loaded (/lib/systemd/system/ceph-rbd-mirror@.service; enabled;
> vendor preset: enabled)
> Active: inactive (dead) since Mon 2020-01-06 16:21:44 EST; 3s ago
> Process: 910178 ExecStart=/usr/bin/rbd-mirror -f --cluster ${CLUSTER} --id
> admin --setuser ceph --setgroup ceph (code=exited, status=0/SUCCESS)
> Main PID: 910178 (code=exited, status=0/SUCCESS)
>
> Jan 06 16:21:44 ceph1-dc2 systemd[1]: Started Ceph rbd mirror daemon.
> Jan 06 16:21:44 ceph1-dc2 rbd-mirror[910178]: 2020-01-06 16:21:44.462916
> 7f76ecf88780 -1 auth: unable to find a keyring on
> /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,:
> (2) No such file or directory
> Jan 06 16:21:44 ceph1-dc2 rbd-mirror[910178]: 2020-01-06 16:21:44.462949
> 7f76ecf88780 -1 monclient: ERROR: missing keyring, cannot use cephx for
> authentication
> Jan 06 16:21:44 ceph1-dc2 rbd-mirror[910178]: failed to initialize: (2) No
> such file or directory2020-01-06 16:21:44.463874 7f76ecf88780 -1
> rbd::mirror::Mirror: 0x558d3ce6ce20 init: error connecting to local cluster
>
> -------------------------------------------
>
> I also tried running the ExecStart command manually, substituting in
> different values for the parameters, and just never got it to work. If more
> info is needed, please don’t hesitate to ask. Thanks in advance!
Can you double-check that you really have
"/etc/ceph/ceph.client.admin.keyring" on the node "ceph1-dc2" and that
the "ceph" user can actually access it? If you run "sudo -u ceph rbd
ls" can it properly connect to the local cluster? Is selinux or
apparmor enabled and preventing access to the necessary files?
>
> -Miguel
> _______________________________________________
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
--
Jason
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
