[ceph-users] Erasure coded calculation
Hello everyone! I'm trying to calculate the theoretical usable storage of a ceph cluster with erasure coded pools. I have 8 nodes and the profile for all data pools will be k=6 m=2. If every node has 6 x 1TB wouldn't the calculation be like this: RAW capacity: 8Nodes x 6Disks x 1TB = 48TB Loss to m=2: 48TB / 8Nodes x 2m = 12TB EC capacity: 48TB - 12TB = 36TB At the moment I have one cluster with 8 nodes and different disks than the sample (but every node has the same amount of disks and the same sized disks). The output of ceph df detail is: --- RAW STORAGE --- CLASS SIZE AVAILUSED RAW USED %RAW USED hdd109 TiB 103 TiB 5.8 TiB 5.9 TiB 5.41 TOTAL 109 TiB 103 TiB 5.8 TiB 5.9 TiB 5.41 --- POOLS --- POOL ID PGS STORED OBJECTS %USED MAX AVAIL device_health_metrics 11 51 MiB 48 0 30 TiB rep_data_fs 2 32 14 KiB3.41k 0 30 TiB rep_meta_fs 3 32 227 MiB1.72k 0 30 TiB ec_bkp14 32 4.2 TiB1.10M 6.11 67 TiB So ec_bkp1 uses 4.2TiB an there are 67TiB free usable Storage. This means total EC usable storage would be 71.2TiB. But calculating with the 109TiB RAW storage, shouldn't it be 81.75? Are the 10TiB just some overhead (that would be much overhead) or is the calculation not correct? And what If I want to expand the cluster in the first sample above by three nodes with 6 x 2TB, which means not the same sized disks as the others. Will the calculation with the same EC profile still be the same? RAW capacity: 8Nodes x 6Disks x 1TB + 3Nodes x 6Disks x 2TB = 84TB Loss to m=2: 84TB / 11Nodes x 2m = 15.27TB EC capacity: 84TB - 15.27TB = 68.72TB Thanks in advance, Simon ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Cephfs mount not recovering after icmp-not-reachable
Hello everyone! We had a switch outage and the ceph kernel mount did not work anymore. This is the fstab entry: 10.99.10.1:/somefolder /cephfs ceph _netdev,nofail,name=cephcluster,secret=IsSecret 0 0 I reproduced it with disabling the vlan on the switch on which the ceph is reachable, which gives a icmp-not-reachable. I did this for five minutes, after that, "ls /cephfs" just gives a "permission denied" in dmesg i can see this: [ 1412.994921] libceph: mon1 10.99.10.4:6789 session lost, hunting for new mon [ 1413.009325] libceph: mon0 10.99.10.1:6789 session established [ 1452.998646] libceph: mon2 10.99.15.3:6789 session lost, hunting for new mon [ 1452.998679] libceph: mon0 10.99.10.1:6789 session lost, hunting for new mon [ 1461.989549] libceph: mon4 10.99.15.5:6789 socket closed (con state CONNECTING) --- [ 1787.045148] libceph: mon3 10.99.15.4:6789 socket closed (con state CONNECTING) [ 1787.062587] libceph: mon0 10.99.10.1:6789 session established [ 1787.086103] libceph: mon4 10.99.15.5:6789 session established [ 1814.028761] libceph: mds0 10.99.10.4:6801 socket closed (con state OPEN) [ 1815.029811] libceph: mds0 10.99.10.4:6801 connection reset [ 1815.029829] libceph: reset on mds0 [ 1815.029831] ceph: mds0 closed our session [ 1815.029833] ceph: mds0 reconnect start [ 1815.052219] ceph: mds0 reconnect denied [ 1815.052229] ceph: dropping dirty Fw state for 9d9085da1340 1099512175611 [ 1815.052231] ceph: dropping dirty+flushing Fw state for 9d9085da1340 1099512175611 [ 1815.273008] libceph: mds0 10.99.10.4:6801 socket closed (con state NEGOTIATING) [ 1816.033241] ceph: mds0 rejected session [ 1829.018643] ceph: mds0 hung [ 1880.088504] ceph: mds0 came back [ 1880.088662] ceph: mds0 caps renewed [ 1880.094018] ceph: get_quota_realm: ino (1000afe.fffe) null i_snap_realm [ 1881.100367] ceph: get_quota_realm: ino (1000afe.fffe) null i_snap_realm [ 2046.768969] conntrack: generic helper won't handle protocol 47. Please consider loading the specific helper module. [ 2061.731126] ceph: get_quota_realm: ino (1000afe.fffe) null i_snap_realm Is this a bug to report or wrong configuration? Did someone else had this before? To solve the problem, a simple remount does the trick. Thanks in advance Simon ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] configure fuse in fstab
Hello Everyone! I'm trying to mount the ceph, with the fuse client under debian 9 (ceph-fuse 10.2.11-2). Ceph is on the latest Octopus release. The direct command is working, but writing it in fstab does not. Command I use: ceph-fuse --id dev.wsc -k /etc/ceph/ceph.clinet.dev.wsc.keyring -r /testing/dev.wsc/userdata /mnt/ceph /etc/ceph.conf: [global] fsid = [fsid] mon_host = 10.99.15.1 10.99.15.2 10.99.15.3 10.99.15.4 10.99.15.5 /etc/ceph/ceph.clinet.dev.wsc.keyring: [client.dev.wsc] key = [key] fstab: none /mnt/ceph fuse.ceph ceph.id=dev.wsc,ceph.conf=/etc/ceph/ceph.conf,ceph.client_mountpoint=/testing/dev.wsc/userdata,_netdev,defaults 0 0 With mount -a it tells me "ceph mount failed with (1) Operation not permitted". Where can I find authentication logs on the cluster? Thanks in advance and kind regards, Simon ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: configure fuse in fstab
Hello Stefan! Thanks for the input. Yes that was a typo. I have created the file /etc/ceph/ceph.client.dev.wsc.key and entered just the key without anything else. fstab line now looks like this: none/mnt/ceph fuse.ceph ceph.id=dev.wsc,ceph.client_mountpoint=/testing/dev.wsc/userdata,_netdev,defaults 0 0 I still get the same "Permission denied" message. But this fuse command also works: ceph-fuse --id dev.wsc -r /testing/dev.wsc/userdata /mnt/ceph Does anyone have any other ideas? Thanks, Simon Von: Stefan Kooman Gesendet: Donnerstag, 1. Juli 2021 16:42:56 An: Simon Sutter; ceph-users@ceph.io Betreff: Re: [ceph-users] configure fuse in fstab On 7/1/21 4:14 PM, Simon Sutter wrote: > Hello Everyone! > > I'm trying to mount the ceph, with the fuse client under debian 9 (ceph-fuse > 10.2.11-2). > > Ceph is on the latest Octopus release. > The direct command is working, but writing it in fstab does not. > > > Command I use: > ceph-fuse --id dev.wsc -k /etc/ceph/ceph.clinet.dev.wsc.keyring -r > /testing/dev.wsc/userdata /mnt/ceph > > > /etc/ceph.conf: > > [global] > fsid = [fsid] > mon_host = 10.99.15.1 10.99.15.2 10.99.15.3 10.99.15.4 10.99.15.5 > > > /etc/ceph/ceph.clinet.dev.wsc.keyring: ^^ I guess that is a typo > > [client.dev.wsc] > key = [key] Can you try to put the file "ceph.client.dev.wsc.key" in /etc/ceph and only put the key in it (nothing else) ... and try again Gr. Stefan P.s. There should be no need for ceph.conf=/etc/ceph/ceph.conf as its the default location. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Performance optimization
Hello everyone! I have built two clusters with old hardware, which is lying around, the possibility to upgrade is there. The clusters main usecase is hot backup. This means it's getting written 24/7 where 99% is writing and 1% is reading. It should be based on harddisks. At the moment, the nodes look like this: 8 Nodes Worst CPU: i7-3930K (up to i7-6850K) Worst ammount of RAM: 24GB (up to 64GB) HDD Layout: 1x 1TB 4x 2TB 1x 6TB all sata, some just 5400rpm I had to put the OS on the 6TB HDDs, because there are no more sata connections on the motherboard. The servers, which have to be backed up, have mounted the ceph with cephfs. 99% of the files, that have to be backed up, are harddisk images, so sizes from 5GB to 1TB. All files are written to an erasure-coded pool with k=6 m=2, compression is on passive snappy, default settings. I'm getting really bad performace with this setup. This is a bench, run with: "rados -p ec_test bench -b 524288 60 write" while normal operations: Total time run: 63.4957 Total writes made: 459 Write size: 524288 Object size:524288 Bandwidth (MB/sec): 3.61442 Stddev Bandwidth: 3.30073 Max bandwidth (MB/sec): 16 Min bandwidth (MB/sec): 0 Average IOPS: 7 Stddev IOPS:6.6061 Max IOPS: 32 Min IOPS: 0 Average Latency(s): 2.151 Stddev Latency(s): 2.3661 Max latency(s): 14.0916 Min latency(s): 0.0420954 Cleaning up (deleting benchmark objects) Removed 459 objects Clean up completed and total clean up time :35.6908 [root@testnode01 ~]# ceph osd perf osd commit_latency(ms) apply_latency(ms) 6 655655 9 13 13 11 15 15 7 17 17 10 19 19 8 12 12 24 153153 25 22 22 47 20 20 46 23 23 45 43 43 44 8 8 16 26 26 15 18 18 14 14 14 13 23 23 12 47 47 18 595595 1 20 20 38 25 25 17 17 17 0 317317 37 19 19 19 14 14 2 16 16 39 9 9 20 16 16 3 18 18 40 10 10 21 23 23 4 17 17 41 29 29 5 18 18 42 16 16 22 16 16 23 13 13 26 20 20 27 10 10 28 28 28 29 13 13 30 34 34 31 10 10 32 31 31 33 44 44 34 21 21 35 22 22 36 295295 43 9 9 What do you think is the most obvious Problem? - The one 6TB disk, per node? - The OS on the 6TB disk? What would you suggest? What I hope to replace with this setup: 6 servers, each with 4x3TB disks, with lvm, no redundancy. (two times, that's why I have set up two clusters) Thanks in advance Simon ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Performance optimization
Hello Thanks for this first input, I already found, at least one of those 6TB Disks is a WD Blue WD60EZAZ which is according to WD with SMR. I will replace everything with SMR in it, but in the process of replacing hardware, should I replace all disks with for example all 3TB disks? And what do you think about having the os on one of the disks, used by ceph? Thanks in advance, Simon Von: Kai Börnert Gesendet: Montag, 6. September 2021 10:54:24 An: ceph-users@ceph.io Betreff: [ceph-users] Re: Performance optimization Hi, are any of those old disks SMR ones? Because they will absolutely destroy any kind of performance (ceph does not use writecaches due to powerloss concerns, so they kinda do their whole magic for each writerequest). Greetings On 9/6/21 10:47 AM, Simon Sutter wrote: > Hello everyone! > > I have built two clusters with old hardware, which is lying around, the > possibility to upgrade is there. > The clusters main usecase is hot backup. This means it's getting written 24/7 > where 99% is writing and 1% is reading. > > > It should be based on harddisks. > > > > At the moment, the nodes look like this: > 8 Nodes > Worst CPU: i7-3930K (up to i7-6850K) > > Worst ammount of RAM: 24GB (up to 64GB) > HDD Layout: > 1x 1TB > 4x 2TB > 1x 6TB > all sata, some just 5400rpm > > I had to put the OS on the 6TB HDDs, because there are no more sata > connections on the motherboard. > > The servers, which have to be backed up, have mounted the ceph with cephfs. > 99% of the files, that have to be backed up, are harddisk images, so sizes > from 5GB to 1TB. > > All files are written to an erasure-coded pool with k=6 m=2, compression is > on passive snappy, default settings. > > I'm getting really bad performace with this setup. > This is a bench, run with: "rados -p ec_test bench -b 524288 60 write" while > normal operations: > > Total time run: 63.4957 > Total writes made: 459 > Write size: 524288 > Object size:524288 > Bandwidth (MB/sec): 3.61442 > Stddev Bandwidth: 3.30073 > Max bandwidth (MB/sec): 16 > Min bandwidth (MB/sec): 0 > Average IOPS: 7 > Stddev IOPS:6.6061 > Max IOPS: 32 > Min IOPS: 0 > Average Latency(s): 2.151 > Stddev Latency(s): 2.3661 > Max latency(s): 14.0916 > Min latency(s): 0.0420954 > Cleaning up (deleting benchmark objects) > Removed 459 objects > Clean up completed and total clean up time :35.6908 > > [root@testnode01 ~]# ceph osd perf > osd commit_latency(ms) apply_latency(ms) >6 655655 >9 13 13 > 11 15 15 >7 17 17 > 10 19 19 >8 12 12 > 24 153153 > 25 22 22 > 47 20 20 > 46 23 23 > 45 43 43 > 44 8 8 > 16 26 26 > 15 18 18 > 14 14 14 > 13 23 23 > 12 47 47 > 18 595595 >1 20 20 > 38 25 25 > 17 17 17 >0 317317 > 37 19 19 > 19 14 14 >2 16 16 > 39 9 9 > 20 16 16 >3 18 18 > 40 10 10 > 21 23 23 >4 17 17 > 41 29 29 >5 18 18 > 42 16 16 > 22 16 16 > 23 13 13 > 26 20 20 > 27 10 10 > 28 28 28 > 29 13 13 > 30 34 34 > 31 10 10 > 32 31 31 > 33 44 44 > 34
[ceph-users] Re: Performance optimization
Hello > > > > >> - The one 6TB disk, per node? > > > > > > You get bad distribution of data, why not move drives around between > > these to clusters, so you have more the same in each. > > > > > > > I would assume that this behaves exactly the other way around. As long > > as you have the same number of block devices with the same size > > distribution in each node you will get an even data distribution. > > > > If you have a node with 4 3TB drives and one with 4 6TB drives Ceph > > cannot use the 6TB drives efficiently. > > > He has 2 clusters thus 3TB -> cluster 1, 6TB -> cluster eg. Sorry for the bad Information. I have two clusters, but my question was about just one of them. Yes Robert is right, instead of this configuration: | node1 | node2 | node3 | node4 | node5 | node6 | node7 | node8 | | 1x1TB | 1x1TB | 1x1TB | 1x1TB | 1x1TB | 1x1TB | 1x1TB | 1x1TB | | 4x2TB | 4x2TB | 4x2TB | 4x2TB | 4x2TB | 4x2TB | 4x2TB | 4x2TB | | 1x6TB | 1x6TB | 1x6TB | 1x6TB | 1x6TB | 1x6TB | 1x6TB | 1x6TB | This: | node1 | node2 | node3 | node4 | node5 | node6 | node7 | node8 | | 6x3TB | 6x3TB | 6x3TB | 6x3TB | 6x3TB | 6x3TB | 6x3TB | 6x3TB | Would this even be a noticeable performance difference? Because if I'm not mistaken, ceph will try to fill every disk on one node to the same percentage. And about Erasure Coded: what would be the recommended specification? Because replicated uses so much more storage, it wasn't really an option until now. We didn't have any problems with CPU utilization and I can go to 32GB for every node, and 64 for MDS nodes. Thanks Von: Marc Gesendet: Montag, 6. September 2021 13:53:06 An: Robert Sander; ceph-users@ceph.io Betreff: [ceph-users] Re: Performance optimization > > >> - The one 6TB disk, per node? > > > > You get bad distribution of data, why not move drives around between > these to clusters, so you have more the same in each. > > > > I would assume that this behaves exactly the other way around. As long > as you have the same number of block devices with the same size > distribution in each node you will get an even data distribution. > > If you have a node with 4 3TB drives and one with 4 6TB drives Ceph > cannot use the 6TB drives efficiently. > He has 2 clusters thus 3TB -> cluster 1, 6TB -> cluster eg. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Performance optimization
I never looked into RocksDB, because I thought writing data 24/7 does not benefit from caching. But this is metadata storage, so I might profit from it. Due to lack of sata ports, is it possible to save all RocksDB's on one ssd? It should still be faster, to write it to just one ssd, instead of writing it to the disk directly. Tahnks, Simon Von: Robert Sander Gesendet: Montag, 6. September 2021 16:48:52 An: Simon Sutter; Marc; ceph-users@ceph.io Betreff: Re: [ceph-users] Re: Performance optimization Am 06.09.21 um 16:44 schrieb Simon Sutter: > |node1|node2|node3|node4|node5|node6|node7|node8| > |1x1TB|1x1TB|1x1TB|1x1TB|1x1TB|1x1TB|1x1TB|1x1TB| > |4x2TB|4x2TB|4x2TB|4x2TB|4x2TB|4x2TB|4x2TB|4x2TB| > |1x6TB|1x6TB|1x6TB|1x6TB|1x6TB|1x6TB|1x6TB|1x6TB| "ceph osd df tree" should show the data distribution among the OSDs. Are all of these HDDs? Are these HDDs equipped with RocksDB on SSD? HDD only will have abysmal performance. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Upgrading to Octopus
Hello everybody In octopus there are some interesting looking features, so I tried to upgrading my Centos 7 test nodes, according to: https://docs.ceph.com/docs/master/releases/octopus/ Everything went fine and the cluster is healthy. To test out the new dashboard functions, I tried to install it, but there are missing dependencies: yum install ceph-mgr-dashboard.noarch . --> Finished Dependency Resolution Error: Package: 2:ceph-mgr-dashboard-15.2.1-0.el7.noarch (Ceph-noarch) Requires: python3-routes Error: Package: 2:ceph-mgr-dashboard-15.2.1-0.el7.noarch (Ceph-noarch) Requires: python3-jwt Error: Package: 2:ceph-mgr-dashboard-15.2.1-0.el7.noarch (Ceph-noarch) Requires: python3-cherrypy Installing them with pip3 does of course make no difference, because those are yum dependencies. Does anyone know a workaround? Do I have to upgrade to Centos 8 for this to work? Thanks in advance, Simon ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Upgrading to Octopus
Hello Khodayar Of cours I tried installing them with yum... They are not available in the centos base or epel repos, here are the ones, which are available: [root@node1 ~]# yum list | egrep "cherrypy|jwt|routes" python-cherrypy.noarch 3.2.2-4.el7@base python-cherrypy2.noarch2.3.0-19.el7 @epel python-jwt.noarch 1.5.3-1.el7@base python-routes.noarch 1.13-2.el7 @epel nodejs-jwt-simple.noarch 0.2.0-1.el7epel python36-jwt.noarch1.6.4-2.el7epel How do I get either: The right packages or a workaround because i can install the dependencies with pip? Regards, Simon Von: Khodayar Doustar Gesendet: Mittwoch, 22. April 2020 20:02:04 An: Simon Sutter Cc: ceph-users@ceph.io Betreff: Re: [ceph-users] Upgrading to Octopus Hi Simon, Have you tried installing them with yum? On Wed, Apr 22, 2020 at 6:16 PM Simon Sutter mailto:ssut...@hosttech.ch>> wrote: Hello everybody In octopus there are some interesting looking features, so I tried to upgrading my Centos 7 test nodes, according to: https://docs.ceph.com/docs/master/releases/octopus/ Everything went fine and the cluster is healthy. To test out the new dashboard functions, I tried to install it, but there are missing dependencies: yum install ceph-mgr-dashboard.noarch . --> Finished Dependency Resolution Error: Package: 2:ceph-mgr-dashboard-15.2.1-0.el7.noarch (Ceph-noarch) Requires: python3-routes Error: Package: 2:ceph-mgr-dashboard-15.2.1-0.el7.noarch (Ceph-noarch) Requires: python3-jwt Error: Package: 2:ceph-mgr-dashboard-15.2.1-0.el7.noarch (Ceph-noarch) Requires: python3-cherrypy Installing them with pip3 does of course make no difference, because those are yum dependencies. Does anyone know a workaround? Do I have to upgrade to Centos 8 for this to work? Thanks in advance, Simon ___ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-le...@ceph.io<mailto:ceph-users-le...@ceph.io> ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Upgrading to Octopus
Khodayar, I added all those repos, but sitll, those packages are missing. I can of course search for the exact package name like this: [root@node1 ~]# yum search python3-cherrypy Loaded plugins: fastestmirror, langpacks, priorities Loading mirror speeds from cached hostfile * base: pkg.adfinis-sygroup.ch * centos-ceph-nautilus: pkg.adfinis-sygroup.ch * centos-nfs-ganesha28: pkg.adfinis-sygroup.ch * centos-openstack-stein: pkg.adfinis-sygroup.ch * centos-qemu-ev: pkg.adfinis-sygroup.ch * centos-sclo-rh: pkg.adfinis-sygroup.ch * centos-sclo-sclo: pkg.adfinis-sygroup.ch * epel: pkg.adfinis-sygroup.ch * extras: pkg.adfinis-sygroup.ch * updates: pkg.adfinis-sygroup.ch Warning: No matches found for: python3-cherrypy No matches found But as you can see, it cannot find it. Anything else I can try? Regards, Simon Von: Khodayar Doustar Gesendet: Donnerstag, 23. April 2020 14:41:38 An: Simon Sutter Cc: ceph-users@ceph.io Betreff: Re: [ceph-users] Re: Upgrading to Octopus Simon, You can try to search for the exact package name, you can try these repos as well: yum<https://www.server-world.info/en/command/html/yum.html> -y install epel-release centos-release-ceph-nautilus centos-release-openstack-stein On Thu, Apr 23, 2020 at 11:57 AM Simon Sutter mailto:ssut...@hosttech.ch>> wrote: Hello Khodayar Of cours I tried installing them with yum... They are not available in the centos base or epel repos, here are the ones, which are available: [root@node1 ~]# yum list | egrep "cherrypy|jwt|routes" python-cherrypy.noarch 3.2.2-4.el7@base python-cherrypy2.noarch2.3.0-19.el7 @epel python-jwt.noarch 1.5.3-1.el7@base python-routes.noarch 1.13-2.el7 @epel nodejs-jwt-simple.noarch 0.2.0-1.el7epel python36-jwt.noarch1.6.4-2.el7epel How do I get either: The right packages or a workaround because i can install the dependencies with pip? Regards, Simon Von: Khodayar Doustar mailto:dous...@rayanexon.ir>> Gesendet: Mittwoch, 22. April 2020 20:02:04 An: Simon Sutter Cc: ceph-users@ceph.io<mailto:ceph-users@ceph.io> Betreff: Re: [ceph-users] Upgrading to Octopus Hi Simon, Have you tried installing them with yum? On Wed, Apr 22, 2020 at 6:16 PM Simon Sutter mailto:ssut...@hosttech.ch><mailto:ssut...@hosttech.ch<mailto:ssut...@hosttech.ch>>> wrote: Hello everybody In octopus there are some interesting looking features, so I tried to upgrading my Centos 7 test nodes, according to: https://docs.ceph.com/docs/master/releases/octopus/ Everything went fine and the cluster is healthy. To test out the new dashboard functions, I tried to install it, but there are missing dependencies: yum install ceph-mgr-dashboard.noarch . --> Finished Dependency Resolution Error: Package: 2:ceph-mgr-dashboard-15.2.1-0.el7.noarch (Ceph-noarch) Requires: python3-routes Error: Package: 2:ceph-mgr-dashboard-15.2.1-0.el7.noarch (Ceph-noarch) Requires: python3-jwt Error: Package: 2:ceph-mgr-dashboard-15.2.1-0.el7.noarch (Ceph-noarch) Requires: python3-cherrypy Installing them with pip3 does of course make no difference, because those are yum dependencies. Does anyone know a workaround? Do I have to upgrade to Centos 8 for this to work? Thanks in advance, Simon ___ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io<mailto:ceph-users@ceph.io>> To unsubscribe send an email to ceph-users-le...@ceph.io<mailto:ceph-users-le...@ceph.io><mailto:ceph-users-le...@ceph.io<mailto:ceph-users-le...@ceph.io>> ___ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-le...@ceph.io<mailto:ceph-users-le...@ceph.io> ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Upgrading to Octopus
Hello,
Yes I upgraded the system to Centos8 and now I can install the dashboard module.
But the problem now is, I cannot log in to the dashboard.
I deleted every cached file on my end and reinstalled the mgr and dashboard
several times.
If I try to log in with a wrong password, it tells me that it's wrong, but if i
use the right password, it just gives me a "500 Internal Server Error".
I enabled the debug mode for the mgr: ceph config set mgr
mgr/dashboard/log_level debug
But in the /var/log/ceph/ceph-mgr.node1.log it just tells me this generic
message (ips replaced with 0.0.0.0):
2020-04-28T14:11:15.191+0200 7f0baba8c700 0 [dashboard DEBUG request]
[:::0.0.0.0:61383] [POST] [None] /api/auth
2020-04-28T14:11:15.282+0200 7f0bcf164700 0 log_channel(cluster) log [DBG] :
pgmap v316: 273 pgs: 273 active+clean; 2.4 TiB data, 7.1 TiB used, 18 TiB / 25
TiB avail
2020-04-28T14:11:15.453+0200 7f0baba8c700 0 [dashboard DEBUG controllers.auth]
Login successful
2020-04-28T14:11:15.453+0200 7f0baba8c700 0 [dashboard ERROR request]
[:::0.0.0.0:61383] [POST] [500] [0.264s] [513.0B]
[100ecd9a-5d09-419f-8b9f-31bc3d4042b4] /api/auth
2020-04-28T14:11:15.453+0200 7f0baba8c700 0 [dashboard ERROR request]
[b'{"status": "500 Internal Server Error", "detail": "The server encountered an
unexpected condition which prevented it from fulfilling the request.",
"request_id": "100ecd9a-5d09-419f-8b9f-31bc3d4042b4"}
']
2020-04-28T14:11:15.454+0200 7f0baba8c700 0 [dashboard INFO request]
[:::0.0.0.0:61383] [POST] [500] [0.264s] [513.0B]
[100ecd9a-5d09-419f-8b9f-31bc3d4042b4] /api/auth
How can I find out, where the problem is?
Thanks in advance,
Simon
Von: gert.wieberd...@ziggo.nl
Gesendet: Donnerstag, 23. April 2020 20:34:58
An: ceph-users@ceph.io
Betreff: [ceph-users] Re: Upgrading to Octopus
Hello Simon,
I think that Khodayar is right. I managed to install a new Ceph cluster on
CentOS 8.1. Therefore you will need the ceph-el8.repo for the time being. For
some reason, "they" left the py3 packages you mentioned out of EPEL (as with
leveldb, but this package appeared luckily last week in EPEL).
Please find below the ceph-el8.repo file, which you have to create in
/etc/yum.repos.d/
[copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8]
name=Copr repo for ceph-el8 owned by ktdreyer
baseurl=https://download.copr.fedorainfracloud.org/results/ktdreyer/ceph-el8/epel-8-$basearch/
type=rpm-md
skip_if_unavailable=True
gpgcheck=1
gpgkey=https://download.copr.fedorainfracloud.org/results/ktdreyer/ceph-el8/pubkey.gpg
repo_gpgcheck=0
enabled=1
enabled_metadata=1
This repository - and CentOS 8.x - should have been sufficient to bring up a
fresh Ceph cluster.
Please let me know if you still have problems in configuring your Ceph cluster.
rgds,
-gw
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Upgrading to Octopus
Hello Gert,
I recreated the self signed certificate.
SELinux was disabled and I temporarely disabled the firewall.
It still doesn't work and there is no entry in journalctl -f.
Somewhere there is still something from the previous nautilus or centos7
installation, causing this problem.
I think I'll have to reinstall the node.
I'll update you.
Thanks and kind regards,
Simon
Von: Gert Wieberdink
Gesendet: Dienstag, 28. April 2020 21:16:10
An: Simon Sutter; ceph-users@ceph.io
Betreff: Re: [ceph-users] Re: Upgrading to Octopus
Sorry for the typo: must be journalctl -f instead of syslogctl -f.
-gw
On Tue, 2020-04-28 at 19:12 +, Gert Wieberdink wrote:
Hello Simon,ceph-mgr and dashboard installation should be
straightforward.
These are tough ones (internal server error 500). Did you create a self
signed cert for dashboard?Did you check firewalld (port 8443) and/or
SELinux? Does syslogctl -f show anything?
rgds,-gw
On Tue, 2020-04-28 at 12:17 +0000, Simon Sutter wrote:
Hello,
Yes I upgraded the system to Centos8 and now I can install the
dashboard module.
But the problem now is, I cannot log in to the dashboard.
I deleted every cached file on my end and reinstalled the mgr and
dashboard several times.
If I try to log in with a wrong password, it tells me that it's
wrong, but if i use the right password, it just gives me a "500
Internal Server Error".
I enabled the debug mode for the mgr: ceph config set mgr
mgr/dashboard/log_level debug
But in the /var/log/ceph/ceph-mgr.node1.log it just tells me this
generic message (ips replaced with 0.0.0.0):
2020-04-28T14:11:15.191+0200 7f0baba8c700 0 [dashboard DEBUG
request] [:::0.0.0.0:61383] [POST] [None] /api/auth
2020-04-28T14:11:15.282+0200 7f0bcf164700 0 log_channel(cluster) log
[DBG] : pgmap v316: 273 pgs: 273 active+clean; 2.4 TiB data, 7.1 TiB
used, 18 TiB / 25 TiB avail
2020-04-28T14:11:15.453+0200 7f0baba8c700 0 [dashboard DEBUG
controllers.auth] Login successful
2020-04-28T14:11:15.453+0200 7f0baba8c700 0 [dashboard ERROR
request] [:::0.0.0.0:61383] [POST] [500] [0.264s] [513.0B]
[100ecd9a-5d09-419f-8b9f-31bc3d4042b4] /api/auth
2020-04-28T14:11:15.453+0200 7f0baba8c700 0 [dashboard ERROR
request] [b'{"status": "500 Internal Server Error", "detail": "The
server encountered an unexpected condition which prevented it from
fulfilling the request.", "request_id": "100ecd9a-5d09-419f-8b9f-
31bc3d4042b4"}
']
2020-04-28T14:11:15.454+0200 7f0baba8c700 0 [dashboard INFO request]
[:::0.0.0.0:61383] [POST] [500] [0.264s] [513.0B] [100ecd9a-5d09-
419f-8b9f-31bc3d4042b4] /api/auth
How can I find out, where the problem is?
Thanks in advance,
Simon
Von:
<mailto:gert.wieberd...@ziggo.nl>
gert.wieberd...@ziggo.nl
<
<mailto:gert.wieberd...@ziggo.nl>
gert.wieberd...@ziggo.nl
>
Gesendet: Donnerstag, 23. April 2020 20:34:58
An:
<mailto:ceph-users@ceph.io>
ceph-users@ceph.io
Betreff: [ceph-users] Re: Upgrading to Octopus
Hello Simon,
I think that Khodayar is right. I managed to install a new Ceph
cluster on CentOS 8.1. Therefore you will need the ceph-el8.repo for
the time being. For some reason, "they" left the py3 packages you
mentioned out of EPEL (as with leveldb, but this package appeared
luckily last week in EPEL).
Please find below the ceph-el8.repo file, which you have to create in
/etc/yum.repos.d/
[copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8]
name=Copr repo for ceph-el8 owned by ktdreyer
baseurl=
<https://download.copr.fedorainfracloud.org/results/ktdreyer/ceph-el8/epel-8-$basearch/>
https://download.copr.fedorainfracloud.org/results/ktdreyer/ceph-el8/epel-8-$basearch/
type=rpm-md
skip_if_unavailable=True
gpgcheck=1
gpgkey=
<https://download.copr.fedorainfracloud.org/results/ktdreyer/ceph-el8/pubkey.gpg>
https://download.copr.fedorainfracloud.org/results/ktdreyer/ceph-el8/pubkey.gpg
repo_gpgcheck=0
enabled=1
enabled_metadata=1
This repository - and CentOS 8.x - should have been sufficient to
bring up a fresh Ceph cluster.
Please let me know if you still have problems in configuring your
Ceph cluster.
rgds,
-gw
___
ceph-users mailing list --
<mailto:ceph-users@ceph.io>
ceph-users@ceph.io
To unsubscribe send an email to
<mailto:ceph-users-le...@ceph.io>
ceph-users-le...@ceph.io
___
ceph-users mailing list --
<mailto:ceph-users@ceph.io>
ceph-users@ceph.io
To unsubscribe send an email to
<mailto:ceph-users-le...@ceph.io>
ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Add lvm in cephadm
Hello Everyone, The new cephadm is giving me a headache. I'm setting up a new testenvironment, where I have to use lvm partitions, because I don't have more Hardware. I could't find any information about the compatibility of existing lvm partitions and cephadm/octopus. I tried the old method, where I had to use ceph-volume to use an existing lvm: ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Add lvm in cephadm
Sorry I missclicked, here the second part: ceph-volume --cluster ceph lvm prepare --data /dev/centos_node1/ceph But that gives me just: Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new f3b442b1-68f7-456a-9991-92254e7c9c30 stderr: [errno 13] RADOS permission denied (error connecting to the cluster) --> RuntimeError: Unable to create a new OSD id So, how do I create an osd out of an existing lvm partition with cephadm? Thanks in advance Simon Von: Simon Sutter Gesendet: Dienstag, 5. Mai 2020 10:43:10 An: ceph-users@ceph.io Betreff: [ceph-users] Add lvm in cephadm Hello Everyone, The new cephadm is giving me a headache. I'm setting up a new testenvironment, where I have to use lvm partitions, because I don't have more Hardware. I could't find any information about the compatibility of existing lvm partitions and cephadm/octopus. I tried the old method, where I had to use ceph-volume to use an existing lvm: ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io hosttech GmbH | Simon Sutter hosttech.ch<https://www.hosttech.ch> WE LOVE TO HOST YOU. create your own website! more information & online-demo: www.website-creator.ch<http://www.website-creator.ch<http://www.website-creator.ch<http://www.website-creator.ch>> ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Add lvm in cephadm
Hello, Thank you very much Joshua, it worked. I have set up three nodes with the cephadm tool, which was very easy. But I asked myself, what if node 1 goes down? Before cephadm I just could manage everything from the other nodes with the ceph commands. Now I'm a bit stuck, because this cephadm container is just running on one node. I've installed it on the second one, but i'm getting a "[errno 13] RADOS permission denied (error connecting to the cluster)". Do I need some special "cephadm" keyring from the first node? Which one? And where to put it? Caphadm might be an easy to handle solution, but for me as a beginner, the added layer is very complicated to get in. We are trying to build a new Ceph cluster (never got in touch with it before) but I might not go with octopus, but instead use nautilus with ceph-deploy. That's a bit easyer to understand, and the documentation out there is way better. Thanks in advance, Simon Von: Joshua Schmid Gesendet: Dienstag, 5. Mai 2020 16:39:29 An: Simon Sutter Cc: ceph-users@ceph.io Betreff: Re: [ceph-users] Re: Add lvm in cephadm On 20/05/05 08:46, Simon Sutter wrote: > Sorry I missclicked, here the second part: > > > ceph-volume --cluster ceph lvm prepare --data /dev/centos_node1/ceph > But that gives me just: > > Running command: /usr/bin/ceph-authtool --gen-print-key > Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd > --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new > f3b442b1-68f7-456a-9991-92254e7c9c30 > stderr: [errno 13] RADOS permission denied (error connecting to the cluster) > --> RuntimeError: Unable to create a new OSD id Hey Simon, This still works but is now encapsulated in a cephadm command. ceph orch daemon add osd : so in your case: ceph orch daemon add osd $host:centos_node1/ceph hth -- Joshua Schmid Software Engineer SUSE Enterprise Storage ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: how to restart daemons on 15.2 on Debian 10
Hello Michael,
I had the same problems. It's very unfamiliar, if you never worked with the
cephadm tool.
The Way I'm doing it is to go into the cephadm container:
# cephadm shell
Here you can list all containers (for each service, one container) with the
orchestration tool:
# ceph orch ps
and then restart it with the orchestration tool:
# ceph orch restart {name from ceph orch ps}
Hope it helps.
Ceers,
Simon
Von: Ml Ml
Gesendet: Freitag, 15. Mai 2020 12:27:09
An: ceph-users
Betreff: [ceph-users] how to restart daemons on 15.2 on Debian 10
Hello List,
how do you restart daemons (mgr, mon, osd) on 15.2.1?
It used to be something like:
systemctl stop ceph-osd@10
Or:
systemctl start ceph-mon@ceph03
however, those command do nothing on my setup.
Is this because i use cephadm and that docker stuff?
The Logs also seem to be missing.
/var/log/ceph/5436dd5d-83d4-4dc8-a93b-60ab5db145df is pretty empty.
I feel like i am missing a lot of documentation here? Can anyone point
me to my missing parts?
Thanks a lot.
Cheers,
Michael
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] mds container dies during deployment
Hello everyone I've got a fresh ceph octopus installation and I'm trying to set up a cephfs with erasure code configuration. The metadata pool was set up as default. The erasure code pool was set up with this command: -> ceph osd pool create ec-data_fs 128 erasure default Enabled overwrites: -> ceph osd pool set ec-data_fs allow_ec_overwrites true And create fs: -> ceph fs new ec-data_fs meta_fs ec-data_fs --force Then I tried deploying the mds, but this fails: -> ceph orch daemon add mds ec-data_fs magma01 returns: -> Deployed mds.ec-data_fs.magma01.ujpcly on host 'magma01' The mds daemon is not there. Aparently the container dies without any information, as seen in the journal: May 25 16:11:56 magma01 podman[9348]: 2020-05-25 16:11:56.670510456 +0200 CEST m=+0.186462913 container create 0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90 (image=docker.io/ceph/ceph:v15, name=competent_cori) May 25 16:11:56 magma01 systemd[1]: Started libpod-conmon-0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90.scope. May 25 16:11:56 magma01 systemd[1]: Started libcontainer container 0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90. May 25 16:11:57 magma01 podman[9348]: 2020-05-25 16:11:57.112182262 +0200 CEST m=+0.628134873 container init 0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90 (image=docker.io/ceph/ceph:v15, name=competent_cori) May 25 16:11:57 magma01 podman[9348]: 2020-05-25 16:11:57.137011897 +0200 CEST m=+0.652964354 container start 0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90 (image=docker.io/ceph/ceph:v15, name=competent_cori) May 25 16:11:57 magma01 podman[9348]: 2020-05-25 16:11:57.137110412 +0200 CEST m=+0.653062853 container attach 0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90 (image=docker.io/ceph/ceph:v15, name=competent_cori) May 25 16:11:57 magma01 systemd[1]: libpod-0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90.scope: Consumed 327ms CPU time May 25 16:11:57 magma01 podman[9348]: 2020-05-25 16:11:57.182968802 +0200 CEST m=+0.698921275 container died 0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90 (image=docker.io/ceph/ceph:v15, name=competent_cori) May 25 16:11:57 magma01 podman[9348]: 2020-05-25 16:11:57.413743787 +0200 CEST m=+0.929696266 container remove 0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90 (image=docker.io/ceph/ceph:v15, name=competent_cori) Can someone help me debugging this? Cheers Simon ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: mds container dies during deployment
Hello, Didn't read the right one: https://docs.ceph.com/docs/master/cephadm/install/#deploy-mdss There it says, how to do it right. The command I was using, was just to add a mds daemon if you have already one. Hopes it helps others. Cheers, Simon Von: Simon Sutter Gesendet: Montag, 25. Mai 2020 16:44:54 An: ceph-users@ceph.io Betreff: [ceph-users] mds container dies during deployment Hello everyone I've got a fresh ceph octopus installation and I'm trying to set up a cephfs with erasure code configuration. The metadata pool was set up as default. The erasure code pool was set up with this command: -> ceph osd pool create ec-data_fs 128 erasure default Enabled overwrites: -> ceph osd pool set ec-data_fs allow_ec_overwrites true And create fs: -> ceph fs new ec-data_fs meta_fs ec-data_fs --force Then I tried deploying the mds, but this fails: -> ceph orch daemon add mds ec-data_fs magma01 returns: -> Deployed mds.ec-data_fs.magma01.ujpcly on host 'magma01' The mds daemon is not there. Aparently the container dies without any information, as seen in the journal: May 25 16:11:56 magma01 podman[9348]: 2020-05-25 16:11:56.670510456 +0200 CEST m=+0.186462913 container create 0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90 (image=docker.io/ceph/ceph:v15, name=competent_cori) May 25 16:11:56 magma01 systemd[1]: Started libpod-conmon-0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90.scope. May 25 16:11:56 magma01 systemd[1]: Started libcontainer container 0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90. May 25 16:11:57 magma01 podman[9348]: 2020-05-25 16:11:57.112182262 +0200 CEST m=+0.628134873 container init 0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90 (image=docker.io/ceph/ceph:v15, name=competent_cori) May 25 16:11:57 magma01 podman[9348]: 2020-05-25 16:11:57.137011897 +0200 CEST m=+0.652964354 container start 0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90 (image=docker.io/ceph/ceph:v15, name=competent_cori) May 25 16:11:57 magma01 podman[9348]: 2020-05-25 16:11:57.137110412 +0200 CEST m=+0.653062853 container attach 0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90 (image=docker.io/ceph/ceph:v15, name=competent_cori) May 25 16:11:57 magma01 systemd[1]: libpod-0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90.scope: Consumed 327ms CPU time May 25 16:11:57 magma01 podman[9348]: 2020-05-25 16:11:57.182968802 +0200 CEST m=+0.698921275 container died 0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90 (image=docker.io/ceph/ceph:v15, name=competent_cori) May 25 16:11:57 magma01 podman[9348]: 2020-05-25 16:11:57.413743787 +0200 CEST m=+0.929696266 container remove 0fdf8c508b330adac713ffb04c72b5df770277ad191d844888f7387f28e3cc90 (image=docker.io/ceph/ceph:v15, name=competent_cori) Can someone help me debugging this? Cheers Simon ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io hosttech GmbH | Simon Sutter hosttech.ch<https://www.hosttech.ch> WE LOVE TO HOST YOU. create your own website! more information & online-demo: www.website-creator.ch<http://www.website-creator.ch<http://www.website-creator.ch<http://www.website-creator.ch>> ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Ceph client on rhel6?
Hello again, I have a new question: We want to upgrade a server, with an os based on rhel6. The ceph cluster is atm on octopus. How can I install the client packages to mount cephfs and do a backup of the server? Is it even possible? Are the client packages from hammer compatible with the octopus release? Thanks in advance, Simon ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Deploy nfs from cephadm
Hello Ceph users, I'm trying to deploy nfs-ganesha with cephadm on octopus. According to the docs, it's as easy as running the command in the docs: https://docs.ceph.com/docs/master/cephadm/install/#deploying-nfs-ganesha ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Deploy nfs from cephadm
:CRIT :dbus_bus_get failed (Failed to connect to socket /run/dbus/system_bus_socket: No such file or d> Jun 02 15:17:48 testnode1 bash[63432]: 02/06/2020 13:17:48 : epoch 5ed6517a : testnode1 : ganesha.nfsd-1[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection Jun 02 15:17:48 testnode1 bash[63432]: 02/06/2020 13:17:48 : epoch 5ed6517a : testnode1 : ganesha.nfsd-1[main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs Jun 02 15:17:48 testnode1 bash[63432]: 02/06/2020 13:17:48 : epoch 5ed6517a : testnode1 : ganesha.nfsd-1[main] __Register_program :DISP :MAJ :Cannot register NFS V3 on UDP Jun 02 15:17:48 testnode1 podman[63531]: 2020-06-02 15:17:48.976740123 +0200 CEST m=+2.565960305 container died aab4e893b162a643181cea3a9f5f687aae236eb2f4a7f6fad27d503d1fdee893 (image=docker.io/ceph/ceph:v15, name=ceph-915cdf28-8f66-11e> Jun 02 15:17:49 testnode1 podman[63531]: 2020-06-02 15:17:49.051611802 +0200 CEST m=+2.640831963 container remove aab4e893b162a643181cea3a9f5f687aae236eb2f4a7f6fad27d503d1fdee893 (image=docker.io/ceph/ceph:v15, name=ceph-915cdf28-8f66-1> Jun 02 15:17:49 testnode1 systemd[1]: ceph-915cdf28-8f66-11ea-bb83-ac1f6b4cd516@nfs.cephnfs.testnode1.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Jun 02 15:17:49 testnode1 podman[64059]: 2020-06-02 15:17:49.158593029 +0200 CEST m=+0.082356027 container create d55e6b6797ad129c6de503c0f6fac6b127fcf978ae44b636a54b0b93ad5010b2 (image=docker.io/ceph/ceph:v15, name=ceph-915cdf28-8f66-1> Jun 02 15:17:49 testnode1 podman[64059]: 2020-06-02 15:17:49.481433978 +0200 CEST m=+0.405196986 container init d55e6b6797ad129c6de503c0f6fac6b127fcf978ae44b636a54b0b93ad5010b2 (image=docker.io/ceph/ceph:v15, name=ceph-915cdf28-8f66-11e> Jun 02 15:17:49 testnode1 podman[64059]: 2020-06-02 15:17:49.495210618 +0200 CEST m=+0.418973669 container start d55e6b6797ad129c6de503c0f6fac6b127fcf978ae44b636a54b0b93ad5010b2 (image=docker.io/ceph/ceph:v15, name=ceph-915cdf28-8f66-11> Jun 02 15:17:49 testnode1 podman[64059]: 2020-06-02 15:17:49.495293786 +0200 CEST m=+0.419056865 container attach d55e6b6797ad129c6de503c0f6fac6b127fcf978ae44b636a54b0b93ad5010b2 (image=docker.io/ceph/ceph:v15, name=ceph-915cdf28-8f66-1> Jun 02 15:17:49 testnode1 bash[64057]: rados_connect: -13 Jun 02 15:17:49 testnode1 bash[64057]: Can't connect to cluster: -13 Jun 02 15:17:49 testnode1 podman[64059]: 2020-06-02 15:17:49.55130226 +0200 CEST m=+0.475065324 container died d55e6b6797ad129c6de503c0f6fac6b127fcf978ae44b636a54b0b93ad5010b2 (image=docker.io/ceph/ceph:v15, name=ceph-915cdf28-8f66-11ea> Jun 02 15:17:49 testnode1 podman[64059]: 2020-06-02 15:17:49.633569478 +0200 CEST m=+0.557332529 container remove d55e6b6797ad129c6de503c0f6fac6b127fcf978ae44b636a54b0b93ad5010b2 (image=docker.io/ceph/ceph:v15, name=ceph-915cdf28-8f66-1> Jun 02 15:17:49 testnode1 systemd[1]: ceph-915cdf28-8f66-11ea-bb83-ac1f6b4cd516@nfs.cephnfs.testnode1.service: Failed with result 'exit-code'. I can see, one container fails to connect to the cluster, but wher can I find out why? Thanks in Advance and sorry for the split mail, Simon Von: Simon Sutter Gesendet: Dienstag, 2. Juni 2020 16:26:15 An: ceph-users@ceph.io Betreff: [ceph-users] Deploy nfs from cephadm Hello Ceph users, I'm trying to deploy nfs-ganesha with cephadm on octopus. According to the docs, it's as easy as running the command in the docs: https://docs.ceph.com/docs/master/cephadm/install/#deploying-nfs-ganesha ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io hosttech GmbH | Simon Sutter hosttech.ch<https://www.hosttech.ch> WE LOVE TO HOST YOU. create your own website! more information & online-demo: www.website-creator.ch<http://www.website-creator.ch<http://www.website-creator.ch<http://www.website-creator.ch>> ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Deploy nfs from cephadm
Hello,
Thank you very much.
I was a bit worried about all the other messages, especially those two from a
different container (started before the right one?):
Jun 03 08:22:23 testnode1 bash[3169]: rados_connect: -13
Jun 03 08:22:23 testnode1 bash[3169]: Can't connect to cluster: -13
Nevertheless, installing and starting rpcbind did it for me.
After I configured the ganesha service in
"/var/lib/ceph/{UUID}/nfs.cephnfs.testnode1/etc/ganesha/ganesha.conf" I was
able to start and accesss cephfs from a nfs client.
Maybe a hint in the docs, to install rpcbind, or to enable just nfsv4 would
also help others.
Best Regards,
Simon
Von: Michael Fritch
Gesendet: Dienstag, 2. Juni 2020 22:33:46
An: ceph-users@ceph.io
Betreff: [ceph-users] Re: Deploy nfs from cephadm
Hi,
Do you have a running rpcbind service?
$ systemctl status rpcbind
NFSv3 requires rpcbind, but this dependency will be removed in a later release
of Octopus. I've updated the tracker with more detail.
Hope this helps,
-Mike
John Zachary Dover wrote:
> I've created a docs tracker ticket for this issue:
>
> https://tracker.ceph.com/issues/45819
Bug #45819: Possible error in deploying-nfs-ganesha docs - Orchestrator -
Ceph<https://tracker.ceph.com/issues/45819>
tracker.ceph.com
Redmine
>
> Zac
> Ceph Docs
>
> On Wed, Jun 3, 2020 at 12:34 AM Simon Sutter
> > Sorry, allways the wrong button...
> >
> > So I ran the command:
> > ceph orch apply nfs cephnfs cephfs.backuptest.data
> >
> > And there is now a not working container:
> > ceph orch ps:
> > nfs.cephnfs.testnode1testnode1 error 6m ago 71m
> >docker.io/ceph/ceph:v15
> >
> >
> > journalctl tells me this:
> >
> > Jun 02 15:17:45 testnode1 systemd[1]: Starting Ceph nfs.cephnfs.testnode1
> > for 915cdf28-8f66-11ea-bb83-ac1f6b4cd516...
> > Jun 02 15:17:45 testnode1 podman[63413]: Error: no container with name or
> > ID ceph-915cdf28-8f66-11ea-bb83-ac1f6b4cd516-nfs.cephnfs.testnode1 found:
> > no such container
> > Jun 02 15:17:45 testnode1 systemd[1]: Started Ceph nfs.cephnfs.testnode1
> > for 915cdf28-8f66-11ea-bb83-ac1f6b4cd516.
> > Jun 02 15:17:45 testnode1 podman[63434]: 2020-06-02 15:17:45.867685349
> > +0200 CEST m=+0.080338785 container create
> > 7290cc21b0e2498876773f1ef2a2be24abf62e9ed058d60e79c8f3d3d3e9e0d3 (image=
> > docker.io/ceph/ceph:v15, name=ceph-915cdf28-8f66-1>
> > Jun 02 15:17:46 testnode1 podman[63434]: 2020-06-02 15:17:46.196760186
> > +0200 CEST m=+0.409413617 container init
> > 7290cc21b0e2498876773f1ef2a2be24abf62e9ed058d60e79c8f3d3d3e9e0d3 (image=
> > docker.io/ceph/ceph:v15, name=ceph-915cdf28-8f66-11e>
> > Jun 02 15:17:46 testnode1 podman[63434]: 2020-06-02 15:17:46.211149759
> > +0200 CEST m=+0.423803191 container start
> > 7290cc21b0e2498876773f1ef2a2be24abf62e9ed058d60e79c8f3d3d3e9e0d3 (image=
> > docker.io/ceph/ceph:v15, name=ceph-915cdf28-8f66-11>
> > Jun 02 15:17:46 testnode1 podman[63434]: 2020-06-02 15:17:46.21122888
> > +0200 CEST m=+0.423882373 container attach
> > 7290cc21b0e2498876773f1ef2a2be24abf62e9ed058d60e79c8f3d3d3e9e0d3 (image=
> > docker.io/ceph/ceph:v15, name=ceph-915cdf28-8f66-11>
> > Jun 02 15:17:46 testnode1 bash[63432]: rados_connect: -13
> > Jun 02 15:17:46 testnode1 bash[63432]: Can't connect to cluster: -13
> > Jun 02 15:17:46 testnode1 podman[63434]: 2020-06-02 15:17:46.300445833
> > +0200 CEST m=+0.513099326 container died
> > 7290cc21b0e2498876773f1ef2a2be24abf62e9ed058d60e79c8f3d3d3e9e0d3 (image=
> > docker.io/ceph/ceph:v15, name=ceph-915cdf28-8f66-11e>
> > Jun 02 15:17:46 testnode1 podman[63434]: 2020-06-02 15:17:46.391730251
> > +0200 CEST m=+0.604383723 container remove
> > 7290cc21b0e2498876773f1ef2a2be24abf62e9ed058d60e79c8f3d3d3e9e0d3 (image=
> > docker.io/ceph/ceph:v15, name=ceph-915cdf28-8f66-1>
> > Jun 02 15:17:46 testnode1 podman[63531]: 2020-06-02 15:17:46.496154808
> > +0200 CEST m=+0.085374929 container create
> > aab4e893b162a643181cea3a9f5f687aae236eb2f4a7f6fad27d503d1fdee893 (image=
> > docker.io/ceph/ceph:v15, name=ceph-915cdf28-8f66-1>
> > Jun 02 15:17:46 testnode1 podman[63531]: 2020-06-02 15:17:46.81399203
> > +0200 CEST m=+0.403212198 container init
> > aab4e893b162a643181cea3a9f5f687aae236eb2f4a7f6fad27d503d1fdee893 (image=
> > docker.io/ceph/ceph:v15, name=ceph-915cdf28-8f66-11ea>
> > Jun 02 15:17:46 testnode1 podman[63531]: 2020-06-02 15:17:46.828546918
> > +0200 CEST m=+0.417767036 container start
> > aab4e893b162a643181cea3a9f5f687aae2
[ceph-users] Change mon bind address / Change IPs with the orchestrator
Hello,
I think I missunderstood the internal / public network concepts in the docs
https://docs.ceph.com/docs/master/rados/configuration/network-config-ref/.
Now there are two questions:
- Is it somehow possible to bind the MON daemon to 0.0.0.0?
I tried it with manually add the ip in /var/lib/ceph/{UUID}/mon.node01/config
[mon.node01]
public bind addr = 0.0.0.0
But that does not work, in netstat I can see, the mon still binds to it's
internal ip. Is this an expected behaviour?
If I set this value to the public ip, the other nodes cannot communicate with
it, so this leads to the next question:
- What's the Right way to correct the problem with the orchestrator?
So the correct way to configure the ip's, would be to set every mon, mds and so
on, to the public ip and just let the osd's stay on their internal ip.
(described here
https://docs.ceph.com/docs/master/rados/configuration/network-config-ref/)
Do I have to remove every daemon and redeploy them with "ceph orch daemon rm" /
"ceph orch apply"?
Or do I have to go to every node and manually apply the settings in the daemon
config file?
Thanks in advance,
Simon
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Change mon bind address / Change IPs with the orchestrator
Hello,
Ok, thanks Wido.
I have now sorted out the correct network configuration and deployed new mon's
with the new IP.
Everything is now on the new IP and works so far.
Simon
Von: Wido den Hollander
Gesendet: Donnerstag, 4. Juni 2020 08:47:29
An: Simon Sutter; ceph-users@ceph.io
Betreff: Re: [ceph-users] Change mon bind address / Change IPs with the
orchestrator
On 6/3/20 4:49 PM, Simon Sutter wrote:
> Hello,
>
>
> I think I missunderstood the internal / public network concepts in the docs
> https://docs.ceph.com/docs/master/rados/configuration/network-config-ref/.
>
> Now there are two questions:
>
> - Is it somehow possible to bind the MON daemon to 0.0.0.0?
No
> I tried it with manually add the ip in /var/lib/ceph/{UUID}/mon.node01/config
Won't work :-)
>
>
> [mon.node01]
> public bind addr = 0.0.0.0
>
>
> But that does not work, in netstat I can see, the mon still binds to it's
> internal ip. Is this an expected behaviour?
>
Yes. This is not an orchestrator thing, this is how the MONs work. They
need to bind to a specific IP and that can't be 0.0.0.0
You then need to make sure proper routing is in place so all clients and
OSDs can talk to the MONs.
So don't attempt anything like NAT. Make sure everything works with
proper IP-routing.
Wido
> If I set this value to the public ip, the other nodes cannot communicate with
> it, so this leads to the next question:
>
> - What's the Right way to correct the problem with the orchestrator?
> So the correct way to configure the ip's, would be to set every mon, mds and
> so on, to the public ip and just let the osd's stay on their internal ip.
> (described here
> https://docs.ceph.com/docs/master/rados/configuration/network-config-ref/)
>
> Do I have to remove every daemon and redeploy them with "ceph orch daemon rm"
> / "ceph orch apply"?
>
> Or do I have to go to every node and manually apply the settings in the
> daemon config file?
>
>
> Thanks in advance,
>
>
> Simon
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Cephadm and Ceph versions
Hello Andy, I had mixed experiences with cephadm. What I would do: Check if all your daemons indeed are running in the according containers on every node. You can check it with "ceph orch ps" If that is the case, you can get rid of the old rpms and install the new ceph-common v15 rpm. You do not need all the rpms you had before, because everything should now be inside those containers. Technically you don't need any ceph rpm installed on any of the nodes except for cephadm, but with the ceph-common rpm the conversion is a bit easier. Please don't do this directly on a production running system and test it first in a testenvironment. Simon Von: bioh...@yahoo.com Gesendet: Freitag, 5. Juni 2020 11:18:41 An: ceph-users@ceph.io Betreff: [ceph-users] Cephadm and Ceph versions Hi I had a cluster on v13 (mimic) and have converted it to Octopus (15.2.3) and using Cephadm. In the dashboard is showing as all v15 What do I need to do with the Ceph rpms that are installed as they are all Ceph version 13. Do I remove them and install Ceph rpms with version 15 ? Regards Andy ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Octopus: orchestrator not working correctly with nfs
Hello I know that nfs on octopus is still a bit under development. I'm trying to deploy nfs daemons and have some issues with the orchestartor. For the other daemons, for example monitors, I can issue the command "ceph orch apply mon 3" This will tell the orchestrator to deploy or remove monitor daemons until there are three of them. The command does not work with nfs, and now the orchestrator is a bit missconfigured... And with missconfigured I mean, that I have now a nfs daemon on node 1 and the orchestrator wants to create another one on node 1 but with wrong settings (it fails). Also a "ceph orch apply nfs –unconfigured" does not work, so I can't manually manage the nfs containers. Is there a manual way to tell ceph orch, to not create or remove nfs daemons? then I would be able to set them up manually. Or a manual way of configuring the orchestrator so it does the right thing. Thanks in advance Simon ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Octopus: orchestrator not working correctly with nfs
Hello, Did I not provide enough information, or simply nobody knows how to solve the problem? Should I write to the ceph tracker or does this just produce unnecessary overhead? Thanks in advance, Simon Von: Simon Sutter Gesendet: Montag, 8. Juni 2020 10:56:00 An: ceph-users@ceph.io Betreff: [ceph-users] Octopus: orchestrator not working correctly with nfs Hello I know that nfs on octopus is still a bit under development. I'm trying to deploy nfs daemons and have some issues with the orchestartor. For the other daemons, for example monitors, I can issue the command "ceph orch apply mon 3" This will tell the orchestrator to deploy or remove monitor daemons until there are three of them. The command does not work with nfs, and now the orchestrator is a bit missconfigured... And with missconfigured I mean, that I have now a nfs daemon on node 1 and the orchestrator wants to create another one on node 1 but with wrong settings (it fails). Also a "ceph orch apply nfs –unconfigured" does not work, so I can't manually manage the nfs containers. Is there a manual way to tell ceph orch, to not create or remove nfs daemons? then I would be able to set them up manually. Or a manual way of configuring the orchestrator so it does the right thing. Thanks in advance Simon ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io hosttech GmbH | Simon Sutter hosttech.ch<https://www.hosttech.ch> WE LOVE TO HOST YOU. create your own website! more information & online-demo: www.website-creator.ch<http://www.website-creator.ch<http://www.website-creator.ch<http://www.website-creator.ch>> ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Octopus: orchestrator not working correctly with nfs
Hello, Well, yes and no, In the stability section (https://docs.ceph.com/docs/octopus/cephadm/stability/) there is written, it's still under development. But in the set-up docs it's written whitout any hint of development: https://docs.ceph.com/docs/octopus/cephadm/install/#deploying-nfs-ganesha. If the orchestrator is not working, then this would not be a problem for me, because I could deploy them manually. But at the moment, the orchestrator is trying to deploy daemons and I don't know how to stop it. Regards Simon Von: Amudhan P Gesendet: Donnerstag, 11. Juni 2020 09:32:36 An: Simon Sutter Cc: ceph-users@ceph.io Betreff: Re: [ceph-users] Re: Octopus: orchestrator not working correctly with nfs Hi, I have not worked with orchestrator but I remember I read somewhere that NFS implementation is not supported. Refer Cephadm documentation and for NFS you have configure nfs Ganesha. You can manage NFS thru dashboard but for that you have initial config in dashboard and in nfsganaesha you have refer it. Regards Amudhan On Thu 11 Jun, 2020, 11:40 AM Simon Sutter, mailto:ssut...@hosttech.ch>> wrote: Hello, Did I not provide enough information, or simply nobody knows how to solve the problem? Should I write to the ceph tracker or does this just produce unnecessary overhead? Thanks in advance, Simon ________ Von: Simon Sutter mailto:ssut...@hosttech.ch>> Gesendet: Montag, 8. Juni 2020 10:56:00 An: ceph-users@ceph.io<mailto:ceph-users@ceph.io> Betreff: [ceph-users] Octopus: orchestrator not working correctly with nfs Hello I know that nfs on octopus is still a bit under development. I'm trying to deploy nfs daemons and have some issues with the orchestartor. For the other daemons, for example monitors, I can issue the command "ceph orch apply mon 3" This will tell the orchestrator to deploy or remove monitor daemons until there are three of them. The command does not work with nfs, and now the orchestrator is a bit missconfigured... And with missconfigured I mean, that I have now a nfs daemon on node 1 and the orchestrator wants to create another one on node 1 but with wrong settings (it fails). Also a "ceph orch apply nfs –unconfigured" does not work, so I can't manually manage the nfs containers. Is there a manual way to tell ceph orch, to not create or remove nfs daemons? then I would be able to set them up manually. Or a manual way of configuring the orchestrator so it does the right thing. Thanks in advance Simon ___ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-le...@ceph.io<mailto:ceph-users-le...@ceph.io> hosttech GmbH | Simon Sutter hosttech.ch<http://hosttech.ch><https://www.hosttech.ch> WE LOVE TO HOST YOU. create your own website! more information & online-demo: www.website-creator.ch<http://www.website-creator.ch><http://www.website-creator.ch<http://www.website-creator.ch<http://www.website-creator.ch>> ___ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-le...@ceph.io<mailto:ceph-users-le...@ceph.io> ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Octopus: orchestrator not working correctly with nfs
Hello, You just copied the same message. I'll make a ticket in the tracker. Regards, Simon Von: Amudhan P Gesendet: Donnerstag, 11. Juni 2020 09:32:36 An: Simon Sutter Cc: ceph-users@ceph.io Betreff: Re: [ceph-users] Re: Octopus: orchestrator not working correctly with nfs Hi, I have not worked with orchestrator but I remember I read somewhere that NFS implementation is not supported. Refer Cephadm documentation and for NFS you have configure nfs Ganesha. You can manage NFS thru dashboard but for that you have initial config in dashboard and in nfsganaesha you have refer it. Regards Amudhan On Thu 11 Jun, 2020, 11:40 AM Simon Sutter, mailto:ssut...@hosttech.ch>> wrote: Hello, Did I not provide enough information, or simply nobody knows how to solve the problem? Should I write to the ceph tracker or does this just produce unnecessary overhead? Thanks in advance, Simon ____ Von: Simon Sutter mailto:ssut...@hosttech.ch>> Gesendet: Montag, 8. Juni 2020 10:56:00 An: ceph-users@ceph.io<mailto:ceph-users@ceph.io> Betreff: [ceph-users] Octopus: orchestrator not working correctly with nfs Hello I know that nfs on octopus is still a bit under development. I'm trying to deploy nfs daemons and have some issues with the orchestartor. For the other daemons, for example monitors, I can issue the command "ceph orch apply mon 3" This will tell the orchestrator to deploy or remove monitor daemons until there are three of them. The command does not work with nfs, and now the orchestrator is a bit missconfigured... And with missconfigured I mean, that I have now a nfs daemon on node 1 and the orchestrator wants to create another one on node 1 but with wrong settings (it fails). Also a "ceph orch apply nfs –unconfigured" does not work, so I can't manually manage the nfs containers. Is there a manual way to tell ceph orch, to not create or remove nfs daemons? then I would be able to set them up manually. Or a manual way of configuring the orchestrator so it does the right thing. Thanks in advance Simon ___ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-le...@ceph.io<mailto:ceph-users-le...@ceph.io> hosttech GmbH | Simon Sutter hosttech.ch<http://hosttech.ch><https://www.hosttech.ch> WE LOVE TO HOST YOU. create your own website! more information & online-demo: www.website-creator.ch<http://www.website-creator.ch><http://www.website-creator.ch<http://www.website-creator.ch<http://www.website-creator.ch>> ___ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-le...@ceph.io<mailto:ceph-users-le...@ceph.io> ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Hello,
When you deploy ceph to other nodes with the orchestrator, they "just" have the
containers you deployed to them.
This means in your case, you started the monitor container on ceph101 and you
must have installed at least the ceph-common package (else the ceph command
would not work).
If you enter the command ceph -s there is no configuration file, nor is there a
key file (ceph does not know, where to connect).
Ceph's configuration directory defaults to /etc/ceph/ (should be empty or not
exist on ceph101).
So in your case, you can either create the configuration files manually (read
throug "ceph auth" and how the config and the keyring file should look like),
or just copy the ceph.conf and the admin keyring to /etc/ceph/ on ceph101.
Regards,
Simon
Von: cemzafer
Gesendet: Montag, 15. Juni 2020 21:27:30
An: ceph-us...@ceph.com
Betreff: [ceph-users] Error initializing cluster client: ObjectNotFound('RADOS
object not found (error calling conf_read_file)')
I have installed simple ceph system with two nodes (ceph100, ceph101)
with cephadm and ceph orch host add command. I put the ssh-copy-id -f -i
/etc/ceph/ceph.pub key to the second host (ceph101). I can execute the
ceph -s command from the first host(ceph100) but when I execute the
command in the second host(ceph101), I get the following error.
Error initializing cluster client: ObjectNotFound('RADOS object not
found (error calling conf_read_file)')
Also, when I execute the 'ceph orch ps' command the output seems
suspicious to me.
NAME HOST STATUSREFRESHED AGE
VERSIONIMAGE NAMEIMAGE ID CONTAINER ID
mon.ceph101ceph101 starting - -
Has anyone any idea what could be the problem or anyone give me a fine
link for the octopus cephadm installation?
Regards.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Current status of multipe cephfs
Hello, What is the current status, of using multiple cephfs? In octopuss I get lots of warnings, that this feature is still not fully tested, but the latest entry regarding multiple cephfs in the mailinglist is from about 2018. Is someone using multiple cephfs in production? Thanks in Advance, Simon ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Orchestrator: Cannot add node after mistake
Hello, I did a mistake, while deploying a new node on octopus. The node is a fresh installed Centos8 machine. Bevore I did a "ceph orch host add node08" I pasted the wrong command: ceph orch daemon add osd node08:cl_node08/ceph That did not return anything, so I tried to add the node first with the host add command, but now I get an error: Error ENOENT: New host node08 (node08) failed check: ['Traceback (most recent call last):', ' File "", line 4580, in ', ' File "", line 3592, in command_check_host', "UnboundLocalError: local variable 'container_path' referenced before assignment"] I'm not a developer, so I don't know where to look, and how to fix this. I tried to reboot every node, to see if it's just a cached problem, but no luck there. Do any of you know how to fix this? Thanks in advance, Simon ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Orchestrator: Cannot add node after mistake
After some hours of searching around in the docs and finaly taking a look at the sourcecode of cephadm, I figured out it has to do with the docker software (podman in my case). So I figured out that podman is not installed on the new node. I never installed podman on any node so I don't know if my installation was not good, or if cephadm just had a problem installing it. Also the message in the error is not very meaningful. For example, if I forget to install python3, the "python3 is not installed" message is very clear. Simon ________ Von: Simon Sutter Gesendet: Donnerstag, 18. Juni 2020 14:09:18 An: ceph-users@ceph.io Betreff: [ceph-users] Orchestrator: Cannot add node after mistake Hello, I did a mistake, while deploying a new node on octopus. The node is a fresh installed Centos8 machine. Bevore I did a "ceph orch host add node08" I pasted the wrong command: ceph orch daemon add osd node08:cl_node08/ceph That did not return anything, so I tried to add the node first with the host add command, but now I get an error: Error ENOENT: New host node08 (node08) failed check: ['Traceback (most recent call last):', ' File "", line 4580, in ', ' File "", line 3592, in command_check_host', "UnboundLocalError: local variable 'container_path' referenced before assignment"] I'm not a developer, so I don't know where to look, and how to fix this. I tried to reboot every node, to see if it's just a cached problem, but no luck there. Do any of you know how to fix this? Thanks in advance, Simon ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: How to ceph-volume on remote hosts?
Hello,
If you do it, like Sebastian told you, you would automatically deploy osd's.
For a beginner I would recommend to do it semi-automated, so you know a bit
more what's going on.
So first do the "ceph orch device ls" which should print every disk, in all
nodes.
Then I recommend to zap devices you want to use first, even if it says you can
use them:
ceph orch device zap {node} /dev/sd? --force
And then you can add devices one by one:
ceph orch daemon add osd {node}:/dev/sd?
Just to top it off.
Regards,
Simon
-Ursprüngliche Nachricht-
Von: Sebastian Wagner [mailto:swag...@suse.com]
Gesendet: Mittwoch, 24. Juni 2020 14:06
An: steven prothero ; ceph-users@ceph.io
Betreff: [ceph-users] Re: How to ceph-volume on remote hosts?
Am 24.06.20 um 05:15 schrieb steven prothero:
> Hello,
>
> I am new to CEPH and on a few test servers attempting to setup and
> learn a test ceph system.
>
> I started off the install with the "Cephadm" option and it uses podman
> containers.
> Followed steps here:
> https://docs.ceph.com/docs/master/cephadm/install/
>
> I ran the bootstrap, added remote hosts, added monitors and everything
> is looking good.
>
> Now I would like to add OSDs...
>
> On the bootstrapped server i did a :
>
> ceph-volume lvm prepare --data /dev/sda6
>and then the "activate" and "ceph orch daemon add osd (etc)" to add
> it and it works...
>
> But now I am ready to add OSDs on the remote nodes. I am not able to
> find documentation or examples on how to do :
>
> ceph-volume lvm prepare & activate steps on the remote hosts.
>
> How do we prepare & activate the remote hosts disks?
ceph orch apply osd
as described in
https://docs.ceph.com/docs/master/cephadm/install/#deploy-osds
should do the trick. In case it doesn't, what's the output of
ceph orch device ls
?
>
> Thank you very much for your input,
>
> Cheers
> Steve
> ___
> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
> email to ceph-users-le...@ceph.io
>
--
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg). Geschäftsführer: Felix Imendörffer
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Feedback of the used configuration
Hello, After two months of the "ceph try and error game", I finally managed to get an Octopuss cluster up and running. The unconventional thing about it is, it's just for hot backups, no virtual machines on there. All the nodes are without any caching ssd's, just plain hdd's. At the moment there are eight of them with a total of 50TB. We are planning to go up to 25 and bigger disks so we end on 300TB-400TB I decided to go with cephfs, because I don't have any experience in things like S3 and I need to read the same file system from more than one client. I made one cephfs with a replicated pool. On there I added erasure-coded pools to save some Storage. To add those pools, I did it with the setfattr command like this: setfattr -n ceph.dir.layout.pool -v ec_data_server1 /cephfs/nfs/server1 Some of our servers cannot use cephfs (old kernels, special OS's) so I have to use nfs. This is set up with the included ganesha-nfs. Exported is the /cephfs/nfs folder and clients can mount folders below this. There are two final questions: - Was it right to go with the way of "mounting" pools with setfattr, or should I have used multiple cephfs? First I was thinking about using multiple cephfs but there are warnings everywhere. The deeper I got in, the more it seems I would have been fine with multiple cephfs. - Is there a way I don't know, but it would be easier? I still don't know much about Rest, S3, RBD etc... so there may be a better way Other remarks are desired. Thanks in advance, Simon ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Feedback of the used configuration
Hello Paul, Thanks for the Answer. I took a look at the subvolumes, but they are a bit odd in my opinion. If I create one with a subvolume-group, the folder structure will look like this: /cephfs/volumes/group-name/subvolume-name/random-uuid/ And I have to issue two commands, first set the group and then set the volume name, but why so complicated? Wouldn't it be easier to just make subvolumes anywhere inside the cephfs? I can see the intended use for groups, but if I want to publish a pool in some different directory that's not possible (except for setfattr). Without first creating subvolume-groups, the orchestrator creates subvolumes in the /cephfs/volumes/_nogroup/subvolume-name/randmon-uuid/ folder. And the more important question is, why is there a new folder with a random uuid inside the subvolume? I try to understand the points the devs had, when they developed this, but for me, this is something I have to explain to some devs in our team and at the moment I can't. It is indeed easier to deploy but comes with much less flexibility. Maybe something to write in the tracker about? Thanks in advance, Simon Von: Paul Emmerich [mailto:paul.emmer...@croit.io] Gesendet: Mittwoch, 24. Juni 2020 17:35 An: Simon Sutter Cc: ceph-users@ceph.io Betreff: Re: [ceph-users] Feedback of the used configuration Have a look at cephfs subvolumes: https://docs.ceph.com/docs/master/cephfs/fs-volumes/#fs-subvolumes They are internally just directories with quota/pool placement layout/namespace with some mgr magic to make it easier than doing that all by hand Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io<http://www.croit.io> Tel: +49 89 1896585 90 On Wed, Jun 24, 2020 at 4:38 PM Simon Sutter mailto:ssut...@hosttech.ch>> wrote: Hello, After two months of the "ceph try and error game", I finally managed to get an Octopuss cluster up and running. The unconventional thing about it is, it's just for hot backups, no virtual machines on there. All the nodes are without any caching ssd's, just plain hdd's. At the moment there are eight of them with a total of 50TB. We are planning to go up to 25 and bigger disks so we end on 300TB-400TB I decided to go with cephfs, because I don't have any experience in things like S3 and I need to read the same file system from more than one client. I made one cephfs with a replicated pool. On there I added erasure-coded pools to save some Storage. To add those pools, I did it with the setfattr command like this: setfattr -n ceph.dir.layout.pool -v ec_data_server1 /cephfs/nfs/server1 Some of our servers cannot use cephfs (old kernels, special OS's) so I have to use nfs. This is set up with the included ganesha-nfs. Exported is the /cephfs/nfs folder and clients can mount folders below this. There are two final questions: - Was it right to go with the way of "mounting" pools with setfattr, or should I have used multiple cephfs? First I was thinking about using multiple cephfs but there are warnings everywhere. The deeper I got in, the more it seems I would have been fine with multiple cephfs. - Is there a way I don't know, but it would be easier? I still don't know much about Rest, S3, RBD etc... so there may be a better way Other remarks are desired. Thanks in advance, Simon ___ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-le...@ceph.io<mailto:ceph-users-le...@ceph.io> ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Octopus Grafana inside the dashboard
Hello I'm trying to get Grafana working inside the Dashboard. If I press on "Overall Performance" tab, I get an error, because the iframe tries to connect to the internal hostname, which cannot be resolved from my machine. If I directly open grafana, everything works. How can I tell the dashboard, to use the full domain name? I have tried to set "ceph dashboard set-grafana-api-url https://node01.mycorp.local:3000"; but that does not work and always sets itself back to "https://node01:3000";. Thanks in advance, Simon ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] cephfs needs access from two networks
Hello, So I know, the mon services can only bind to just one ip. But I have to make it accessible to two networks because internal and external servers have to mount the cephfs. The internal ip is 10.99.10.1 and the external is some public-ip. I tried nat'ing it with this: "firewall-cmd --zone=public --add-forward-port=port=6789:proto=tcp:toport=6789:toaddr=10.99.10.1 -permanent" So the nat is working, because I get a "ceph v027" (alongside with some gibberish) when I do a telnet "telnet *public-ip* 6789" But when I try to mount it, I get just a timeout: mount - -t ceph *public-ip*:6789:/testing /mnt -o name=test,secretfile=/root/ceph.client. test.key mount error 110 = Connection timed out The tcpdump also recognizes a "Ceph Connect" packet, coming from the mon. How can I get around this problem? Is there something I have missed? Specs: Latest Octopus 15.2.4 Centos 8 8 Nodes No health warnings. Thanks in advance, Simon ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: cephfs needs access from two networks
Hello again So I have changed the network configuration. Now my Ceph is reachable from outside, this also means all osd’s of all nodes are reachable. I still have the same behaviour which is a timeout. The client can resolve all nodes with their hostnames. The mon’s are still listening on the internal network so the nat rule is still there. I have set “public bind addr” to the external ip and restarted the mon but it’s still not working. [root@testnode1 ~]# ceph config get mon.public_bind_addr WHO MASK LEVEL OPTIONVALUERO mon advanced public_bind_addr v2:[ext-addr]:0/0 * Do I have to change them somewhere else too? Thanks in advance, Simon Von: Janne Johansson [mailto:icepic...@gmail.com] Gesendet: 27 August 2020 20:01 An: Simon Sutter Betreff: Re: [ceph-users] cephfs needs access from two networks Den tors 27 aug. 2020 kl 12:05 skrev Simon Sutter mailto:ssut...@hosttech.ch>>: Hello Janne Oh I missed that point. No, the client cannot talk directly to the osds. In this case it’s extremely difficult to set this up. This is an absolute requirement to be a ceph client. How is the mon telling the client, which host and port of the osd, it should connect to? The same port and ip that the ODS called into the mon with when it started up and joined the clusster. Can I have an influence on it? Well, you set the ip on the OSD hosts, and the port ranges in use for OSDs are changeable/settable, but it would not really help the above-mentioned client. Von: Janne Johansson [mailto:icepic...@gmail.com<mailto:icepic...@gmail.com>] Gesendet: 26 August 2020 15:09 An: Simon Sutter mailto:ssut...@hosttech.ch>> Cc: ceph-users@ceph.io<mailto:ceph-users@ceph.io> Betreff: Re: [ceph-users] cephfs needs access from two networks Den ons 26 aug. 2020 kl 14:16 skrev Simon Sutter mailto:ssut...@hosttech.ch>>: Hello, So I know, the mon services can only bind to just one ip. But I have to make it accessible to two networks because internal and external servers have to mount the cephfs. The internal ip is 10.99.10.1 and the external is some public-ip. I tried nat'ing it with this: "firewall-cmd --zone=public --add-forward-port=port=6789:proto=tcp:toport=6789:toaddr=10.99.10.1 -permanent" So the nat is working, because I get a "ceph v027" (alongside with some gibberish) when I do a telnet "telnet *public-ip* 6789" But when I try to mount it, I get just a timeout: mount - -t ceph *public-ip*:6789:/testing /mnt -o name=test,secretfile=/root/ceph.client. test.key mount error 110 = Connection timed out The tcpdump also recognizes a "Ceph Connect" packet, coming from the mon. How can I get around this problem? Is there something I have missed? Any ceph client will need direct access to all OSDs involved also. Your mail doesn't really say if the cephfs-mounting client can talk to OSDs? In ceph, traffic is not shuffled via mons, mons only tell the client which OSDs it needs to talk to, then all IO goes directly from client to any involved OSD servers. -- May the most significant bit of your life be positive. -- May the most significant bit of your life be positive. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Syncing cephfs from Ceph to Ceph
Hello, Is it possible to somehow sync a ceph from one site to a ceph form another site? I'm just using the cephfs feature and no block devices. Being able to sync cephfs pools between two sites would be great for a hot backup, in case one site fails. Thanks in advance, Simon ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Syncing cephfs from Ceph to Ceph
Thanks Stefan, First of all, for a bit more context, we use this ceph cluster just for hot backups, so 99% write 1% read, no need for low latency. Ok so the snapshot function would mean, we would have like a colder backup. Just like a snapshot of a VM, without any incremental functionality, which also means scheduled but huge transfer rates. What about the idea of creating the cluster over two data centers? Would it be possible to modify the crush map, so one pool gets replicated over those two data centers and if one fails, the other one would still be functional? Additionally, would it be possible to prioritize one data center over the other? This would allow saving data from site1 to a pool on site2 in case of a disaster on site1, site2 would still have those Backups. We have a 10G connection with around 0.5ms latency. Thanks in advance, Simon Von: Stefan Kooman Gesendet: Dienstag, 8. September 2020 11:38:29 An: Simon Sutter; ceph-users@ceph.io Betreff: Re: [ceph-users] Syncing cephfs from Ceph to Ceph On 2020-09-08 11:22, Simon Sutter wrote: > Hello, > > > Is it possible to somehow sync a ceph from one site to a ceph form another > site? > I'm just using the cephfs feature and no block devices. > > Being able to sync cephfs pools between two sites would be great for a hot > backup, in case one site fails. It's a work in progress [1]. This might do what you want to right now: [2]. Note: I haven't used [2] myself. Gr. Stefan [1]: https://docs.ceph.com/docs/master/dev/cephfs-mirroring/ [2]: https://github.com/oliveiradan/cephfs-sync ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] New to ceph / Very unbalanced cluster
Hello everybody, I'm very new to ceph and installed a testenvironment (nautilus). The current goal of this cluster is, to be a short period backup. For this goal we want to use older, mixed hardware, so I was thinking, for testing I will set up very unbalanced nodes (you can learn the most, from exceptional circumstances, right?). I created for my cephfs two pools, one for metadata and one for storage data. I have three nodes and the ceph osd tree looks like this: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 21.54213root default -30.75679 host node1 0 hdd 0.75679 osd.0 up 0.00636 1.0 -5 5.39328 host node2 1 hdd 2.66429 osd.1 up 0.65007 1.0 3 hdd 2.72899 osd.3 up 0.65007 1.0 -1015.39206 host node3 5 hdd 7.27739 osd.5 up 1.0 1.0 6 hdd 7.27739 osd.6 up 1.0 1.0 2 ssd 0.38249 osd.2 up 1.0 1.0 4 ssd 0.45479 osd.4 up 1.0 1.0 The PGs and thus the data is extremely unbalanced, you can see it in the ceph osd df overview: ID CLASS WEIGHT REWEIGHT SIZERAW USE DATAOMAPMETA AVAIL %USE VAR PGS STATUS 0 hdd 0.75679 0.00636 775 GiB 651 GiB 650 GiB 88 KiB 1.5 GiB 124 GiB 84.02 7.26 112 up 1 hdd 2.66429 0.65007 2.7 TiB 497 GiB 496 GiB 88 KiB 1.2 GiB 2.2 TiB 18.22 1.57 81 up 3 hdd 2.72899 0.65007 2.7 TiB 505 GiB 504 GiB8 KiB 1.3 GiB 2.2 TiB 18.07 1.56 88 up 5 hdd 7.27739 1.0 7.3 TiB 390 GiB 389 GiB8 KiB 1.2 GiB 6.9 TiB 5.24 0.45 67 up 6 hdd 7.27739 1.0 7.3 TiB 467 GiB 465 GiB 64 KiB 1.3 GiB 6.8 TiB 6.26 0.54 78 up 2 ssd 0.38249 1.0 392 GiB 14 GiB 13 GiB 11 KiB 1024 MiB 377 GiB 3.68 0.32 2 up 4 ssd 0.45479 1.0 466 GiB 28 GiB 27 GiB4 KiB 1024 MiB 438 GiB 6.030.52 4 up TOTAL 22 TiB 2.5 TiB 2.5 TiB 273 KiB 8.4 GiB 19 TiB 11.57 MIN/MAX VAR: 0.32/7.26 STDDEV: 6.87 To counteract this, I tried to turn on the balancer module. The module is decreasing the reweight of the osd0 more and more, while ceph pg stat is telling me, there are more misplaced objects: 144 pgs: 144 active+clean+remapped; 853 GiB data, 2.5 TiB used, 19 TiB / 22 TiB avail; 30 MiB/s wr, 7 op/s; 242259/655140 objects misplaced (36.978%) So my question is: is ceph supposed to do that? Why are all those objects misplaced? Because of those 112 PGs on osd0? Why are there 112 PGs on osd0? I did not set any pg settings except the number: 512 Thank you very much Simon Sutter ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: New to ceph / Very unbalanced cluster
Thank you very much, I couldn't see the forest for the trees. Now I moved a disk and added another one, now the problem is gone, I have 8TB to use. Thanks again. Simon Sutter Von: Reed Dier Gesendet: Mittwoch, 15. April 2020 22:59:12 An: Simon Sutter Cc: ceph-users Betreff: Re: [ceph-users] New to ceph / Very unbalanced cluster Copying the ML, because I forgot to reply-all. Reed On Apr 15, 2020, at 3:58 PM, Reed Dier mailto:reed.d...@focusvq.com>> wrote: The problem is almost certainly stemming from unbalanced OSD distribution among your hosts, and assuming you are using a default 3x replication across hosts crush rule set. You are limited by your smallest bin size. In this case you have a 750GB HDD as the only OSD on node1, so when it wants 3 copies across 3 hosts, there are only ~750GB of space that can fulfill this requirement. Having lots of different size OSDs and differing OSDs in your topology is going to lead to issues of under/over utilization. ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 21.54213root default -3 0.75679 host node1 -5 5.39328 host node2 -1015.39206 host node3 You either need to redistribute your OSDs across your hosts, or possibly rethink your disk strategy. You could move osd.5 to node1, and osd.0 to node2, which would give you roughly 6TiB of usable hdd space across your three nodes. Reed On Apr 15, 2020, at 10:50 AM, Simon Sutter mailto:ssut...@hosttech.ch>> wrote: Hello everybody, I'm very new to ceph and installed a testenvironment (nautilus). The current goal of this cluster is, to be a short period backup. For this goal we want to use older, mixed hardware, so I was thinking, for testing I will set up very unbalanced nodes (you can learn the most, from exceptional circumstances, right?). I created for my cephfs two pools, one for metadata and one for storage data. I have three nodes and the ceph osd tree looks like this: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 21.54213root default -30.75679 host node1 0 hdd 0.75679 osd.0 up 0.00636 1.0 -5 5.39328 host node2 1 hdd 2.66429 osd.1 up 0.65007 1.0 3 hdd 2.72899 osd.3 up 0.65007 1.0 -1015.39206 host node3 5 hdd 7.27739 osd.5 up 1.0 1.0 6 hdd 7.27739 osd.6 up 1.0 1.0 2 ssd 0.38249 osd.2 up 1.0 1.0 4 ssd 0.45479 osd.4 up 1.0 1.0 The PGs and thus the data is extremely unbalanced, you can see it in the ceph osd df overview: ID CLASS WEIGHT REWEIGHT SIZERAW USE DATAOMAPMETA AVAIL %USE VAR PGS STATUS 0 hdd 0.75679 0.00636 775 GiB 651 GiB 650 GiB 88 KiB 1.5 GiB 124 GiB 84.02 7.26 112 up 1 hdd 2.66429 0.65007 2.7 TiB 497 GiB 496 GiB 88 KiB 1.2 GiB 2.2 TiB 18.22 1.57 81 up 3 hdd 2.72899 0.65007 2.7 TiB 505 GiB 504 GiB8 KiB 1.3 GiB 2.2 TiB 18.07 1.56 88 up 5 hdd 7.27739 1.0 7.3 TiB 390 GiB 389 GiB8 KiB 1.2 GiB 6.9 TiB 5.24 0.45 67 up 6 hdd 7.27739 1.0 7.3 TiB 467 GiB 465 GiB 64 KiB 1.3 GiB 6.8 TiB 6.26 0.54 78 up 2 ssd 0.38249 1.0 392 GiB 14 GiB 13 GiB 11 KiB 1024 MiB 377 GiB 3.68 0.32 2 up 4 ssd 0.45479 1.0 466 GiB 28 GiB 27 GiB4 KiB 1024 MiB 438 GiB 6.030.52 4 up TOTAL 22 TiB 2.5 TiB 2.5 TiB 273 KiB 8.4 GiB 19 TiB 11.57 MIN/MAX VAR: 0.32/7.26 STDDEV: 6.87 To counteract this, I tried to turn on the balancer module. The module is decreasing the reweight of the osd0 more and more, while ceph pg stat is telling me, there are more misplaced objects: 144 pgs: 144 active+clean+remapped; 853 GiB data, 2.5 TiB used, 19 TiB / 22 TiB avail; 30 MiB/s wr, 7 op/s; 242259/655140 objects misplaced (36.978%) So my question is: is ceph supposed to do that? Why are all those objects misplaced? Because of those 112 PGs on osd0? Why are there 112 PGs on osd0? I did not set any pg settings except the number: 512 Thank you very much Simon Sutter ___ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-le...@ceph.io<mailto:ceph-users-le...@ceph.io> ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
