[ceph-users] ceph pg stuck active+remapped+backfilling

2024-07-31 Thread Jorge Garcia
We were having an OSD reporting lots of errors, so I tried to remove it by doing: ceph orch osd rm 139 --zap It started moving all the data. Eventually, we got to the point that there's only 1 pg backfilling, but that seems to be stuck now. I think it may be because, in the process, another OSD

[ceph-users] rbd resize progress report?

2021-02-01 Thread Jorge Garcia
I'm trying to resize a block device using "rbd --resize". The block device is pretty huge (100+ TB). The resize has been running for over a week, and I have no idea if it's actually doing anything, or if it's just hanging or in some infinite loop. Is there any way of getting a progress report from

[ceph-users] cephfs vs rbd vs rgw

2021-05-25 Thread Jorge Garcia
This may be too broad of a topic, or opening a can of worms, but we are running a CEPH environment and I was wondering if there's any guidance about this question: Given that some group would like to store 50-100 TBs of data on CEPH and use it from a linux environment, are there any advantages

[ceph-users] cephfs vs rbd

2021-10-08 Thread Jorge Garcia
I was wondering about performance differences between cephfs and rbd, so I deviced this quick test. The results were pretty surprising to me. The test: on a very idle machine, make 2 mounts. One is a cephfs mount, the other an rbd mount. In each directory, copy a humongous .tgz file (1.5 TB) a

[ceph-users] Re: cephfs vs rbd

2021-10-08 Thread Jorge Garcia
> Please describe the client system. > Are you using the same one for CephFS and RBD? Yes > Kernel version? Centos 8 4.18.0-240.15.1.el8_3.x86_64 (but also tried on a Centos 7 machine, similar results) > BM or VM? Bare Metal > KRBD or libvirt/librbd? I'm assuming KRBD. I just did a simple

[ceph-users] Problem with upgrade

2023-10-26 Thread Jorge Garcia
I'm trying to upgrade our 3-monitor cluster from Centos 7 and Nautilus to Rocky 9 and Quincy. This has been a very slow process of upgrading one thing, running the cluster for a while, then upgrading the next thing. I first upgraded to the last Centos 7 and upgraded to Octopus. That worked fine. Th

[ceph-users] Re: Problem with upgrade

2023-10-26 Thread Jorge Garcia
but I have gone to Octopus (no problems) and now to Pacific (problems). On Thu, Oct 26, 2023 at 3:36 PM Tyler Stachecki wrote: > > On Thu, Oct 26, 2023, 6:16 PM Jorge Garcia wrote: > > > > from Centos 7 and Nautilus to > > Rocky 9 and Quincy. > > I hate to b

[ceph-users] Re: Problem with upgrade

2023-10-26 Thread Jorge Garcia
ntos7+Octopus to Rocky8+Octopus to Rocky8+Pacific to Rocky9+Pacific to Rocky9+Quincy I was just hoping that I could skip the Rocky8 installation altogether... On Thu, Oct 26, 2023 at 4:57 PM Tyler Stachecki wrote: > On Thu, Oct 26, 2023 at 6:52 PM Jorge Garcia wrote: > > > > H

[ceph-users] Re: Problem with upgrade

2023-10-27 Thread Jorge Garcia
l the monitors and all the managers to Pacific and Rocky 9. Now on to the OSDs. Well, maybe next week... On Thu, Oct 26, 2023 at 5:37 PM Tyler Stachecki wrote: > On Thu, Oct 26, 2023, 8:11 PM Jorge Garcia wrote: > >> Oh, I meant that "ceph -s" just hangs. I didn't even tr

[ceph-users] Power outage recovery

2022-09-15 Thread Jorge Garcia
We have a Nautilus cluster that just got hit by a bad power outage. When the admin systems came back up, only the ceph-mgr process was running (all the ceph-mon processes would not start). I tried following the instructions in https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mo

[ceph-users] Re: Power outage recovery

2022-09-15 Thread Jorge Garcia
OK, I'll try to give more details as I remember them. 1. There was a power outage and then power came back up. 2. When the systems came back up, I did a "ceph -s" and it never returned. Further investigation revealed that the ceph-mon processes had not started in any of the 3 monitors. I looke

[ceph-users] Re: Power outage recovery

2022-09-15 Thread Jorge Garcia
change the MDS and RGW's ability to start and stay running. Have you tried just restarting the MDS / RGW daemons again? Respectfully, *Wes Dillingham* w...@wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham> On Thu, Sep 15, 2022 at 5:54 PM Jorge Garcia wrote:

[ceph-users] Almost there - trying to recover cephfs from power outage

2022-09-20 Thread Jorge Garcia
I have been trying to recover our ceph cluster from a power outage. I was able to recover most of the cluster using the data from the OSDs. But the MDS maps were gone, and now I'm trying to recover that. I was looking around and found a section in the Quincy manual titled RECOVERING THE FILE SY

[ceph-users] Re: Almost there - trying to recover cephfs from power outage

2022-09-21 Thread Jorge Garcia
/51341 Also, here was the change which added the --recover flag: https://tracker.ceph.com/issues/51716 There you can see the old process described again. Good luck, Dan On Tue, Sep 20, 2022 at 9:00 PM Jorge Garcia wrote: I have been trying to recover our ceph cluster from a power outage. I was

[ceph-users] Why is my cephfs almostfull?

2023-04-05 Thread Jorge Garcia
We have a ceph cluster with a cephfs filesystem that we use mostly for backups. When I do a "ceph -s" or a "ceph df", it reports lots of space:     data:   pools:   3 pools, 4104 pgs   objects: 1.09 G objects, 944 TiB   usage:   1.5 PiB used, 1.0 PiB / 2.5 PiB avail   GLOBAL:     SI

[ceph-users] Trying to install nautilus, keep getting mimic

2020-01-09 Thread Jorge Garcia
I have been trying to install Nautilus using ceph-deploy, but somehow keep getting mimic installed. I'm not sure where mimic is even coming from... Here's what I'm doing (basically following the docs): * Fresh install of Centos 7.7   sudo yum install -y https://dl.fedoraproject.org/pub/epel/e

[ceph-users] OSDs won't come back after upgrade

2025-01-08 Thread Jorge Garcia
Hello, I'm going down the long and winding road of upgrading our ceph clusters from mimic to the latest version. This has involved slowly going up one release at a time. I'm now going from octopus to pacific, which also involves upgrading the OS on the host systems from Centos 7 to Rocky 9. I fir

[ceph-users] Re: OSDs won't come back after upgrade

2025-01-10 Thread Jorge Garcia
Actually, stupid mistake on my part. I had selinux mode as enforcing. Changed it to disabled, and everything works again. Thanks for the help! ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: OSDs won't come back after upgrade

2025-01-13 Thread Jorge Garcia
lvaro Soto wrote: > > But why do you need to disable selinux for the service to work? You shouldn't > have an issue. > > > On Fri, Jan 10, 2025, 6:20 PM Jorge Garcia wrote: >> >> Actually, stupid mistake on my part. I had selinux mode as enforcing. >> Changed