[ceph-users] Re: Ceph flash deployment

With the latest kernel, this is not valid for all-flash clusters. Simply because cfq is not an option at all there, and readahead usefulness depends on your workload (in other words, it can help or hurt) and therefore cannot be included in a universally-applicable set of tuning recommendations. Als

[ceph-users] Re: Ceph 14.2 - some PGs stuck peering.

W dniu 2020-11-04 01:18, m.sliwin...@lh.pl napisał(a): Just in case - result of ceph report is here: http://paste.ubuntu.com/p/D7yfr3pzr4/ Hi We have a weird issue iwth our ceph cluster - almost all PGs assigned to one specific pool became stuck, locking out all operations without reporting

[ceph-users] Ceph 14.2 - some PGs stuck peering.

Hi We have a weird issue iwth our ceph cluster - almost all PGs assigned to one specific pool became stuck, locking out all operations without reporting any errors. Story: We have 3 different pools, hdd-backed, ssd-backed and nvme-backed. Pool ssh worked fine for few months. Today one of the h

[ceph-users] Ceph 14.2 - stuck peering.

Hi We have weird issue iwth our ceph cluster - almost all PGs assigned to one specific pool became stuck, locking out all operations without reporting any errors. Story: We have 3 different pools, hdd-backed, ssd-backed and nvme-backed. Pool ssh worked fine for few months. Today one of the hos

[ceph-users] Re: Cephadm: module not found

Thanks for really fast answer, however, there is no such line. I have a version v15.2.4 . This file is already quite modified from the version which i have. Probably, this backport

[ceph-users] Re: Ceph flash deployment

Thanks for your useful information. Can you please also point to the kernel and disk configuration that are still valid for bluestore or not? I mean the read_ahead_kb and disk scheduler. Thanks. On Tue, Nov 3, 2020 at 10:55 PM Alexander E. Patrakov wrote: > On Tue, Nov 3, 2020 at 6:30 AM Seena

[ceph-users] Re: Ceph flash deployment

On Tue, Nov 3, 2020 at 6:30 AM Seena Fallah wrote: > > Hi all, > > Does this guid is still valid for a bluestore deployment with nautilus or > octopus? > https://tracker.ceph.com/projects/ceph/wiki/Tuning_for_All_Flash_Deployments Some of the guidance is of course outdated. E.g., at the time of

[ceph-users] Re: How to recover from active+clean+inconsistent+failed_repair? [SOLVED]

Hi Sagara, good to hear. Are you using filestore? I completely missed that. Bluestore-tool would have been useless :) My suspicion is a lost write from cache due to power loss. Best regards, = Frank Schilder AIT Risø Campus Bygning 109, rum S14 _

[ceph-users] Re: How to recover from active+clean+inconsistent+failed_repair? [SOLVED]

Hi Frank Found the issue and fixed. It was a one copy of 0 byte object. Removed it. Deep scrub the PG fixed the issue. # find /var/lib/ceph/osd/ -type f -name "123675e*" /var/lib/ceph/osd/ceph-2/current/3.b_head/DIR_B/DIR_A/DIR_E/123675e.__head_AE97EEAB__3 # ls -l /var/lib/ceph/o

[ceph-users] Re: Inconsistent Space Usage reporting

Thanks. Let me try it and I'll report back. -Original Message- From: Adam Tygart Sent: Tuesday, November 3, 2020 12:42 PM To: Vikas Rana Cc: ceph-users Subject: Re: [ceph-users] Re: Inconsistent Space Usage reporting I'm not sure exactly what you're doing with your volumes. It looks

[ceph-users] Re: Inconsistent Space Usage reporting

I'm not sure exactly what you're doing with your volumes. It looks like fcp might be size 3. nfs is size 1, possibly with an 200TB rbd volume inside nbd mounted into another box. If so, it is likely you can reclaim space from deleted files with fstrim, if your filesystem supports it. -- Adam On

[ceph-users] Re: Inconsistent Space Usage reporting

Any help or direction in this below case is highly appreciated. Thanks, -Vikas -Original Message- From: Vikas Rana Sent: Monday, November 2, 2020 12:53 PM To: ceph-users@ceph.io Subject: [ceph-users] Inconsistent Space Usage reporting Hi Friends, We have some inconsistent storage s

[ceph-users] Re: Restart Error: osd.47 already exists in network host

I'm not familiar with docker yet but apparently the cleanup doesn't work? Would something like this work? docker network disconnect -f host $container Maybe it's the same as 'docker network prune', I don't know, I also don't have a docker environment available and podman seems to work slig

[ceph-users] Re: Cephadm: module not found

Thanks for really fast answer, however, there is no such line. I have a version v15.2.4 . This file is already quite modified from the version which i have. Probably, this backport

[ceph-users] Re: Cephadm: module not found

Sorry, it should be “cephadm enter”. 在 2020年11月3日，22:09，胡 玮文 写道：  Hi Nadiia, Although I don’t have this issue, I think you can apply the fix manually. You just need to use “cephadm exec” to get into the mgr container, and change one line of python code as in https://github.com/ceph/ceph/pul

[ceph-users] Re: Updating client caps online

On Tue, Nov 3, 2020 at 12:07 PM Wido den Hollander wrote: > On 03/11/2020 10:02, Dan van der Ster wrote: > > Hi all, > > > > We still have legacy caps on our nautilus rbd cluster. I just wanted > > to check if this is totally safe (and to post here ftr because I don't > > think this has ever been

[ceph-users] Re: Cephadm: module not found

Hi Nadiia, Although I don’t have this issue, I think you can apply the fix manually. You just need to use “cephadm exec” to get into the mgr container, and change one line of python code as in https://github.com/ceph/ceph/pull/37141/files#diff-5f6d300f6d71c1b58783257d5dc652d507376cb018f227ab6fa

[ceph-users] Re: How to recover from active+clean+inconsistent+failed_repair?

Hi Sagara, did you collect the output of "ceph -w | grep '3\.b'" while the PG was deep-scrubbed? Please open a new thread with the error message in the subject line. This should draw the attention of someone with internal knowledge. Include a short description of how this situation came about w

[ceph-users] Re: Cephadm: module not found

Hi, i am experience the same problem. Could you please advise something how to resolve this issue? The fix should be shipped with 15.2.6 version of "ceph-common" or ceph version? I have my cluster in docker containers and systemd services. How can I upgrade cluster to 15.2.6 if the command f

[ceph-users] Re: How to recover from active+clean+inconsistent+failed_repair?

Hi Frank 1. We will disable the disk controller and disk-level caching to avoid future issues. 2. My pools are: ceph osd lspools 2 cephfs_metadata 3 cephfs_data 4 rbd The PG now inconsistent is 3.b,  therefore, it belongs to cephfs_data pool. Following also shows the PG 3.b belongs to

[ceph-users] Re: Does it make sense to have separate HDD based DB/WAL partition

On 03.11.20 11:03, huxia...@horebdata.cn wrote: > I am faced with a special use Ceph scenario, where no SSD is available. Three > node Ceph cluster, each with 14 HDDs, no SSD. In this case, should i create a > separate DB/WAL partition for each OSD on a HDD? and if yes, how big the > partition?

[ceph-users] Re: Updating client caps online

On 03/11/2020 10:02, Dan van der Ster wrote: Hi all, We still have legacy caps on our nautilus rbd cluster. I just wanted to check if this is totally safe (and to post here ftr because I don't think this has ever been documented) Here are the current caps: [client.images] key = xxx caps mgr

[ceph-users] Does it make sense to have separate HDD based DB/WAL partition

Dear Ceph folks, I am faced with a special use Ceph scenario, where no SSD is available. Three node Ceph cluster, each with 14 HDDs, no SSD. In this case, should i create a separate DB/WAL partition for each OSD on a HDD? and if yes, how big the partition? Which is the best in terms of perform

[ceph-users] Updating client caps online

Hi all, We still have legacy caps on our nautilus rbd cluster. I just wanted to check if this is totally safe (and to post here ftr because I don't think this has ever been documented) Here are the current caps: [client.images] key = xxx caps mgr = "allow r" caps mon = "allow r, allow command \"