[ceph-users] BlueFS spillover detected, why, what?

Hi Recently our ceph cluster (nautilus) is experiencing bluefs spillovers, just 2 osd's and I disabled the warning for these osds. (ceph config set osd.125 bluestore_warn_on_bluefs_spillover false) I'm wondering what causes this and how this can be prevented. As I understand it the rocksdb fo

[ceph-users] Re: CEPH FS is always showing the status as creating

Hello Nathan, Below is the output of ceph status:- cluster: id: a3ede5f7-ade8-4bfd-91f4-568e19ca9e69 health: HEALTH_WARN 1 MDSs report slow metadata IOs Degraded data redundancy: 12563/37689 objects degraded (33.333%), 109 pgs degraded application

[ceph-users] Re: OSD memory leak?

Hi Frank, I didn't get time yet. On our side, I was planning to see if the issue persists after upgrading to v14.2.11 -- it includes some updates to how the osdmap is referenced across OSD.cc. BTW, do you happen to have osd_map_dedup set to false? We do, and that surely increases the osdmap memor

[ceph-users] Re: 5 pgs inactive, 5 pgs incomplete

Hi Martin, have you seen this blog post [1]? It describes how to recover from inactive and incomplete PGs (on a size 1 pool). I haven't tried any of that but it could be worth a try. Apparently it only would work if the affected PGs have 0 objects but that seems to be the case, right? Reg

[ceph-users] Re: CEPH FS is always showing the status as creating

You need to fix the out OSDs first. The default pool size is very likely three and you only have two OSDs up, that's why 33% of your PGs are degraded. I'm pretty sure if you fix that your cephfs will become active. Zitat von Alokkumar Mahajan : Hello Nathan, Below is the output of ceph s

[ceph-users] Re: BlueFS spillover detected, why, what?

Hi Simon As far as I know, RocksDB only uses "leveled" space on the NVME partition. The values are set to be 300MB, 3GB, 30GB and 300GB. Every DB space above such a limit will automatically end up on slow devices. In your setup where you have 123GB per OSD that means you only use 30GB of fast

[ceph-users] Re: Ceph Snapshot Children not exists / children relation broken

On 8/3/20 2:07 PM, Torsten Ennenbach wrote: Hi Jason. Well, I don't tried that, because I am afraid to break something :/ I don’t really understand what are you doing there:( Thanks anyways. May be you catch this [1] bug? I have how-to solution [2] to resolve this, please try again. [1

[ceph-users] Re: BlueFS spillover detected, why, what?

Hi Michael, thanks for the explanation! So if I understand correctly, we waste 93 GB per OSD on unused NVME space, because only 30GB is actually used...? And to improve the space for rocksdb, we need to plan for 300GB per rocksdb partition in order to benefit from this advantage Reducin

[ceph-users] Re: cephadm not working with non-root user

Hi, Any of you used cephadm bootstrap command without root user? On Wed, Aug 19, 2020 at 11:30 AM Amudhan P wrote: > Hi, > > I am trying to install ceph 'octopus' using cephadm. In bootstrap > command, I have specified a non-root user account as ssh-user. > cephadm bootstrap --mon-ip xx.xxx.xx

[ceph-users] Re: 5 pgs inactive, 5 pgs incomplete

Something else to help debugging is ceph pg 17.173 query at the end it should say why the pg is incomplete. -- dan On Thu, Aug 20, 2020 at 10:01 AM Eugen Block wrote: > > Hi Martin, > > have you seen this blog post [1]? It describes how to recover from > inactive and incomplete PGs (on a siz

[ceph-users] Re: BlueFS spillover detected, why, what?

Hi Simon Unfortunately, the other NVME space is wasted or at least, this is the information we gathered during our research. This fact is due to the RocksDB level management which is explained here (https://github.com/facebook/rocksdb/wiki/Leveled-Compaction). I don't think it's a hard limit b

[ceph-users] Re: BlueFS spillover detected, why, what?

Hi Michael, thanks for the pointers! This is our first production ceph cluster and we have to learn as we go... Small files is always a problem for all (networked) filesystems, usually it just trashes performance, but in this case it has another unfortunate side effect with the rocksdb :-( C

[ceph-users] Re: BlueFS spillover detected, why, what?

Hi Simon, starting Nautlus v14.2.10 Bluestore is able to use 'wasted' space at DB volume. see this PR: https://github.com/ceph/ceph/pull/29687 Nice overview on the overall BlueFS/RocksDB design can be find here: https://cf2.cloudferro.com:8080/swift/v1/AUTH_5e376cddf8a94f9294259b5f48d7b2cd/

[ceph-users] Re: 5 pgs inactive, 5 pgs incomplete

All inactive and incomplete PGs are blocked by OSD 81 which does not exist anymore: ... "down_osds_we_would_probe": [ 81 ], "peering_blocked_by": [], "peering_blocked_by_detail": [ { "detail": "peering_blocked_b

[ceph-users] Re: 5 pgs inactive, 5 pgs incomplete

Did you already mark osd.81 as lost? AFAIU you need to `ceph osd lost 81`, and *then* you can try the osd_find_best_info_ignore_history_les option. -- dan On Thu, Aug 20, 2020 at 11:31 AM Martin Palma wrote: > > All inactive and incomplete PGs are blocked by OSD 81 which does not > exist anymo

[ceph-users] Re: 5 pgs inactive, 5 pgs incomplete

Yes we already did that but since the OSD does not exists anymore we get the following error: % ceph osd lost 81 --yes-i-really-mean-it Error ENOENT: osd.81 does not exist So we do not know how we can bring the PGs to notice that OSD 81 does not exist anymore... On Thu, Aug 20, 2020 at 11:41 AM

[ceph-users] Re: 5 pgs inactive, 5 pgs incomplete

On one pool, which was only a test pool, we investigated both OSDs which host the inactive and incomplete PG with the following command: % ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-## --pgid --op list On the primary OSD for the PG we saw no output, but on the secondary we got an o

[ceph-users] Ceph on windows?

Hey guys, it seems like there was a presentation called “ceph on windows” at the Cephalocon 2020, but I cannot find any information on that topic. Is there a video from the presentation out there or any other information? I only found https://ceph2020.sched.com/event/ZDUK/ceph-on-windows-alessa

[ceph-users] Re: BlueFS spillover detected, why, what?

Hi Igor. Could you please tell why this config is in LEVEL_DEV ( https://github.com/ceph/ceph/pull/29687/files#diff-3d7a065928b2852c228ffe669d7633bbR4587)? As it is documented in Ceph we can't use LEVEL_DEV in production environments! Thanks On Thu, Aug 20, 2020 at 1:58 PM Igor Fedotov wrote:

[ceph-users] Ceph mon crash, many osd down

Hi all. My cluster has many log mon scrub mon data 2020-08-20 13:12:16.393 7fe89becc700 0 log_channel(cluster) log [DBG] : scrub ok on 0,1,2,3: ScrubResult(keys {auth=100} crc {auth=3066031631}) 2020-08-20 13:12:16.395 7fe89becc700 0 log_channel(cluster) log [DBG] : scrub ok on 0,1,2,3: Scrub

[ceph-users] Re: OSD memory leak?

Hi Dan and Mark, could you please let me know if you can read the files with the version info I provided in my previous e-mail? I'm in the process of collecting data with more FS activity and would like to send it in a format that is useful for investigation. Right now I'm observing a daily gr

[ceph-users] Re: Ceph on windows?

It's an effort to expose RBD to Windows via a native driver [1]. That driver is basically a thin NBD shim to connect with the rbd-nbd daemon running as a Windows service. On Thu, Aug 20, 2020 at 6:07 AM Stolte, Felix wrote: > > Hey guys, > > it seems like there was a presentation called “ceph on

[ceph-users] Re: BlueFS spillover detected, why, what?

Hi Seena, this parameter isn't intended to be adjusted in production environments - it's supposed that default behavior covers all regular customers' needs. The issue though is that default setting is invalid. It should be 'use_some_extra'. Gonna fix that shortly... Thanks, Igor On 8/

[ceph-users] Re: BlueFS spillover detected, why, what?

Greate, thanks. Is it safe to change it manually in ceph.conf until next nautilus release or should I wait for the next nautilus release for this change? I mean does qa run on this value for this config that we could trust and change it or should we wait until the next nautilus release that qa ran

[ceph-users] Re: BlueFS spillover detected, why, what?

From technical/developer's point of view I don't see any issues with tuning this option. But since now I wouldn't  recommend to enable it in production as it partially bypassed our regular development cycle. Being enabled in master for a while by default allows more develpers to use/try the fea

[ceph-users] Re: BlueFS spillover detected, why, what?

So you won't backport it to nautilus until it gets default to master for a while? On Thu, Aug 20, 2020 at 6:00 PM Igor Fedotov wrote: > From technical/developer's point of view I don't see any issues with > tuning this option. But since now I wouldn't recommend to enable it in > production as i

[ceph-users] Re: BlueFS spillover detected, why, what?

Correct. On 8/20/2020 5:15 PM, Seena Fallah wrote: So you won't backport it to nautilus until it gets default to master for a while? On Thu, Aug 20, 2020 at 6:00 PM Igor Fedotov > wrote: From technical/developer's point of view I don't see any issues with tun

[ceph-users] Re: BlueFS spillover detected, why, what?

So what do you suggest for a short term solution? (I think you won't backport it to nautilus at least about 6 month) Changing db size is too expensive because I should buy new NVME devices with double size and also redeploy all my OSDs. Manual compaction will still have an impact on performance an

[ceph-users] Re: Ceph on windows?

On 8/20/20 2:11 PM, Jason Dillaman wrote: > It's an effort to expose RBD to Windows via a native driver [1]. That > driver is basically a thin NBD shim to connect with the rbd-nbd daemon > running as a Windows service. > > [1] https://github.com/ceph/ceph/pulls?q=is%3Aopen+is%3Apr+label%3Awin32+

[ceph-users] luks / disk encryption best practice

I still need to move from ceph disk to ceph volume. When doing this, I wanted to also start using disk encryption. I am not really interested in encryption offered by the hdd vendors. Is there a best practice or advice what encryption to use ciphers/hash? Stick to the default of CentOS7 or ma

[ceph-users] Re: radosgw beast access logs

Sure. You can track the nautilus backport's progress in https://tracker.ceph.com/issues/47042. On Wed, Aug 19, 2020 at 12:25 PM Wesley Dillingham wrote: > > We would very much appreciate having this backported to nautilus. > > Respectfully, > > Wes Dillingham > w...@wesdillingham.com > LinkedIn

[ceph-users] Re: Bonus Ceph Tech Talk: Edge Application - Stream Multiple Video Sources

And we're live! Please join us and bring questions! https://bluejeans.com/908675367 On 8/17/20 11:03 AM, Mike Perez wrote: Hi all, We have a bonus Ceph Tech Talk for August. Join us August 20th at 17:00 UTC to hear Neeha Kompala and Jason Weng present on Edge Application - Streaming Multipl

[ceph-users] Re: Bonus Ceph Tech Talk: Edge Application - Stream Multiple Video Sources

Can't join as guest without enabling mic and/or camera??? -Original Message- From: Mike Perez [mailto:mipe...@redhat.com] Sent: donderdag 20 augustus 2020 19:03 To: ceph-users@ceph.io Subject: [ceph-users] Re: Bonus Ceph Tech Talk: Edge Application - Stream Multiple Video Sources And

[ceph-users] Re: Bonus Ceph Tech Talk: Edge Application - Stream Multiple Video Sources

Hi...Will it be available on youtube? On Thursday, August 20, 2020, Marc Roos wrote: > > Can't join as guest without enabling mic and/or camera??? > > -Original Message- > From: Mike Perez [mailto:mipe...@redhat.com] > Sent: donderdag 20 augustus 2020 19:03 > To: ceph-users@ceph.io > Subj

[ceph-users] Re: OSD memory leak?

Hi Frank,  I downloaded but haven't had time to get the environment setup yet either.  It might be better to just generate the txt files if you can. Thanks! Mark On 8/20/20 2:33 AM, Frank Schilder wrote: Hi Dan and Mark, could you please let me know if you can read the files with the ve

[ceph-users] Re: Bonus Ceph Tech Talk: Edge Application - Stream Multiple Video Sources

Here's the video in case you missed it: https://www.youtube.com/watch?v=Q8bU-m07Czo On 8/20/20 10:03 AM, Mike Perez wrote: And we're live! Please join us and bring questions! https://bluejeans.com/908675367 On 8/17/20 11:03 AM, Mike Perez wrote: Hi all, We have a bonus Ceph Tech Talk for

[ceph-users] Re: BlueFS spillover detected, why, what?

Honestly I don't have any perfect solution for now. If this is urgent you probably better to proceed with enabling the new DB space management feature. But please do that eventually, modify 1-2 OSDs at the first stage and test them for some period (may be a week or two). Thanks, Igor On

[ceph-users] Re: Bonus Ceph Tech Talk: Edge Application - Stream Multiple Video Sources

thanks! On Thursday, August 20, 2020, Mike Perez wrote: > Here's the video in case you missed it: > > https://www.youtube.com/watch?v=Q8bU-m07Czo > > On 8/20/20 10:03 AM, Mike Perez wrote: >> >> And we're live! Please join us and bring questions! >> >> https://bluejeans.com/908675367 >> >> On 8/1

[ceph-users] Re: BlueFS spillover detected, why, what?

Ok thanks. And also as you mentioned in the doc you shared from cloudferro, It's not good to change `write_buffer_size` for bluestore rocksdb to fit our db? On Fri, Aug 21, 2020 at 1:46 AM Igor Fedotov wrote: > Honestly I don't have any perfect solution for now. > > If this is urgent you probabl

[ceph-users] Re: OSD memory leak?

Hi Dan, no worries. I checked and osd_map_dedup is set to true, the default value. Best regards, = Frank Schilder AIT Risø Campus Bygning 109, rum S14 From: Dan van der Ster Sent: 20 August 2020 09:41 To: Frank Schilder Cc: Mark Nelson; c

[ceph-users] Re: OSD memory leak?

Hi Mark and Dan, I can generate text files. Can you let me know what you would like to see? Without further instructions, I can do a simple conversion and a conversion against the first dump as a base. I will upload an archive with converted files added tomorrow afternoon. Best regards, ==