[ceph-users] Erasure coding and backfilling speed

Hi. I have a Ceph (NVME) based cluster with 12 hosts and 40 OSD's .. currently it is backfilling pg's but I cannot get it to run more than 20 backfilling (pgs) at the same time (6+2 profile) osd_max_backfills = 100 and osd_recovery_max_active_ssd = 50 (non-sane) but it still stops at 20 with 4

[ceph-users] HBase/HDFS on Ceph/CephFS

of the dataset is a bit more than what looks feasible at the moment. Thanks for your reflections/input. Jesper ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: HBase/HDFS on Ceph/CephFS

everything again. I may get equally good data locality with Ceph-based SSD as with local HDDs (which I currently have) Jesper ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Ceph MDS - busy?

Hi. How do I find out if the MDS is "busy" - being the one limiting CephFS metadata throughput. (12.2.8). $ time find . | wc -l 1918069 real8m43.008s user0m2.689s sys 0m7.818s or 3.667ms per file. In the light of "potentially batching" and a network latency of ~0.20ms to the MDS - I

[ceph-users] Re: OSD weight on Luminous

unless uou have enabled some balancing - then this is very normal (actually pretty good normal) Jesper Sent from myMail for iOS Thursday, 14 May 2020, 09.35 +0200 from Florent B. : >Hi, > >I have something strange on a Ceph Luminous cluster. > >All OSDs have the same size,

[ceph-users] Change crush rule on pool

Hi I would like to change the crush rule so data lands on ssd instead of hdd, can this be done on the fly and migration will just happen or do I need to do something to move data? Jesper Sent from myMail for iOS ___ ceph-users mailing list

[ceph-users] Re: Change crush rule on pool

> I would like to change the crush rule so data lands on ssd instead of hdd, > can this be done on the fly and migration will just happen or do I need to > do something to move data? I would actually like to relocate my object store to a new storage tier. Is the best to: 1) create new pool on sto

[ceph-users] Re: Change crush rule on pool

 Can i do that - when the SSDs are allready used in another crush rule - backing and kvm_ssd rbd’s? Jesper Sent from myMail for iOS Saturday, 12 September 2020, 11.01 +0200 from anthony.da...@gmail.com : >If you have capacity to have both online at the same time, why not add the &g

[ceph-users] Re: krdb upmap compatibility

What will actually happen if an old client comes by, potential data damage - or just broken connections from the client? jesper  Sent from myMail for iOS Monday, 26 August 2019, 20.16 +0200 from Paul Emmerich : >4.13 or newer is enough for upmap > >-- >Paul Emmerich > &g

[ceph-users] Re: the ceph rbd read dd with fio performance diffrent so huge?

concurrency is widely different 1:30  Jesper  Sent from myMail for iOS Tuesday, 27 August 2019, 16.25 +0200 from linghucongs...@163.com : >The performance with the dd and fio diffrent is so huge? > >I have 25 OSDS with 8TB hdd. with dd I only get 410KB/s read perfomance,but &g

[ceph-users] Re: Danish ceph users

yes Sent from myMail for iOS Thursday, 29 August 2019, 15.52 +0200 from fr...@dtu.dk : >I would be in. > >= >Frank Schilder >AIT Risø Campus >Bygning 109, rum S14 > > >From: Torben Hørup < tor...@t-hoerup.dk > >Sent: 29 August 2019 14:0

[ceph-users] Re: MDS blocked ops; kernel: Workqueue: ceph-pg-invalid ceph_invalidate_work [ceph]

23db53201013c940bf000 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3e1d0452edceebb903d23db53201013c940bf000 Was capable of deadlocking the kernel when memory pressure caused MDS to reclaim capabilities - smells similar. Jesper ___

[ceph-users] Re: Building a petabyte cluster from scratch

improvements: * Hardware raid with Battery Backed write-cache - will allow OSD to ack writes before hitting spinning rust. * More memory for OSD-level read-caching. * 3x replication instead of EC .. (we have all above in a "similar" setup ~1PB - 10 OSD - hosts). SSD-tiering pool (havent been there - but would like to test it out). -- Jesper ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Building a petabyte cluster from scratch

ly affordable - or have I missed something that makes the math work ? -- Jesper ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Building a petabyte cluster from scratch

; fraction of our dataset - and 10GB cache on all 113 HDD ~1TB effective read-cache - and then writes hitting the battery-backed write-cache - this can overspill and when hitting "cold" data performance varies. But the read/write amplification of EC is still un-manageable

[ceph-users] Performance of old vs new hw?

Hi We have some oldish servers with ssds - all on 25gbit nics. R815 AMD - 2,4ghz+ Is there significant performance benefits in moving to a new NVMe based, new cpus? +20% IOPs? + 50% IOPs? Jesper Sent from myMail for iOS ___ ceph-users mailing

[ceph-users] Re: Ceph Performance of Micron 5210 SATA?

But is random/sequential read performance still good? even during saturated write performance ?  if so the tradeoff could fit quite some applications  Sent from myMail for iOS Friday, 6 March 2020, 14.06 +0100 from vitalif : >Hi, > >Current QLC drives are total shit in terms of steady-stat

[ceph-users] Re: New 3 node Ceph cluster

Hi. Unless there is plans for going to Petabyte scale with it - then I really dont see the benefits of getting CephFS involved over just an RBD image with a VM running standard samba on top. More performant and less complexity to handle - zero gains (by my book) Jesper > Hi, > > I am

[ceph-users] Recommendation for decent write latency performance from HDDs

will be as slow as hitting dead-rust - anything that cannot live with that need to be entirely on SSD/NVMe. Other? Thanks for your input. Jesper ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Recommendation for decent write latency performance from HDDs

iver a fast write cache for smallish writes. Would setting the parameter til 1MB be "insane"? Jesper ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] MDS_CACHE_OVERSIZED warning

32GB); 34400070 inodes in use by clients, 3293 stray files Thanks - Jesper ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] pg_num != pgp_num - and unable to change.

ienced that setting pg_num and pgp_num took immidiate effect? Jesper jskr@dkcphhpcmgt028:/$ sudo ceph version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) jskr@dkcphhpcmgt028:/$ sudo ceph health HEALTH_OK jskr@dkcphhpcmgt028:/$ sudo ceph status cluster: i

[ceph-users] Cannot get backfill speed up

spend more resources on recovery than 328 MiB/s Thanks, . -- Jesper Krogh ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Cephfs metadata and MDS on same node

tency? Anyway, I am just asking for your opinion on this? Pros and cons or even better somebody who actually have tried this? Best regards, Jesper -- Jesper Lykkegaard Karlsen Scientific Computing Centre for Structural Biology Department of Molecular Biology and Genetics A

[ceph-users] Recover data from Cephfs snapshot

ssing some "ceph snapshot recover" command, that can move metadata pointers and make recovery much lighter, or is this just that way it is? Best reagards, Jesper -- Jesper Lykkegaard Karlsen Scientific Computing Centre for Structural Biology Department of Mo

[ceph-users] replacing OSD nodes

even if this is a good idea? It would be nice if Ceph did something similar automatically 🙂 Or maybe Ceph already does something similar, and I have just not been able to find it? If Ceph were to do this, it could be nice if the priority of backfill_wait PGs was rerun, perharps every 24 hours

[ceph-users] Re: replacing OSD nodes

Thanks for you answer Janne. Yes, I am also running "ceph osd reweight" on the "nearfull" osds, once they get too close for comfort. But I just though a continuous prioritization of rebalancing PGs, could make this process more smooth, with less/no need for handheld opera

[ceph-users] Re: replacing OSD nodes

not? Summer vacation? Best, Jesper -- Jesper Lykkegaard Karlsen Scientific Computing Centre for Structural Biology Department of Molecular Biology and Genetics Aarhus University Universitetsbyen 81 8000 Aarhus C E-mail: je...@mbg.au.dk Tlf:+45 50906203

[ceph-users] Re: PG does not become active

two OSDs down. Also, I believe that min_size should never be smaller than “coding” shards, which is 4 in this case. You can either make a new test setup with your three test OSD hosts using EC 2+1 or make e.g. 4+2, but with failure domain set to OSD. Best, Jesper

[ceph-users] Re: PG does not become active

Ah I see, should have look at the “raw” data instead ;-) Then I agree this very weird? Best, Jesper -- Jesper Lykkegaard Karlsen Scientific Computing Centre for Structural Biology Department of Molecular Biology and Genetics Aarhus University Universitetsbyen 81 8000

[ceph-users] Re: cannot set quota on ceph fs root

Hi Frank, I guess there is alway the possibility to set quota on pool level with "target_max_objects" and “target_max_bytes” The cephfs quotas through attributes are only for sub-directories as far as I recall. Best, Jesper ------ Jesper Lykkegaard Karlsen

[ceph-users] Re: replacing OSD nodes

the same amount of PGs are backfilling. Can large disk usage on mons slow down backfill and other operations? Is it dangerous? Best, Jesper -- Jesper Lykkegaard Karlsen Scientific Computing Centre for Structural Biology Department of Molecular Biology and Genetics

[ceph-users] Re: replacing OSD nodes

Cool thanks a lot! I will definitely put it in my toolbox. Best, Jesper -- Jesper Lykkegaard Karlsen Scientific Computing Centre for Structural Biology Department of Molecular Biology and Genetics Aarhus University Universitetsbyen 81 8000 Aarhus C E-mail: je

[ceph-users] Potential bug in cephfs-data-scan?

recently deprecated Octopus, I suspect that this bug is also present in Pacific and Quincy? It might be related to this bug? https://tracker.ceph.com/issues/46166 But symptoms are different. Or, maybe there is a way to disable the following of symlinks in "cephfs-data-scan pg_

[ceph-users] Re: Potential bug in cephfs-data-scan?

Fra: Patrick Donnelly Sendt: 19. august 2022 16:16 Til: Jesper Lykkegaard Karlsen Cc: ceph-users@ceph.io Emne: Re: [ceph-users] Potential bug in cephfs-data-scan? On Fri, Aug 19, 2022 at 5:02 AM Jesper Lykkegaard Karlsen wrote: >> > >Hi,

[ceph-users] Re: Potential bug in cephfs-data-scan?

Actually, it might have worked better if the PG had stayed down while running cephfs-data-scan, as it could only then get file structure from metadata pool and not touch each file/link in data pool? This would at least properly have given the list of files in (only) the affected PG? //Jesper

[ceph-users] Remove corrupt PG

objects in that PG (also with objectstore-tool), but this process is extremely slow. When looping over the >65,000 object, each remove takes ~10 sec and is very compute intensive, which is approximately 7.5 days. Is the a faster way to get around this? Mvh. Jes

[ceph-users] Re: Remove corrupt PG

came back online. Best, Jesper -- Jesper Lykkegaard Karlsen Scientific Computing Centre for Structural Biology Department of Molecular Biology and Genetics Aarhus University Universitetsbyen 81 8000 Aarhus C E-mail: je...@mbg.au.dk Tlf:+45 50906203 > On 31 Aug 2

[ceph-users] Re: Remove corrupt PG

? Best, Jesper -- Jesper Lykkegaard Karlsen Scientific Computing Centre for Structural Biology Department of Molecular Biology and Genetics Aarhus University Universitetsbyen 81 8000 Aarhus C E-mail: je...@mbg.au.dk Tlf:+45 50906203 > On 1 Sep 2022, at 22.01, Jes

[ceph-users] force-create-pg not working

y with something like this?: * set nobackfill and norecovery * delete the pgs shards one by one * unset nobackfill and norecovery Any idea on how to proceed from here is most welcome. Thanks, Jesper -- Jesper Lykkegaard Karlsen Scientific Computing Cen

[ceph-users] Re: force-create-pg not working

p OSD - repair - mark-complete on the primary OSD. A scrub tells me that the "active+clean” state is for real. I also found out the more automated "force-create-pg" command only works on pgs that a in down state. Best, Jesper ------ Jesper Lykkegaar

[ceph-users] cephfs quota used

, Jesper -- Jesper Lykkegaard Karlsen Scientific Computing Centre for Structural Biology Department of Molecular Biology and Genetics Aarhus University Gustav Wieds Vej 10 8000 Aarhus C E-mail: je...@mbg.au.dk Tlf:+45 50906203

[ceph-users] Re: cephfs quota used

h gives me no output. Should that not list all atributes? This is on Rocky Linux kernel 4.18.0-348.2.1.el8_5.x86_64 Best, Jesper ------ Jesper Lykkegaard Karlsen Scientific Computing Centre for Structural Biology Department of Molecular Biology and Genetics Aarhus University Gu

[ceph-users] Re: cephfs quota used

Just tested: getfattr -n ceph.dir.rbytes $DIR Works on CentOS 7, but not on Ubuntu 18.04 eighter. Weird? Best, Jesper -- Jesper Lykkegaard Karlsen Scientific Computing Centre for Structural Biology Department of Molecular Biology and Genetics Aarhus University Gustav

[ceph-users] Re: cephfs quota used

Woops, wrong copy/pasta: getfattr -n ceph.dir.rbytes $DIR works on all distributions I have tested. It is: getfattr -d -m 'ceph.*' $DIR that does not work on Rocky Linux 8, Ubuntu 18.04, but works on CentOS 7. Best, Jesper ------ Jesper Lykkegaard Karlsen

[ceph-users] Re: cephfs quota used

s dir size in "human-readble" It works like a charm and my god it is fast!. Tools like that could be very useful, if provided by the development team 🙂 Best, Jesper -- Jesper Lykkegaard Karlsen Scientific Computing Centre for Structural Biology Department of

[ceph-users] Re: cephfs quota used

Brilliant, thanks Jean-François Best, Jesper -- Jesper Lykkegaard Karlsen Scientific Computing Centre for Structural Biology Department of Molecular Biology and Genetics Aarhus University Gustav Wieds Vej 10 8000 Aarhus C E-mail: je...@mbg.au.dk Tlf:+45 50906203

[ceph-users] Re: cephfs quota used

Not to spam, but to make it output prettier, one can also separate the number from the byte-size prefix. numfmt --to=iec --suffix=B --padding=7 $(getfattr --only-values -n ceph.dir.rbytes $1 2>/dev/nul) | sed -r 's/([0-9])([a-zA-Z])/\1 \2/g; s/([a-zA-Z])([0-9])/\1 \2/g&#x

[ceph-users] Re: cephfs quota used

dir.rbytes $i 2>/dev/nul) | sed -r 's/([0-9])([a-zA-Z])/\1 \2/g; s/([a-zA-Z])([0-9])/\1 \2/g') $i" fi done The above can be run as: ceph_du_dir $DIR with multiple directories: ceph_du_dir $DIR1 $DIR2 $DIR3 .. Or even with wildcard: ceph_du_dir $DIR/* Best, Jesper --

[ceph-users] Healthy objects trapped in incomplete pgs

ible of those files to other location. * recreate pgs: ceph osd force-create-pg * restart recovery:ceph osd unset norecover * copy back in the recovered files Would that work or do you have a better suggestion? Ch

[ceph-users] Cephadm stacktrace on copying ceph.conf

python3.6/site-packages/asyncssh/scp.py", line 224, in make_request raise exc asyncssh.sftp.SFTPFailure: scp: /tmp/var/lib/ceph/5c384430-da91-11ed-af9c-c780a5227aff/config/ceph.conf.new: Permission denied 3/26/24 9:38:09 PM[INF]Updating dkcphhpcmgt028:/var/lib/ceph/5c384430-da91-11ed-af9c-c780a5227aff