[ceph-users] osd daemons still reading disks at full speed while there is no pool activity

2021-11-03 Thread Nikola Ciprich
Hello fellow ceph users, I'm trying to catch ghost here.. On one of our clusters, 6 nodes, 14.2.15, EC pool 4+2, 6*32 SATA bluestore OSDs we got into very strange state. The cluster is clean (except for pgs not deep-scrubbed in time warning, since we've disabled scrubbing while investigating), th

[ceph-users] Re: osd daemons still reading disks at full speed while there is no pool activity

2021-11-03 Thread Eugen Block
Hi, I don't have an explanation but I remember having a similar issue a year ago or so. IIRC a simple OSD restart fixed that, so I never got to the bottom of it. Have you tried to restart OSD daemons? Zitat von Nikola Ciprich : Hello fellow ceph users, I'm trying to catch ghost here.. O

[ceph-users] Re: osd daemons still reading disks at full speed while there is no pool activity

2021-11-03 Thread Nikola Ciprich
Hello Eugen, thank you for you reply. Yes, restarting all OSDs, monitors, also increasing osd_map_cache_size to 5000 (this helped us in case of problem with not pruning OSD maps). none of this helped.. with best regards nik On Wed, Nov 03, 2021 at 11:41:28AM +, Eugen Block wrote: > Hi, > >

[ceph-users] Re: osd daemons still reading disks at full speed while there is no pool activity

2021-11-03 Thread Nikola Ciprich
Hello Josh, > > Was there PG movement (backfill) happening in this cluster recently? > Do the OSDs have stray PGs (e.g. 'ceph daemon osd.NN perf dump | grep > numpg_stray' - run this against an affected OSD from the housing > node)? yes, some nodes have stray pgs (1..5) shell I do something about

[ceph-users] ceph-ansible and crush location

2021-11-03 Thread Simon Oosthoek
Dear list, I've recently found it is possible to supply ceph-ansible with information about a crush location, however I fail to understand how this is actually used. It doesn't seem to have any effect when create a cluster from scratch (I'm testing on a bunch of vm's generated by vagrant and

[ceph-users] Re: Slow S3 Requests

2021-11-03 Thread Eugen Block
Hi, it's possible that the log messages are a consequence of the slow requests. Having only 4 OSDs is kind of a corner case, ceph is designed as a scalable solution, so the real benefits start with a large number of OSDs so you can parallelize many client requests to many OSDs. And with o

[ceph-users] Re: ceph-ansible and crush location

2021-11-03 Thread Simon Oosthoek
On 03/11/2021 15:48, Stefan Kooman wrote: On 11/3/21 15:35, Simon Oosthoek wrote: Dear list, I've recently found it is possible to supply ceph-ansible with information about a crush location, however I fail to understand how this is actually used. It doesn't seem to have any effect when creat

[ceph-users] Re: Upgrade to 16.2.6 and osd+mds crash after bluestore_fsck_quick_fix_on_mount true

2021-11-03 Thread Francois Legrand
Hello, Can you confirm that the bug only affects pacific and not octopus ? Thanks. F. Le 29/10/2021 à 16:39, Neha Ojha a écrit : On Thu, Oct 28, 2021 at 8:11 AM Igor Fedotov wrote: On 10/28/2021 12:36 AM, mgrzybowski wrote: Hi Igor I'm very happy that You ware able to reproduce and find t