Hi,
since you mention HDD pools, maybe the spinners are the bottleneck?
Have you monitored the disk utilization with iostat (or similar)? Do
the HDDs have their DB/WAL on SSD?
Multi-active MDS can easily cause issues with the mds_balancer, so
pinning it to one rank could help. But have you also tried subtree
pinning to distribute the load between active ranks (but without
mds_balancing)? But I'd look at the HDD utilization first.
Regards,
Eugen
Zitat von Greg Ballantine via ceph-users <[email protected]>:
Hi everyone,
I’m reaching out for some guidance on troubleshooting periodic latency
spikes we’re seeing in our Ceph cluster. At times simple file operations
are occasionally taking several seconds to complete and this is impacting
our users.
We suspect the issue lies with our MDS load balancing, specifically an
imbalance where our rank 0 MDS consistently has ~3-4 billion open caps
(which seems to be about where its RAM maxes out), while the rank 1 and
rank 2 sit around 3-7 million.
Cluster Details
-
Ceph Version: 18.2.4
-
OS: QuantaStor 6.6.9
-
MDS Configuration: 3 active, 5 standby.
-
Storage Pools:
-
Data: 16 node HDD pool (8k+3m erasure coding).
-
Metadata: NVMe pool (triple replication).
-
We have ~150 nodes in an HPC cluster that use this storage in som
We’ve tried tuning some ceph config parameters to try and make the MDS
balance better, but this didn’t seem to help much:
-
mds mds_recall_max_caps 50000
-
mds mds_max_caps_per_client 100000
-
mds mds_bal_split_size 5000
-
mds mds_bal_interval 5
Additionally, we’ve also attempted pinning our highest-load user to the
rank 2 MDS. While this did successfully increase utilization on that rank,
we are still seeing significant latency spikes for small file read/write
operations across the board. If it helps, this user has a volume of 505TB,
we have ten other volumes in the 10-100TB range, and then 175 other volumes
under 10TB in usage.
Is our diagnosis correct that the MDS is what’s causing the poor latency
with file operations? If so, is it possible to have our MDS services
distribute load more evenly? And as another question, does the size of a
volume in ceph matter, meaning should we limit volume sizes to under 100TB
for example?
Thanks in advance for any guidance in diagnosing and resolving this issue.
Best regards,
Greg
--
Gregory Ballantine
System Administrator for Research and Enterprise Computing
UMBC - DoIT
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
