IIRC, pg merge is only done one PG at a time, so you have kind of a
„natural“ throttling, but it will take quite some time to complete.
What’s the output of ‚ceph osd df‘? Do you see ceph warnings? Do you
have any more info on the qemu errors, either on client side or in
ceph logs?
I can’t comment on qemu clones, we only do that with rbd (manually) or
let nova/glance/cinder handle it.
Zitat von Robert Lukan via ceph-users <[email protected]>:
Hello Ceph Community,
We are seeking advice regarding a strange behavior with cloning
functionality and a strategy for PG reduction on our current cluster.
*Environment:*
-
*Ceph Version:* Squid 19.2.3 (via Proxmox)
-
*Hardware:* Intel NVMe drives, with very good margin of resources (CPU,
RAM and bandwidth)
-
*Workload:* Active VM storage - RBD only (a couple 100 VMs, but no
extreme load)
*The Issue:* We are experiencing a failure with cloning functionality, but
it appears to be specific to which way we do it:
-
*Fails:* Cloning via QEMU (librbd).
-
*Succeeds:* Cloning via rbd clone or deep-copy.
-
*Succeeds:* Cloning when using krbd enabled on a pool
This cluster was created about a year ago for testing(PoC) and has evolved
into production. Initially cloning worked fine. Due to disk replacements,
expansions and misconfiguration, the autoscaler increased the total PG
count to 16k. We suspect this excessive PG count might be contributing to
the cloning failures, though we aren't certain.
We would like to safely decrease the number of PGs back to a standard range
without destabilizing the running VMs. Our plan is to:
1.
Decrease the PG count in small increments.
2.
Monitor for I/O latency on the Intel NVMes.
*Questions for the list:*
1.
Are there known risks to "stepping down" PGs in increments of, say,
10-20% at a time while the cluster is under load?
2.
Given the 16k PG count, is there a specific "recovery/backfill" throttle
we should prioritize to ensure VM performance isn't impacted during the
merge process?
We are trying to decide if this approach is feasible or if we should create
a new cluster and migrate VMs. Any insights would be greatly appreciated.
Thank you,
Robert
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
