Hi,
first, you could have prevented inactive PGs by temporarily reducing
min_size to 4 (default is k + 1). But don't leave min_size at k,
that's too dangerous.
1. In an EC 4+2 pool design, is such a cascading RGW failure considered
inevitable if enough hosts fail to cause PG unavailability, or are there
established best practices to prevent this behavior?
I am not surprised about the RGW failure, although we haven't seen
that happening on any cluster yet. How many OSD hosts do you have in
total? You could add more nodes and use a different EC profile to be
able to sustain the loss of two hosts without service interruption.
But it really depends on your actual resiliency requirements. Now that
you lost two and had a service disruption, it might be a good idea to
reconsider.
2. Based on our observation that internal RGWs remained healthy, is
isolating customers by assigning separate RGWs to high-traffic and
low-traffic groups considered a recommended architectural approach?
I don't think it makes a difference, inactive PGs stall IO. Maybe
there just wasn't enough client traffic to overload the internal RGWs,
which looks as if they were "healthy"? But from a general perspective,
it can make sense to dedicate RGWs to specific tasks.
Regards,
Eugen
Zitat von Ramin Najjarbashi <[email protected]>:
Hi
We recently experienced an incident in one of our production Ceph clusters
and would greatly appreciate the community’s input.
Our configuration uses Erasure Coding with a 4+2 profile for the main S3
data pools. During the incident, two storage hosts that contained several
OSDs each became unavailable simultaneously. As a result, a large number of
PGs entered an inactive state because the number of remaining fragments was
insufficient for reconstruction under the EC 4+2 profile.
This triggered a cascading failure in the RGW service layer:
• Incoming S3 requests continued flowing into HAProxy.
• RGW attempted to process requests that required I/O to inactive PGs.
• I/O stalls caused excessive queue buildup and memory usage on RGWs.
• Multiple RGW processes were terminated by the OOM Killer.
• RGW daemons repeatedly restarted and were again selected by HAProxy.
• The cycle continued, leading to severe service disruption.
This behavior continued until the failed hosts were recovered and PGs
regained a clean and active state.
One interesting observation was that a separate set of RGW daemons
dedicated only to internal consumers continued operating normally. Their
traffic load was significantly lower, which prevented queue saturation and
memory exhaustion.
We are seeking guidance on two key questions:
1. In an EC 4+2 pool design, is such a cascading RGW failure considered
inevitable if enough hosts fail to cause PG unavailability, or are there
established best practices to prevent this behavior?
2. Based on our observation that internal RGWs remained healthy, is
isolating customers by assigning separate RGWs to high-traffic and
low-traffic groups considered a recommended architectural approach?
Thank you in advance for any advice and shared experiences.
Best regards,
Ramin
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
