[
https://issues.apache.org/jira/browse/CASSSIDECAR-465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arjun Ashok updated CASSSIDECAR-465:
------------------------------------
Description:
*Update (see comments):* _fixing this requires also fixing a bug in
{{RequestExecutor.schedule()}} that causes every delayed retry (including the
existing 503 {{Retry-After}} path) to fire immediately in addition to firing
after the delay. This is a prerequisite for this ticket's fix; see comment for
details._
The sidecar server returns HTTP 429 in two cases: when the concurrent upload
limit is exceeded (SSTable uploads) and when the stream request rate limit is
exhausted (SSTable component streaming). In both cases the client has no
meaningful retry behavior for this response.
BasicRetryPolicy, which all retry policy implementations extend, handles 503
SERVICE_UNAVAILABLE specially by parsing the Retry-After response header and
applying backoff before retrying. HTTP 429 falls into the generic 4xx catch-all
instead, which either retries immediately on a different host with no backoff,
likely hitting the same limit again or fails permanently if no alternative host
is available.
Gap on server side: SSTableUploadHandler does not include a Retry-After header
in its 429 response, unlike the streaming path which does. This means even with
a client-side fix, upload callers cannot benefit from server-indicated retry
timing.
Expected behavior:
BasicRetryPolicy should handle 429 analogously to 503, parsing Retry-After if
present and otherwise applying exponential backoff before retrying. Since all
retry policies extend BasicRetryPolicy, this fix applies uniformly across all
APIs. Additionally, SSTableUploadHandler should include a Retry-After header in
its 429 response so the client can back off for an appropriate duration.
was:
The sidecar server returns HTTP 429 in two cases: when the concurrent upload
limit is exceeded (SSTable uploads) and when the stream request rate limit is
exhausted (SSTable component streaming). In both cases the client has no
meaningful retry behavior for this response.
BasicRetryPolicy, which all retry policy implementations extend, handles 503
SERVICE_UNAVAILABLE specially by parsing the Retry-After response header and
applying backoff before retrying. HTTP 429 falls into the generic 4xx catch-all
instead, which either retries immediately on a different host with no backoff,
likely hitting the same limit again or fails permanently if no alternative host
is available.
Gap on server side: SSTableUploadHandler does not include a Retry-After header
in its 429 response, unlike the streaming path which does. This means even with
a client-side fix, upload callers cannot benefit from server-indicated retry
timing.
Expected behavior:
BasicRetryPolicy should handle 429 analogously to 503, parsing Retry-After if
present and otherwise applying exponential backoff before retrying. Since all
retry policies extend BasicRetryPolicy, this fix applies uniformly across all
APIs. Additionally, SSTableUploadHandler should include a Retry-After header in
its 429 response so the client can back off for an appropriate duration.
> Improve HTTP 429 handling across sidecar server and client
> ----------------------------------------------------------
>
> Key: CASSSIDECAR-465
> URL: https://issues.apache.org/jira/browse/CASSSIDECAR-465
> Project: Sidecar for Apache Cassandra
> Issue Type: Bug
> Components: Bulk Analytics, Client
> Reporter: Arjun Ashok
> Priority: Major
>
> *Update (see comments):* _fixing this requires also fixing a bug in
> {{RequestExecutor.schedule()}} that causes every delayed retry (including the
> existing 503 {{Retry-After}} path) to fire immediately in addition to firing
> after the delay. This is a prerequisite for this ticket's fix; see comment
> for details._
> The sidecar server returns HTTP 429 in two cases: when the concurrent upload
> limit is exceeded (SSTable uploads) and when the stream request rate limit is
> exhausted (SSTable component streaming). In both cases the client has no
> meaningful retry behavior for this response.
> BasicRetryPolicy, which all retry policy implementations extend, handles 503
> SERVICE_UNAVAILABLE specially by parsing the Retry-After response header and
> applying backoff before retrying. HTTP 429 falls into the generic 4xx
> catch-all instead, which either retries immediately on a different host with
> no backoff, likely hitting the same limit again or fails permanently if no
> alternative host is available.
> Gap on server side: SSTableUploadHandler does not include a Retry-After
> header in its 429 response, unlike the streaming path which does. This means
> even with a client-side fix, upload callers cannot benefit from
> server-indicated retry timing.
> Expected behavior:
> BasicRetryPolicy should handle 429 analogously to 503, parsing Retry-After if
> present and otherwise applying exponential backoff before retrying. Since all
> retry policies extend BasicRetryPolicy, this fix applies uniformly across all
> APIs. Additionally, SSTableUploadHandler should include a Retry-After header
> in its 429 response so the client can back off for an appropriate duration.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
