[
https://issues.apache.org/jira/browse/CASSSIDECAR-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18100923#comment-18100923
]
Arjun Ashok commented on CASSSIDECAR-465:
-----------------------------------------
*Prerequisite details:*
The plan is to add a 429 case to {{BasicRetryPolicy}} that reads
{{Retry-After}} and waits before retrying.
Currently {{RequestExecutor.schedule()}}
({{{}client/.../RequestExecutor.java:369-375{}}}) schedules the delayed retry
and runs it immediately anyway on the next line, unconditionally. So the retry
fires right away regardless of any delay {{BasicRetryPolicy}} computes, plus a
redundant second retry.
Net effect if we only fix {{{}BasicRetryPolicy{}}}: instead of failing after
one try, the client fires a burst of retries almost instantly, with no real
wait, straight back at a sidecar that is at capacity.
Note: this also affects the 503 path today.
> Improve HTTP 429 handling across sidecar server and client
> ----------------------------------------------------------
>
> Key: CASSSIDECAR-465
> URL: https://issues.apache.org/jira/browse/CASSSIDECAR-465
> Project: Sidecar for Apache Cassandra
> Issue Type: Bug
> Components: Bulk Analytics, Client
> Reporter: Arjun Ashok
> Priority: Major
>
> *Update (see comments):* _fixing this requires also fixing a bug in
> {{RequestExecutor.schedule()}} that causes every delayed retry (including the
> existing 503 {{Retry-After}} path) to fire immediately in addition to firing
> after the delay. This is a prerequisite for this ticket's fix; see comment
> for details._
> The sidecar server returns HTTP 429 in two cases: when the concurrent upload
> limit is exceeded (SSTable uploads) and when the stream request rate limit is
> exhausted (SSTable component streaming). In both cases the client has no
> meaningful retry behavior for this response.
> BasicRetryPolicy, which all retry policy implementations extend, handles 503
> SERVICE_UNAVAILABLE specially by parsing the Retry-After response header and
> applying backoff before retrying. HTTP 429 falls into the generic 4xx
> catch-all instead, which either retries immediately on a different host with
> no backoff, likely hitting the same limit again or fails permanently if no
> alternative host is available.
> Gap on server side: SSTableUploadHandler does not include a Retry-After
> header in its 429 response, unlike the streaming path which does. This means
> even with a client-side fix, upload callers cannot benefit from
> server-indicated retry timing.
> Expected behavior:
> BasicRetryPolicy should handle 429 analogously to 503, parsing Retry-After if
> present and otherwise applying exponential backoff before retrying. Since all
> retry policies extend BasicRetryPolicy, this fix applies uniformly across all
> APIs. Additionally, SSTableUploadHandler should include a Retry-After header
> in its 429 response so the client can back off for an appropriate duration.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
