Bharath Reddy Gunapati created FLINK-40281:
----------------------------------------------
Summary: Improve HTTP sink failure observability
Key: FLINK-40281
URL: https://issues.apache.org/jira/browse/FLINK-40281
Project: Flink
Issue Type: Improvement
Components: Connectors / HTTP
Reporter: Bharath Reddy Gunapati
The HTTP lookup source exposes useful operational signals for HTTP calls,
including retry-related metrics and metadata-driven error visibility. The HTTP
sink currently has only
basic failure observability: failed records increment the generic sink
send-error counter, logs contain limited failure context, and the post-request
callback receives a null
response for client-side exceptions without the original cause.
As the sink gains configurable response classification and retry behavior,
operators need clearer visibility into why writes fail or retry. This ticket
improves HTTP sink observability for retryable responses, fatal responses,
ignored responses, retry exhaustion, and client-side exceptions.
Proposed Scope
- Add HTTP sink metrics for retryable response failures, fatal response
failures, ignored responses, retry exhaustion, and client-side exceptions.
- Add retry-attempt visibility, for example a retry attempt counter.
- Add status-code visibility, either per status code or grouped by status
class.
- Improve sink failure logs to include endpoint, HTTP method, status code
where available, batch size/request entry count, and retry attempt.
- Improve callback/failure reporting so client-side exceptions are observable
instead of only appearing as a null response.
- Document the new metrics and failure observability behavior.
Acceptance Criteria
- HTTP sink exposes metrics that distinguish retryable response failures,
fatal response failures, ignored responses, retry exhaustion, and client-side
exceptions.
- Sink logs include enough context to diagnose failed writes without enabling
full request/response body logging.
- Batch mode logs/metrics identify the number of request entries affected by
a single HTTP response.
- Client-side exceptions are observable through logs and/or callback/failure
reporting with the original cause preserved.
- Existing sink behavior remains unchanged unless users opt into any new
callback behavior.
- Unit tests cover metrics/logging or callback behavior for retryable, fatal,
ignored, exhausted, and exception paths.
- Documentation describes the new observability signals.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
