Hi Yeikel, Neat question! And thanks for the link to the RestClient code; very helpful.
I don't believe there's a way to configure Kafka Connect to add these headers to forwarded requests right now. You may be able to do some kind of out-of-band proxy magic to intercept forwarded requests and insert the proper headers there? I don't see a reason for Kafka Connect to only forward authorization headers, even after examining the PR [1] and corresponding Jira ticket [2] that altered the RestClient class to begin including authorization headers in forwarded REST requests. We may be able to tweak the RestClient to include all headers instead of just the authorization header. I know that this doesn't help your immediate situation, but if other committers and contributors agree that the change would be beneficial, we may be able to include it in the next release (which may be 3.7.0, or a patch release for 3.4, 3.5, or 3.6). Alternatively, we may have to gate such a change behind a feature flag (either a coarse-grained boolean that enables/disables forwarding of all non-authorization headers, or more fine-grained logic such as include/exclude lists or even regexes), which would require a KIP and may take longer to release. I've CC'd the dev list to gather their perspective on this potential change, and to solicit their input on possible workarounds that may be useful to you sooner than the next release takes place. [1] - https://github.com/apache/kafka/pull/6791 [2] - https://issues.apache.org/jira/browse/KAFKA-8404 Cheers, Chris On Fri, Oct 6, 2023 at 10:14 PM Yeikel Santana <em...@yeikel.com> wrote: > Hello everyone, > > I'm currently running Kafka Connect behind a firewall that mandates the > inclusion of a specific header. This situation becomes particularly > challenging when forwarding requests among multiple workers, as it appears > that only the Authorization header is included in the request. > > I'm wondering if there's a way to customize the headers of Kafka Connect > before they are forwarded between workers. From my observations, it seems > that this capability may not be available[1], and only the response headers > can be customized. > > I'd appreciate any realistic alternatives or suggestions you may have in > mind. > > Thanks! > > > > > > > [1] > https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/RestClient.java#L191-L198
