efcasado opened a new pull request, #25325: URL: https://github.com/apache/pulsar/pull/25325
### Motivation The current implementation of the Kafka Connect adaptor silently discards headers from the Kafka Connect SourceRecord instead of mapping them to Pulsar message properties. This is particularly impactful when using [Debezium's Outbox Event Router SMT](https://debezium.io/documentation//reference/3.5/transformations/outbox-event-router.html) with the [table.fields.additional.placement](https://debezium.io/documentation//reference/3.5/transformations/outbox-event-router.html#emitting-messages-with-additional-fields) option, which allows outbox table columns to be placed in the `header`, `envelope`, or `partition` of the resulting record. Columns configured with header placement are silently ignored because the adaptor never propagates Connect record headers to Pulsar message properties. Since Kafka Connect headers and Pulsar message properties are semantically equivalent (string key/value metadata outside the payload), the expected behavior is a direct mapping. A similar issue was previously identified for `pulsar-io-kafka` and addressed in [#17829](https://github.com/apache/pulsar/pull/17829). However, `kafka-connect-adaptor` was not covered by that fix and still silently discards Kafka Connect SourceRecord headers. ### Modifications `AbstractKafkaConnectSource` previously returned a hard-coded empty map from `getProperties()`, causing all Kafka Connect headers to be silently dropped. This change maps headers from the transformed `SourceRecord` to Pulsar message properties, preserving metadata that would otherwise be lost in transit (eg. columns placed in header position via Debezium's Outbox Event Router SMT and `table.fields.additional.placement`). ### Verifying this change - [x] Make sure that the change passes the CI checks. This change added tests and can be verified as follows: - Extended the existing `KafkaConnectSourceTest` test suite with an additional test to confirm Kafka headers are correctly propagated to the resulting record. ### Documentation - [ ] `doc` <!-- Your PR contains doc changes. --> - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later --> - [x] `doc-not-needed` <!-- Your PR changes do not impact docs --> - [ ] `doc-complete` <!-- Docs have been already added --> ### Matching PR in forked repository PR in forked repository: https://github.com/efcasado/pulsar/pull/2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
