KelvinChi opened a new pull request, #9371:
URL: https://github.com/apache/seatunnel/pull/9371
The following content cannot be set in the config configuration file to
consume different tables in Kafka: database.include or table.include. (#6870)
### Purpose of this pull request
This PR fixes the issue where `database.include` and `table.include`
configurations couldn't be properly set in Kafka source config for JSON format
consumption. The fix ensures these filters are correctly applied when parsing
Canal/OGG/Maxwell JSON formats.
### Does this PR introduce any user-facing change?
Yes. Users can now properly configure `database.include` and `table.include`
in Kafka source config to filter specific tables when consuming:
1. Canal JSON format
2. OGG JSON format
3. Maxwell JSON format
Previously these configurations were silently ignored, now they will take
effect as expected.
### How was this patch tested?
1. Added unit tests covering all three JSON format parsers with
database/table filters
2. Manual verification using:
- Kafka with Canal-formatted messages
- Kafka with OGG-formatted messages
- Kafka with Maxwell-formatted messages
3. Verified the filters work correctly in both included and excluded
scenarios
Test config example:
```conf
source {
Kafka {
format = canal_json
database.include = "test_db"
table.include = "target_table"
# ... other configs
}
}
```
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [x] If you are contributing the connector code, please check that the
following files are updated:
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
