JNSimba opened a new pull request, #709: URL: https://github.com/apache/doris-flink-connector/pull/709
## Problem Summary Incremental reads currently use a single table-wide split per time window and reject checkpoint restores with a different source parallelism. This change discovers row-binlog tablets through the query-plan API and distributes each finite window across readers using tablet-based hashing. A new window is discovered only after all readers finish the current work. - Persist tablet IDs and time bounds in stream splits without a reader ID. Repartition pending tablet groups using the current parallelism and queue multiple splits per reader on restore; retain Flink's normal reader-state restoration. - Publish completed-window offsets from the Enumerator after checkpoint completion, with serialized asynchronous writes. Remove reader-side publication and the custom FetcherManager wrappers in both Flink modules. - Request FE-side forwarding for incremental tablet discovery and restrict incremental queries to the assigned tablets. Retain deserialization of version 3 and version 4 split state; new state uses version 5. ## Validation - Flink 1.20 and Flink 2.3: `mvn -Pflink1/-Pflink2 -pl flink-doris-connector-flink1/flink-doris-connector-flink2 -am -DskipTests test` completed successfully, including production/test compilation and formatting checks. - Added coverage for pending-split rescaling, waiting for restored reader work before advancing windows, and split-state compatibility. The tests have not been rerun against the final revision. - End-to-end rescaling has not been tested. Reader-restored tablet groups can run on different readers from subsequent windows; strict per-tablet output ordering across that boundary is not guaranteed by the source window-completion check alone. ## Checklist 1. Does it affect the original behavior: Yes 2. Has unit tests been added: Yes 3. Has document been added or modified: No 4. Does it need to update dependencies: No 5. Are there any changes that cannot be rolled back: New version 5 checkpoint state cannot be read by older connector builds; rollback requires a checkpoint compatible with the older build. ## Further comments Incremental tablet discovery requires server-side support for forwarding the query-plan request and planning row-binlog tablet scans correctly. This draft is submitted for review before end-to-end rescaling validation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
