SEPURI-SAI-KRISHNA commented on issue #12483: URL: https://github.com/apache/seatunnel/issues/12483#issuecomment-6208371339
@goutamadwant a connector-only PR of mine just produced measurements for the `updated-modules-integration-test-part-N` jobs you describe, so I am leaving them here rather than opening a separate issue. They support your point that these shards always run the full engine set, and they put numbers on the cost. The change is #12650, two files in `seatunnel-connectors-v2/connector-common`. Because nearly every connector depends on that module, `it-modules` expands to the whole connector E2E closure. Fork run `38523089111`, attempt 4, JDK 11 and 17: | shard | `timeout-minutes` | JDK 11 | JDK 17 | | --- | --- | --- | --- | | part-1, part-2 | 180 | success | success | | part-3 | 300 | success | success | | part-4 | 200 | success | success | | part-5 | 180 | cancelled at 3h01m | cancelled at 3h01m | | part-6 | 210 | cancelled at 3h31m | cancelled at 3h31m | | part-7 | 210 | failure (unrelated flake) | success | | part-8 | 210 | success | success | Both legs of part-5 and part-6 hit their caps to the minute, on separate runners. That is workload rather than luck. **How far part-5 got.** It was given 10 modules and finished 6 inside 180 minutes: `cdc-sqlserver-e2e`, `file-cos-e2e`, `neo4j-e2e`, `cdc-oceanbase-e2e`, `web3j-e2e`, `file-ftp-e2e`. Never started: `connector-jdbc-e2e-part-2`, `activemq-e2e`, `email-e2e`, `fluss-e2e`. Roughly 30 minutes per module, so the shard needs somewhere near 330 to 360 minutes as it stands. GitHub's ceiling for a hosted job is 360. **Why the caps look arbitrary.** They are 180, 180, 300, 200, 180, 210, 210, 210 across the eight shards, which reads as reactive bumping. The split is round robin by list index in `get_sub_update_it_modules` (`i % total_num == current_num`), so it is blind to module cost: part-5 happened to draw three of the heaviest E2E modules by position alone. If your reduced engine set lands, a 40 to 55% cut would bring part-5 from roughly 300 minutes of work to roughly 140 to 180, which is inside its current cap without touching any timeout. So on this evidence the engine set is the lever, not the caps. One caveat on reading the above: this is a single PR on one fork, so treat it as one data point rather than a rate. Happy to pull the same numbers from another run if that would help. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
