SEPURI-SAI-KRISHNA commented on PR #13010: URL: https://github.com/apache/seatunnel/pull/13010#issuecomment-6543900546
@DanielLeens thanks for the review, and sorry for the slow reply. **Issue 1.** The job log shows a different cause than the checkpoint timing. Zeta does commit on a finished batch job: `KafkaIT.testSourceKafkaStartConfigWithCommitOffset` asserts a committed offset and passed in the same job. The commit here failed because of a broker race: ``` 10:11:42,093 kafka: Creating topic __consumer_offsets ... 10:11:42,193 WARN KafkaSourceReader - Failed to commit consumer offsets for checkpoint 9223372036854775807 Caused by: org.apache.kafka.common.errors.CoordinatorNotAvailableException ``` Each test class starts its own broker, and in `KafkaFormatIT` no consumer group commits before this test does. So the first commit in that broker triggers the creation of `__consumer_offsets`, arrives 100 ms later before a coordinator is ready, and the source only logs the failure. The new commit calls `describeConsumerGroups` for the run's group before each job. The admin client retries the coordinator lookup until the broker answers, and it commits nothing, so the offset assertion still passes only on SeaTunnel's own commit. I left the job config alone (no `read_limit` or `checkpoint.interval`), because the final checkpoint already commits. That the source does not retry a retriable commit failure is a separate product question. I will look at it on its own rather than in this test PR. **Issue 2.** `getPostgreSinkTableList`, which `checkCompatibleFormat` uses, truncates the table after every read, so the second run already started empty. Rows could still be left over if an earlier test failed between its job and its check, so each run now reads (and so truncates) the table before the job. **Issue 3.** `TimeoutException` is imported in the lines this PR adds, and the Spark settings are removed from the conf. Checks: spotless clean on the module; the new helper compiles against kafka-clients 3.4.0, the version the test uses. Docker is not available here, so CI is the runtime check. @SEZ9 this is the fix I described on #12966. I pushed it since it only touches the test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
