[
https://issues.apache.org/jira/browse/FLINK-40109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18120185#comment-18120185
]
Martijn Visser commented on FLINK-40109:
----------------------------------------
The keytab line is never missing: the container log is rolled shortly after
startup, and the login line lands in jobmanager.log.1 or taskmanager.log.1,
which the test does not read. All 10 failed checks in the 8 failing runs I
downloaded read a split log. None of the other 34 secured-test logs was split.
The rollover itself is FLINK-XXXXX. Last two weeks: 6 of 113 misc runs on
master, 5 of 88 on release-2.3, e.g.
https://github.com/apache/flink/actions/runs/36019885287 and
https://github.com/apache/flink/actions/runs/35553004650.
> YARNSessionFIFOSecuredITCase.testDetachedMode is flaky: Kerberos keytab check
> runs before the YARN application is killed
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40109
> URL: https://issues.apache.org/jira/browse/FLINK-40109
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, Tests
> Affects Versions: 2.3.1, 2.4.0
> Reporter: Martijn Visser
> Assignee: Martijn Visser
> Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 2.4.0
>
>
> {{YARNSessionFIFOSecuredITCase.testDetachedMode}} fails intermittently on
> master since FLINK-40099, e.g. GHA run 29005398820 (job 86077406039, module
> {{misc}}):
> {noformat}
> org.apache.flink.util.FlinkException: Exhausted retry attempts.
> at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:175)
> at
> org.apache.flink.yarn.YARNSessionFIFOSecuredITCase.verifyKerberosKeytabInLogs(YARNSessionFIFOSecuredITCase.java:209)
> at
> org.apache.flink.yarn.YARNSessionFIFOITCase.runDetachedModeTest(YARNSessionFIFOITCase.java:201)
> {noformat}
> Root cause
> FLINK-40099 moved the Kerberos keytab log verification into a callback that
> runs after the job reaches FINISHED but before the application is killed, and
> polls the logs with {{waitUntilCondition(condition, 500)}}. Two problems:
> The {{500}} {{int}} literal binds to the {{waitUntilCondition(condition, int
> retryAttempts)}} overload (500 attempts × 100ms, then {{"Exhausted retry
> attempts."}}) rather than the intended millisecond interval.
> More fundamentally, reading the JobManager log while the application is still
> alive is racy. The container {{jobmanager.log}} that is in scope does not
> reliably contain the keytab login line before teardown: the bootstrap output
> (which carries the login line) and the running-JobManager output land in
> different log files, and the pre-kill content is not durably readable. The
> keytab line is present once the application has been torn down and its logs
> flushed.
> The original FLINK-17662 concern (a short-lived TaskManager's startup login
> line being briefly unreadable ~200ms after teardown) is real, but the fix for
> it should be a bounded post-kill poll, not relocating the check to before the
> kill.
> Proposed fix
> Revert FLINK-40099's pre-kill callback and read the container logs after the
> application is killed (output flushed), replacing the original single read
> with a bounded {{CommonTestUtils.waitUtil(...)}} poll that fails with a
> message naming the log file and the missing strings.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
