[
https://issues.apache.org/jira/browse/FLINK-40109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18120048#comment-18120048
]
Martijn Visser commented on FLINK-40109:
----------------------------------------
Four more in the last week, two in each shape.
master pushes, the poll timing out (jobmanager.log once, taskmanager.log once):
https://github.com/apache/flink/actions/runs/36169063884/job/108189546561
(2026-09-25, Java 17, misc)
https://github.com/apache/flink/actions/runs/36251484841/job/108431829012
(2026-09-26, Java 17, misc)
{code}
[ERROR] org.apache.flink.yarn.YARNSessionFIFOSecuredITCase.testDetachedMode --
Time elapsed: 85.01 s <<< ERROR!
java.util.concurrent.TimeoutException: Kerberos keytab login [Login successful
for user, using keytab file] not found in jobmanager.log for application
application_1790360272882_0003 within the timeout; inspect the uploaded
container logs.
at
org.apache.flink.core.testutils.CommonTestUtils.waitUtil(CommonTestUtils.java:218)
at
org.apache.flink.yarn.YARNSessionFIFOSecuredITCase.verifyResultContainsKerberosKeytab(YARNSessionFIFOSecuredITCase.java:204)
{code}
release-2.3 nightlies, which still have the single read, on the original
assertion:
https://github.com/apache/flink/actions/runs/36210383509/job/108324228012
(2026-09-26, Hadoop 3.1.3, misc)
https://github.com/apache/flink/actions/runs/36287526040/job/108535567351
(2026-09-27, Java 11, misc)
{code}
[ERROR] org.apache.flink.yarn.YARNSessionFIFOSecuredITCase.testDetachedMode --
Time elapsed: 26.11 s <<< FAILURE!
org.opentest4j.AssertionFailedError:
Expecting value to be true but was false
at
org.apache.flink.yarn.YARNSessionFIFOSecuredITCase.verifyResultContainsKerberosKeytab(YARNSessionFIFOSecuredITCase.java:201)
{code}
> YARNSessionFIFOSecuredITCase.testDetachedMode is flaky: Kerberos keytab check
> runs before the YARN application is killed
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40109
> URL: https://issues.apache.org/jira/browse/FLINK-40109
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, Tests
> Affects Versions: 2.4.0
> Reporter: Martijn Visser
> Assignee: Martijn Visser
> Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 2.4.0
>
>
> {{YARNSessionFIFOSecuredITCase.testDetachedMode}} fails intermittently on
> master since FLINK-40099, e.g. GHA run 29005398820 (job 86077406039, module
> {{misc}}):
> {noformat}
> org.apache.flink.util.FlinkException: Exhausted retry attempts.
> at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:175)
> at
> org.apache.flink.yarn.YARNSessionFIFOSecuredITCase.verifyKerberosKeytabInLogs(YARNSessionFIFOSecuredITCase.java:209)
> at
> org.apache.flink.yarn.YARNSessionFIFOITCase.runDetachedModeTest(YARNSessionFIFOITCase.java:201)
> {noformat}
> Root cause
> FLINK-40099 moved the Kerberos keytab log verification into a callback that
> runs after the job reaches FINISHED but before the application is killed, and
> polls the logs with {{waitUntilCondition(condition, 500)}}. Two problems:
> The {{500}} {{int}} literal binds to the {{waitUntilCondition(condition, int
> retryAttempts)}} overload (500 attempts × 100ms, then {{"Exhausted retry
> attempts."}}) rather than the intended millisecond interval.
> More fundamentally, reading the JobManager log while the application is still
> alive is racy. The container {{jobmanager.log}} that is in scope does not
> reliably contain the keytab login line before teardown: the bootstrap output
> (which carries the login line) and the running-JobManager output land in
> different log files, and the pre-kill content is not durably readable. The
> keytab line is present once the application has been torn down and its logs
> flushed.
> The original FLINK-17662 concern (a short-lived TaskManager's startup login
> line being briefly unreadable ~200ms after teardown) is real, but the fix for
> it should be a bounded post-kill poll, not relocating the check to before the
> kill.
> Proposed fix
> Revert FLINK-40099's pre-kill callback and read the container logs after the
> application is killed (output flushed), replacing the original single read
> with a bounded {{CommonTestUtils.waitUtil(...)}} poll that fails with a
> message naming the log file and the missing strings.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
