Ramin Gharib created FLINK-40099:
------------------------------------
Summary: YARNSessionFIFOSecuredITCase.testDetachedMode is flaky:
Kerberos keytab log check reads a killed TaskManager's log
Key: FLINK-40099
URL: https://issues.apache.org/jira/browse/FLINK-40099
Project: Flink
Issue Type: Bug
Reporter: Ramin Gharib
Assignee: Ramin Gharib
{{YARNSessionFIFOSecuredITCase.testDetachedMode}} (and
{{{}testDetachedModeSecureWithPreInstallKeytab{}}}) is flaky. It fails at
{{verifyResultContainsKerberosKeytab}} with:
{code:java}
assertThat(jobManagerRunsWithKerberos && taskManagerRunsWithKerberos).isTrue();
// was false
Expecting value to be true but was false{code}
h3. Root cause:
verifyResultContainsKerberosKeytab does a single-shot search of jobmanager.log
and taskmanager.log for "Login successful for user" and "using keytab file". It
runs after runDetachedModeTest has already killed the YARN application.
The TaskManager container is short-lived. In an observed failure it was
launched, ran the job to FINISHED, was SIGTERM-killed (exit 143) all within
about 3 seconds, and the test read its log about 200ms after the kill. At that
instant the TaskManager's startup keytab-login line was not yet durably
readable, so the assertion flipped to false. The JobManager log check passed.
It is a sub-second race between the TaskManager log flush and the kill plus
read.
This matches the long-standing intermittent failure tracked as FLINK-17662
(closed Cannot Reproduce), which reported the same assertion.
h3. Fix:
Verify the keytab-login strings while the containers are still alive, before
the application is killed, and poll the log files instead of reading them once.
The token and ACL checks are unchanged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
