[
https://issues.apache.org/jira/browse/FLINK-40619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dale Lane updated FLINK-40619:
------------------------------
Description:
{{DynamicKafkaSourceReaderTest.testHandleSourceEventRetainsRemovedClusterOffsetsUntilExpired}}
fails intermittently under CPU contention with
{{java.util.NoSuchElementException: No value present}}
{{at java.util.Optional.orElseThrow(Optional.java:377)}}
{{at
DynamicKafkaSourceReaderTest.testHandleSourceEventRetainsRemovedClusterOffsetsUntilExpired(DynamicKafkaSourceReaderTest.java:294)}}
(It passes consistently on an unloaded developer machine, but fails
intermittently in pull request workflows.)
The behaviour under test is time-based. When a cluster is removed,
{{DynamicKafkaSourceReader}} gives the affected splits a deadline of "now plus
the configured retention", and {{snapshotState()}} prunes splits whose deadline
has passed before returning them.
The test configures a retention window of one second and then asserts against
the retained split across several {{snapshotState()}} calls. The whole sequence
must therefore complete within one second of wall-clock time. A garbage
collection or scheduling pause longer than that expires the split before the
assertions read it, and the lookup finds nothing.
The reader reads the current time directly from the system clock at each of the
three points where retention is stamped or evaluated. There is no way for a
test to control the passage of time, so an expiry test has to pick a real-time
window and race it. The short window is fragile when running in the Github CI
workers under load (and increasing it to a long window would not assert that
expiry behaviour is really correct).
was:
{{DynamicKafkaSourceReaderTest.testHandleSourceEventRetainsRemovedClusterOffsetsUntilExpired}}
fails intermittently under CPU contention with
{{java.util.NoSuchElementException: No value present
at java.util.Optional.orElseThrow(Optional.java:377)
at
DynamicKafkaSourceReaderTest.testHandleSourceEventRetainsRemovedClusterOffsetsUntilExpired(DynamicKafkaSourceReaderTest.java:294)}}
(It passes consistently on an unloaded developer machine, but fails
intermittently in pull request workflows.)
The behaviour under test is time-based. When a cluster is removed,
{{DynamicKafkaSourceReader}} gives the affected splits a deadline of "now plus
the configured retention", and {{snapshotState()}} prunes splits whose deadline
has passed before returning them.
The test configures a retention window of one second and then asserts against
the retained split across several {{snapshotState()}} calls. The whole sequence
must therefore complete within one second of wall-clock time. A garbage
collection or scheduling pause longer than that expires the split before the
assertions read it, and the lookup finds nothing.
The reader reads the current time directly from the system clock at each of the
three points where retention is stamped or evaluated. There is no way for a
test to control the passage of time, so an expiry test has to pick a real-time
window and race it. The short window is fragile when running in the Github CI
workers under load (and increasing it to a long window would not assert that
expiry behaviour is really correct).
> DynamicKafkaSourceReaderTest.testHandleSourceEventRetainsRemovedClusterOffsetsUntilExpired
> flaky - removed-cluster retention expiry cannot be tested deterministically
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40619
> URL: https://issues.apache.org/jira/browse/FLINK-40619
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
> Reporter: Dale Lane
> Priority: Major
> Labels: test-flaky, test-stability
>
> {{DynamicKafkaSourceReaderTest.testHandleSourceEventRetainsRemovedClusterOffsetsUntilExpired}}
> fails intermittently under CPU contention with
> {{java.util.NoSuchElementException: No value present}}
> {{at java.util.Optional.orElseThrow(Optional.java:377)}}
> {{at
> DynamicKafkaSourceReaderTest.testHandleSourceEventRetainsRemovedClusterOffsetsUntilExpired(DynamicKafkaSourceReaderTest.java:294)}}
> (It passes consistently on an unloaded developer machine, but fails
> intermittently in pull request workflows.)
> The behaviour under test is time-based. When a cluster is removed,
> {{DynamicKafkaSourceReader}} gives the affected splits a deadline of "now
> plus the configured retention", and {{snapshotState()}} prunes splits whose
> deadline has passed before returning them.
> The test configures a retention window of one second and then asserts against
> the retained split across several {{snapshotState()}} calls. The whole
> sequence must therefore complete within one second of wall-clock time. A
> garbage collection or scheduling pause longer than that expires the split
> before the assertions read it, and the lookup finds nothing.
> The reader reads the current time directly from the system clock at each of
> the three points where retention is stamped or evaluated. There is no way for
> a test to control the passage of time, so an expiry test has to pick a
> real-time window and race it. The short window is fragile when running in the
> Github CI workers under load (and increasing it to a long window would not
> assert that expiry behaviour is really correct).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
