[
https://issues.apache.org/jira/browse/FLINK-40618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purushottam Sinha updated FLINK-40618:
--------------------------------------
Description:
SourceTopicIntegrityTest is a heavyweight integration test -- it starts a
Testcontainers Kafka broker and a 3-TaskManager MiniCluster and drives
savepoint/restore -- but its *Test suffix places it in the unit-test surefire
execution (forkCount=4, 1GB heap). Its heavyweight siblings in the same
package, KafkaSourceITCase and KafkaSourceMigrationITCase, are already named
*ITCase and run in the integration-test execution (forkCount=2, 2GB heap,
reuseForks=false).
h3. Proposed change
Rename SourceTopicIntegrityTest -> SourceTopicIntegrityITCase so it is picked
up by the integration-test phase, consistent with those siblings. This reduces
the number of concurrent Testcontainers-Kafka + MiniCluster forks and gives the
test more heap and a fresh JVM per class. This is a test-classification cleanup
justified by the precedence of KafkaSourceITCase / KafkaSourceMigrationITCase.
was:
SourceTopicIntegrityTest is a heavyweight integration test -- it starts a
Testcontainers Kafka broker and a 3-TaskManager MiniCluster and drives
savepoint/restore -- but its *Test suffix places it in the unit-test surefire
execution (forkCount=4, 1GB heap). Its heavyweight siblings in the same
package, KafkaSourceITCase and KafkaSourceMigrationITCase, are already named
*ITCase and run in the integration-test execution (forkCount=2, 2GB heap,
reuseForks=false).
h3. Proposed change
Rename SourceTopicIntegrityTest -> SourceTopicIntegrityITCase so it is picked
up by the integration-test phase, consistent with those siblings. This reduces
the number of concurrent Testcontainers-Kafka + MiniCluster forks and gives the
test more heap and a fresh JVM per class. This is a test-classification cleanup
justified by the precedence of KafkaSourceITCase / KafkaSourceMigrationITCase.
h3. Relationship to the observed flakiness (FLINK-40620)
The test has been seen failing intermittently on CI with "Timeout waiting for
100 records in topic SourceTopicIntegrityTest_sink-topic after PT2M". Note this
is NOT a delivery-starvation timeout: the sink actually receives MORE than the
expected count -- the failing logs show "Found 150 records ... (expected: 100)"
(and up to 200 in other runs). The at-least-once sink redelivers records across
source re-reads/failover, and KafkaSourceTestEnv.waitForRecordsInTopic asserts
an EXACT count (count == expected), which can never recover once the count
overshoots, so it waits out the full PT2M.
The assertion-level fix for that is tracked separately in FLINK-40620
(dalelane, PR https://github.com/apache/flink-connector-kafka/pull/308), which
runs each subscription mode twice and asserts "no records lost" for
at-least-once while keeping the exact count only for exactly-once. This ITCase
move is complementary (reduces parallel resource pressure / matches
convention); it is not itself the fix for the over-count assertion.
Kept separate from FLINK-40620 by agreement on the PRs (PR #307 for this
change).
> Connectors/Kafka: Run SourceTopicIntegrity test in the integration-test phase
> -----------------------------------------------------------------------------
>
> Key: FLINK-40618
> URL: https://issues.apache.org/jira/browse/FLINK-40618
> Project: Flink
> Issue Type: Bug
> Reporter: Purushottam Sinha
> Priority: Major
> Labels: pull-request-available
>
> SourceTopicIntegrityTest is a heavyweight integration test -- it starts a
> Testcontainers Kafka broker and a 3-TaskManager MiniCluster and drives
> savepoint/restore -- but its *Test suffix places it in the unit-test surefire
> execution (forkCount=4, 1GB heap). Its heavyweight siblings in the same
> package, KafkaSourceITCase and KafkaSourceMigrationITCase, are already named
> *ITCase and run in the integration-test execution (forkCount=2, 2GB heap,
> reuseForks=false).
> h3. Proposed change
> Rename SourceTopicIntegrityTest -> SourceTopicIntegrityITCase so it is picked
> up by the integration-test phase, consistent with those siblings. This
> reduces the number of concurrent Testcontainers-Kafka + MiniCluster forks and
> gives the test more heap and a fresh JVM per class. This is a
> test-classification cleanup justified by the precedence of KafkaSourceITCase
> / KafkaSourceMigrationITCase.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
