Purushottam Sinha created FLINK-40445:
-----------------------------------------
Summary: Autoscaler: integration test for Kafka/Pulsar source
partition-metric detection
Key: FLINK-40445
URL: https://issues.apache.org/jira/browse/FLINK-40445
Project: Flink
Issue Type: Bug
Components: Autoscaler, Kubernetes Operator
Reporter: Purushottam Sinha
Problem
ScalingMetricCollector derives a source's partition count by parsing
Kafka/Pulsar metric names (PartitionMetricNameParser, added in FLINK-40443 / PR
#1187). This path has no automated coverage against a real broker, so a
regression in partition detection could go unnoticed until it affects source
autoscaling.
Evidence
- flink-autoscaler/.../ScalingMetricCollector.java →
updateKafkaPulsarSourceNumPartitions uses PartitionMetricNameParser; only
unit/differential tests exist today.
- FLINK-40443 was verified manually against Apache Kafka 3.9.1 and Pulsar 3.3.1
on minikube, not in CI.
- Existing precedent: flink-autoscaler-plugin-jdbc uses Testcontainers for the
JDBC state store.
Proposed fix
- Add a Testcontainers-based integration test (separate PR; not blocking
1.16.0) running a source job against real Kafka and Pulsar.
- Assert derived numSourcePartitions for: multi-topic summing, dotted/hyphen
topic names, source parallelism > 1, and multiple source vertices in one job.
Acceptance
- Test asserts detected partition count equals broker ground truth for each
dimension above.
- Runs in CI.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
