ai-yang opened a new issue, #3104:
URL: https://github.com/apache/rocketmq-dashboard/issues/3104
# [Studio][Bug] Cross-metric samples keep missing native alerts permanently
active
## Severity
Medium-High. The defect makes the native alert state and
recovery-notification path untrustworthy, although it does not directly
interrupt RocketMQ message traffic.
## Problem
Native alert reconciliation treats a sample from any metric as proof that
every matching rule fingerprint is still present. It checks the collection
scope, instance, and resource labels, but does not require the sample metric to
equal the rule metric.
Because a fingerprint is derived from `ruleId + instanceId + labels`,
another metric with the same labels can keep a missing metric's old state in
`FIRING` or `ACKED` forever.
## Reproduction
1. Create a native business rule for `consumer.delay.seconds > 10`, scoped
to instance `local` and consumer group `orders`; configure a notification
channel if recovery delivery is to be observed.
2. Produce a delay sample of `120` and let the rule enter `FIRING`.
3. In the next successful full collection, keep the consumer group but make
its consumption timestamp unavailable.
4. The business collector omits `consumer.delay.seconds` but still emits
`consumer.lag.total` with `{consumerGroup=orders}`.
5. Process that collection.
### Actual behavior
The lag sample is added to the delay rule's `presentKeys`. The previous
delay state remains active; no `RESOLVED` event or recovery notification is
emitted.
### Expected behavior
Only a sample whose metric equals the rule metric may keep that rule's
fingerprint present. A missing delay metric in a successful scope must resolve
its prior active state even when another same-label metric remains.
## Impact
- The UI continuously reports an incident that no longer has a metric sample.
- Acknowledged alerts also remain active indefinitely.
- Operators never receive the recovery notification.
- Repeated false incidents reduce confidence in native alerting.
## Root cause
`NativeAlertProcessor.reconcileMissingActiveStates` builds `presentKeys` by
applying `NativeAlertRuleScopeMatcher`, which intentionally matches instance
and resource labels only. It omitted `sample.metricKey() == rule.metric`.
The real Apache business collector makes this stable: `consumer.lag.total`
is always emitted for a collected group, while `consumer.delay.seconds` is
conditional on a consumption timestamp being available; both use the same
consumer-group label.
## Regression boundary / duplicate check
- #2679 and merged replacement PR #2957 reconcile alerts when a monitored
resource disappears and no sample remains for its fingerprint.
- This case is distinct: the resource and labels still exist, but one metric
disappears and another metric impersonates its presence.
- #2951/#2960 concern blank metric-name normalization, not cross-metric
state reconciliation.
- #2823 scopes persisted metric snapshot history by cluster; it does not
reconcile missing samples or construct `presentKeys`.
- Searches for `reconcileMissingActiveStates`, `presentKeys`, `another
metric`, and `consumer.delay.seconds resolved` found no matching issue or PR.
## Suggested acceptance criteria
- Require exact metric equality while constructing reconciliation
`presentKeys`.
- Preserve current behavior for a genuinely present target metric.
- Preserve the full-scope-unavailable guard against false recovery.
- Add a regression test that verifies `RESOLVED` state persistence,
lifecycle event creation, original labels, and invocation of the
recovery-notification enqueue path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
