lucliu1108 commented on PR #22452:
URL: https://github.com/apache/kafka/pull/22452#issuecomment-4792346328
Hi @alanlau28 , thanks for the PR!
Based on Matthias' suggestion, another approach is to follow the pattern
adapted by `process-latency` metrics. In `StreamThread`, after
`taskManager.process()` returns:
```
final long processLatency = advanceNowAndComputeLatency();
totalProcessLatency += processLatency;
if (processed > 0) {
processRateSensor.record(processed, now);
processLatencySensor.record(processLatency / (double) processed, now);
...
}
```
`advanceNowAndComputeLatency()` reads the wall clock once and reuses it for
the whole batch's metrics — one syscall covers all records processed in that
taskManager.process().
For e2e latency, we could apply a similar per-task pattern. The code now
reads the wallclock inside `maybeRecordE2ELatency` the first time a record hits
a terminal node, then caches it for the rest of that input record's fan-out.
The cost is one read per input record.
We could push the read up a level to the whole `taskManager.process()`
batch. To be specific, `maybeRecordE2ELatency` stops reading the clock and just
appends the record's timestamp to a per-task buffer. Then at the end of the
while loop in `TaskExecutor.processTask` (the per-task numIterations) : read
the wallClock once and traverse through the buffer, recording one sensor sample
per buffered timestamp using that single `now`. Then clear the buffer.
In this way, the cost goes from one syscall per input record to one per task
call. **However, the tradeoff is it is not recording the true latency, but
rather "the latency that a record would have if it finished at the end of its
batch", which is always ≥ the real value. It's a conservative upper bound.
Also, MIN is biased by T_batch_end − max(record.ts in batch); in workloads with
uniform real latency, the new proposal artificially widens the MIN/MAX range**
---
Alternatively, another approach is to read wallClock time every K records
during processing. Records within the same K-window share the same cached now.
Records in different K-windows get different `now`. However, how to choose this
`K` for batching is another problem worth considering.
Happy to hear thoughts from others on this trade-off (and better options).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
