Ning Zhang created KAFKA-9360:
---------------------------------
Summary: emitting checkpoint and heartbeat set to false will not
disable the activity in their SourceTask
Key: KAFKA-9360
URL: https://issues.apache.org/jira/browse/KAFKA-9360
Project: Kafka
Issue Type: Improvement
Components: mirrormaker
Affects Versions: 2.4.0
Reporter: Ning Zhang
Fix For: 2.5.0
Attachments: Screen Shot 2020-01-02 at 2.55.38 PM.png
`emit.heartbeats.enabled` and `emit.checkpoints.enabled` are supposed to be the
knobs to control if the heartbeat message or checkpoint message will be sent or
not to the topics respectively. In our experiments, setting them to false will
not suspend the activity in their SourceTasks, e.g. MirrorHeartbeatTask,
MirrorCheckpointTask.
The observations are, when setting those knobs to false, huge volume of
`SourceRecord` are being sent without interval, causing significantly high CPU
usage of MirrorMaker 2 instance and congesting the single partition of the
heartbeat topic and checkpoint topic.
The proposed fix in the following PR is to (1) explicitly check if `interval`
is set to negative (e.g. -1), when the `emit.heartbeats.enabled` or
`emit.checkpoints.enabled` is off. (2) if `interval` is indeed set to negative,
put the thread in sleep mode (e.g. 5 seconds) and return null, to prevent it
from running the remaining logic.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
