Hey all. We are trying to create a controlled lag (a lag of 30 secs). We are doing that by inspecting a timestamp field for each msg on a specific topic and doing an actual Thread.sleep for a computed time in the consumer thread (until the msg is 30 secs old). We were hoping to see that eventually most of the messages are 30 seconds old and the created lag stands for 30 secs of processing rate. This works for us in low throughput but on high rates we are starting to witness strange behavior: lag on partitions is starting to grow in an uncontrollable manner and becomes uneven per partition. We have tried multiple variations of this: sleeping per bulk of msgs, sleeping per msg, capping the sleep time etc but failed to resolve this.
Has anyone tried to do this? any ideas? Thanks, Ido -- Thanks, Ido
