scott-hendricks commented on pull request #9736: URL: https://github.com/apache/kafka/pull/9736#issuecomment-744538298
> Can you say a bit more about the decision to put the timestamp into the message, rather than using the timestamp which Kafka itself puts on every message? I want to get true End to End message latency, from when the Producer is told to send the message until when the Consumer receives the message to process it. I wanted this timestamp to live outside the Kafka broker/client, as that is what customers will see. We can also validate the numbers within Kafka on this if we want, I just wanted to make sure we captured every bit of latency available. We're running this test on systems that are millisecond synced to get an accurate reading. This actually came about when discussing https://stackoverflow.com/questions/20520492/how-to-minimize-the-latency-involved-in-kafka-messaging-framework/20585525#20585525 for a completely unrelated reason. That does not paint Kafka in a good light, so I took it upon myself to try and get some real world latency numbers. > Is there a reason to have a timestamp size other than 8 bytes? I wanted to future proof this code against any potential changes in JVM or other architectures. I can remove it if you want. >ConfigurableProducerWorker looks a lot like ProduceBenchWorker. Maybe it's OK to have two classes here, but I'd like to understand why we should. ProduceBenchWorker is also intended to be configurable, so I'm not sure why as a user I'd use one or the other. Can we merge these? That's because it is based on the `ProduceBenchWorker`, however enough of the class was changing that I wanted to keep backward compatible tests working. The `Throttle` architecture didn't exactly work for the variable throughput generation. In order to add partition-specific testing I had to limit the test to 1 topic. I am by no means an experienced Java engineer and I couldn't find a way of modifying the original, or even subclassing it, without forcing the spec to change. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
