Hi Thomas, >> Has Samza been tested against newer broker versions using the new message format, and if so does it have a significant performance impact?
We have not benchmarked Kafka broker performance with the new message format. Any benchmarking may not be reliably reproducible since there are many variables (message sizes, compressibility, how "saturated" the brokers are at the instant you run the benchmark). I'd suggest some general pointers on this. - First define the metric you're trying to optimize - broker-side throughput, Samza's throughput, broker-cpu utilization? - Specify what the acceptable value of the metric is for your current setup. - Then, measure it for your workload. For all you know, the performance might be "good enough" >> Are their plans to move Samza to the new consumer? We certainly have plans to move to the "new" consumer for reasons unrelated to throughput on the client-side (eg: SSL, long-term support from the Kafka community). However, these plans have not gotten enough traction. Please let me know if there are further questions. -- Jagadish On Mon, Jul 9, 2018 at 9:47 AM, Thomas Becker <thomas.bec...@tivo.com> wrote: > Anyone have any input here? > > On Mon, 2018-07-02 at 11:50 +0000, Thomas Becker wrote: > > Hey folks, > > I have a question regarding potential performance impacts of running > > Samza against newer Kafka brokers. We have languished on the old on- > > disk message format for Kafka for some time, and want to upgrade to > > the newer format which supports timestamps. Samza currently accounts > > for quite a bit of our message consumption and I am concerned that it > > will cause a broker performance hit due to downconversion of messages. > > I know Samza uses the old SimpleConsumer internally which does not > > support the newer format. Has Samza been tested against newer broker > > versions using the new message format, and if so does it have a > > significant performance impact? Are their plans to move Samza to the > > new consumer? > > > Regards, > > Tommy Becker > > > ________________________________ > > > This email and any attachments may contain confidential and privileged > material for the sole use of the intended recipient. Any review, copying, > or distribution of this email (or any attachments) by others is prohibited. > If you are not the intended recipient, please contact the sender > immediately and permanently delete this email and any attachments. No > employee or agent of TiVo Inc. is authorized to conclude any binding > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo > Inc. may only be made by a signed written agreement. > > > ________________________________ > > This email and any attachments may contain confidential and privileged > material for the sole use of the intended recipient. Any review, copying, > or distribution of this email (or any attachments) by others is prohibited. > If you are not the intended recipient, please contact the sender > immediately and permanently delete this email and any attachments. No > employee or agent of TiVo Inc. is authorized to conclude any binding > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo > Inc. may only be made by a signed written agreement. > -- Jagadish V, Graduate Student, Department of Computer Science, Stanford University
