Thanks for your reply Jagadish. We will certainly do some testing, I was just 
curious if anyone had tried or knew what we could expect. The broker format 
change is actually over 2 years old now, so I assumed someone had tried this by 
now ;)

-Tommy

On Mon, 2018-07-09 at 12:22 -0700, Jagadish Venkatraman wrote:

Hi Thomas,


Has Samza been tested against newer broker

versions using the new message format, and if so does it have a

significant performance impact?


We have not benchmarked Kafka broker performance with the new message

format.

Any benchmarking may not be reliably reproducible since there are many

variables (message sizes,

compressibility, how "saturated" the brokers are at the instant you run the

benchmark).


I'd suggest some general pointers on this.


   - First define the metric you're trying to optimize - broker-side

   throughput, Samza's throughput, broker-cpu utilization?

   - Specify what the acceptable value of the metric is for your current

   setup.

   - Then, measure it for your workload. For all you know, the performance

   might be "good enough"


Are their plans to move Samza to the

new consumer?


We certainly have plans to move to the "new" consumer for reasons unrelated

to throughput on the client-side

(eg: SSL, long-term support from the Kafka community). However, these plans

have not gotten enough traction.


Please let me know if there are further questions.


-- Jagadish


On Mon, Jul 9, 2018 at 9:47 AM, Thomas Becker 
<thomas.bec...@tivo.com<mailto:thomas.bec...@tivo.com>>

wrote:


Anyone have any input here?


On Mon, 2018-07-02 at 11:50 +0000, Thomas Becker wrote:


Hey folks,


I have a question regarding potential performance impacts of running


Samza against newer Kafka brokers. We have languished on the old on-


disk message  format for Kafka for some time, and want to upgrade to


the newer format which supports timestamps. Samza currently accounts


for quite a bit of our message consumption and I am concerned that it


will cause a broker performance hit due to downconversion of messages.


I know Samza uses the old SimpleConsumer internally which does not


support the newer format. Has Samza been tested against newer broker


versions using the new message format, and if so does it have a


significant performance impact? Are their plans to move Samza to the


new consumer?



Regards,


Tommy Becker



________________________________



This email and any attachments may contain confidential and privileged

material for the sole use of the intended recipient. Any review, copying,

or distribution of this email (or any attachments) by others is prohibited.

If you are not the intended recipient, please contact the sender

immediately and permanently delete this email and any attachments. No

employee or agent of TiVo Inc. is authorized to conclude any binding

agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo

Inc. may only be made by a signed written agreement.



________________________________


This email and any attachments may contain confidential and privileged

material for the sole use of the intended recipient. Any review, copying,

or distribution of this email (or any attachments) by others is prohibited.

If you are not the intended recipient, please contact the sender

immediately and permanently delete this email and any attachments. No

employee or agent of TiVo Inc. is authorized to conclude any binding

agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo

Inc. may only be made by a signed written agreement.






________________________________

This email and any attachments may contain confidential and privileged material 
for the sole use of the intended recipient. Any review, copying, or 
distribution of this email (or any attachments) by others is prohibited. If you 
are not the intended recipient, please contact the sender immediately and 
permanently delete this email and any attachments. No employee or agent of TiVo 
Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by 
email. Binding agreements with TiVo Inc. may only be made by a signed written 
agreement.

Reply via email to