Ok that makes sense, thanks ! The next question I have regarding performance is about the way Kafka writes in the data files. I often hear Kafka is very performant because it writes in an append-only fashion. So even with hard disk (not SSD) we get a great performance because it writes in sequence.
I could understand that if Kafka was only writing to one file. But in reality it s writing to N files, N being the number of partitions hosted by the broker. So even though it appends the data to each file, overall I assume it is not writing in sequence on the disk. Am I wrong ? -----Original Message----- From: Tauzell, Dave [mailto:dave.tauz...@surescripts.com] Sent: 08 March 2017 22:09 To: users@kafka.apache.org Subject: RE: Performance and Encryption I think because the product batches messages which could be for different topics. -Dave -----Original Message----- From: Nicolas MOTTE [mailto:nicolas.mo...@amadeus.com] Sent: Wednesday, March 8, 2017 2:41 PM To: users@kafka.apache.org Subject: Performance and Encryption Hi everyone, I understand one of the reasons why Kafka is performant is by using zero-copy. I often hear that when encryption is enabled, then Kafka has to copy the data in user space to decode the message, so it has a big impact on performance. If it is true, I don t get why the message has to be decoded by Kafka. I would assume that whether the message is encrypted or not, Kafka simply receives it, appends it to the file, and when a consumer wants to read it, it simply reads at the right offset... Also I m wondering if it s the case if we don t use keys (pure queuing system with key=null). Cheers Nico This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.
