Hi Rohit, librdkafka will produce messages with zero copy (unless compression is enabled). And kafkacat will do this if you provide it one or more files as arguments in producer mode: kafkacat -b <broker> -t <topic> -P <your_file> <another_file> <the_thirdest_of_files> (each file is sent as one message)
Just make sure to keep message.max.bytes synchronized across all clients and brokers. Magnus 2014-11-14 2:34 GMT+01:00 Jun Rao <jun...@gmail.com>: > Both the Kafka client and broker need to allocate memory for the whole > message. So, the larger the message, the more memory fragmentation it may > cause, which can lead to GC/OOME issues. > > Thanks, > > Jun > > On Wed, Nov 12, 2014 at 9:23 PM, Rohit Pujari <rpuj...@hortonworks.com> > wrote: > > > I'm thinking of using Kafka for transporting binary files (tiff, jpeg, > > pdf). These files are anywhere between 10 KB to 5MB. Thought behind > > considering Kafka is - It serves as a staging area for the files and > > facilitates asynchronous ingestion in near-real-time. > > > > Any thoughts on using Kafka for binary payloads? gotchas, watch outs? > > > > Thanks, > > Rohit Pujari > > Solutions Architect, Hortonworks > > rpuj...@hortonworks.com > > > > -- > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity > to > > which it is addressed and may contain information that is confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified > that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender > immediately > > and delete it from your system. Thank You. > > >
