Re: Compression in kafka 0.8

2013-06-05 Thread Jun Rao
There is no code change needed in the consumer. The message format stores the compression codec and the consumer library knows how to parse the data. Thanks, Jun On Wed, Jun 5, 2013 at 12:13 AM, Jason Rosenberg wrote: > In this case, does the consumer code need to change, account for the > co

Re: Compression in kafka 0.8

2013-06-05 Thread Jason Rosenberg
In this case, does the consumer code need to change, account for the compression (or is this handled automatically in the consumer apis?). If we decide to start compressing messages to a topic, can a consumer seamlessly move between the transition from uncompressed to compressed? Jason On Tue,

Re: Compression in kafka 0.8

2013-06-04 Thread Vadim Keylis
Thanks so much Neha On Tue, Jun 4, 2013 at 9:57 PM, Neha Narkhede wrote: > The only way of storing compressed data on the server is to have the > producer send it compressed. The server sends the data in compressed format > to the consumers. > > Thanks, > Neha > > > On Tue, Jun 4, 2013 at 9:55 P

Re: Compression in kafka 0.8

2013-06-04 Thread Neha Narkhede
The only way of storing compressed data on the server is to have the producer send it compressed. The server sends the data in compressed format to the consumers. Thanks, Neha On Tue, Jun 4, 2013 at 9:55 PM, Vadim Keylis wrote: > Good evening. I would like to preserve logs in kafka for 2 days.

Compression in kafka 0.8

2013-06-04 Thread Vadim Keylis
Good evening. I would like to preserve logs in kafka for 2 days. Due to the size of data it must be compressed otherwise it will take enormous amount of disk space. Its my understanding that the only way to store messages in kafka compressed is to enable compression on the producer side, is that co