Hi Maxime, Do you mean KafkaMigrationTool or MirrorMaker ? Are you trying to mirror data between a Kafka 0.7 and Kafka 0.8 cluster ? We have fixed the KafkaMigrationTool to always turn off shallow iteration since we have to decompress data in 0.7 format before sending it to a 0.8 broker.
Thanks, Neha On Wed, Jan 23, 2013 at 6:39 AM, Maxime Brugidou <maxime.brugi...@gmail.com>wrote: > Hi all, > > I am working with MirrorMaker on the 0.8 branch (so I haven't tested this > on 0.7 but it looks similar). > > When I enable shallow.iterator.enable=true on the consumer config, (and > disable compression on the producer config, as mentioned in the wiki) the > throughput is really good but the data produced is unreadable since the > messages are taken "as is" by the producer and i just get a bunch of bytes > stored with the ByteBufferMessageSet header and all the info. > > I tested using the kafka-console-consumer to read my mirrored data and it's > unreadable (unless I pipe the data to gunzip, which manages to read it). > > Did I miss something? Is there anything i can activate on the producer side > of MirrorMaker to send the shallow message "as is"? Did anyone manage to > make MirrorMaker work with shallow.iterator.enable=true? > > Thanks for your help, > Maxime >
