> On Feb. 23, 2014, 10:38 p.m., Jun Rao wrote: > > clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java, > > lines 101-112 > > <https://reviews.apache.org/r/18299/diff/3/?file=500841#file500841line101> > > > > This is actually not the right place to add the offset and the crc for > > the shallow iterator. This method is probably intended for writing the > > produce request to the socket without first serializing the request into a > > separate buffer. However, we don't do that now and this method is not > > currently being used. On the producer side, when a Records is ready to be > > sent, we get its underlying ByteBuffer and use it in the ProducerRequest, > > which then serializes the bytes into a send ByteBuffer. > > > > So, we need to add the offset/crc for the shallow message when a > > Records is ready to be sent. Potentially, we can add a close() method in > > Record and call it in RecordAccumlator.ready() when a Records is determined > > to be ready. The offset/crc for the shallow message can be added in close().
Yep, thanks Jun. > On Feb. 23, 2014, 10:38 p.m., Jun Rao wrote: > > clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java, > > lines 149-160 > > <https://reviews.apache.org/r/18299/diff/3/?file=500841#file500841line149> > > > > Instead of copying the data to a new ByteBuffer, we can let Record > > reuse the portion of the original ByteBuffer passed into > > CompressedRecordsIterator. Not sure how it can be done: the uncompressed message size is most likely larger than the compressed version, so the recordBuffer's size would most likely be larger than the size we just read out the compressed data. > On Feb. 23, 2014, 10:38 p.m., Jun Rao wrote: > > clients/src/main/java/org/apache/kafka/common/record/Record.java, lines > > 87-101 > > <https://reviews.apache.org/r/18299/diff/3/?file=500843#file500843line87> > > > > This doesn't handle compressed codec. Yes and that is because this function will only called by the MessageRecords. - Guozhang ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18299/#review35245 ----------------------------------------------------------- On Feb. 22, 2014, 1:57 a.m., Guozhang Wang wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/18299/ > ----------------------------------------------------------- > > (Updated Feb. 22, 2014, 1:57 a.m.) > > > Review request for kafka. > > > Bugs: KAFKA-1253 > https://issues.apache.org/jira/browse/KAFKA-1253 > > > Repository: kafka > > > Description > ------- > > In-place compression with Crc32 pre-computation. > > In-place decompression embedded in nested iterator. > > > Diffs > ----- > > > clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java > PRE-CREATION > > clients/src/main/java/org/apache/kafka/common/record/ByteBufferOutputStream.java > PRE-CREATION > > clients/src/main/java/org/apache/kafka/common/record/CompressedMemoryRecords.java > PRE-CREATION > clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java > 9d8935fa3beeb2a78b109a41ed76fd4374239560 > clients/src/main/java/org/apache/kafka/common/record/Record.java > f1dc9778502cbdfe982254fb6e25947842622239 > clients/src/main/java/org/apache/kafka/common/utils/Utils.java > 9c34e7dc82f33df7406cad0e64eb6a896d068dc6 > clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java > b0745b528cef929c4273f7e2ac4de1476cfc25ad > clients/src/test/java/org/apache/kafka/common/record/RecordTest.java > ae54d67da9907b0a043180c7395a1370b3d0528d > clients/src/test/java/org/apache/kafka/common/utils/CrcTest.java > PRE-CREATION > clients/src/test/java/org/apache/kafka/test/TestUtils.java > 36cfc0fda742eb501af2c2c0330e3f461cf1f40c > > Diff: https://reviews.apache.org/r/18299/diff/ > > > Testing > ------- > > > Thanks, > > Guozhang Wang > >
