Yeah, I remember hitting that bug in the Go implementation. I feel like the security implications docs should also mention limits on number of segments, since I can't imagine a case where you would use the full 2^32 space.
On Mon, Jul 10, 2017, 6:15 PM David Renshaw <[email protected]> wrote: > It looks like there's a bug where `Serialize.read()` will throw that > exception if it reads a segment count greater than 2**31. It is supposed to > throw this exception > <https://github.com/capnproto/capnproto-java/blob/633b1f619939dae9089a45f208523d58d159a3f5/runtime/src/main/java/org/capnproto/Serialize.java#L67-L69> > in that case, but that can fail when there is an overflow. > > The segment count should not ever be that large, so there's probably also > a bug somewhere on the writing side. > > On Mon, Jul 10, 2017 at 12:40 PM, Kenton Varda <[email protected]> > wrote: > >> Hi Farid, >> >> Does the problem happen around 2^31 bytes in? My guess is that the >> library is using an `int` somewhere where it should be using a `long`. Or, >> perhaps it's trying to map more than 2^31 bytes in a single ByteBuffer, >> which won't work since ByteBuffer seems to use `int`s for indexes. >> >> David, any thoughts? >> >> -Kenton >> >> On Fri, Jul 7, 2017 at 11:55 AM, Farid Zakaria <[email protected] >> > wrote: >> >>> Hi everyone, >>> >>> I'm looking for some guidance on what I may be doing wrong. >>> I'm serializing (unpacked) multiple MessageBuilder to FileChannel via a >>> BufferedOutputStreamWrapper >>> >>> Here is a snippet of the code in Kotlin >>> >>> val partitionFileInputStream = FileOutputStream(filename, true).channel >>> val buffered = BufferedOutputStreamWrapper(partitionFileInputStream) >>> >>> val recordIterator = RecordIterator(dataSource) >>> recordIterator.asSequence().map { row -> >>> converter(row) >>> }.forEach { message -> >>> Serialize.write(buffered, message) >>> } >>> >>> buffered.flush() >>> buffered.close() >>> >>> >>> >>> I'm writing millions of records to a file which is several GB in size at >>> the end. >>> >>> I then try to read the file: >>> >>> >>> val fileChannel = RandomAccessFile(filePath.toFile(), "r").getChannel() >>> >>> for(message in SerializedIterator(fileChannel)) { >>> val record = message.getRoot(SomeClass.Object.factory) >>> } >>> >>> >>> here is the iterator implementation: >>> >>> >>> class SerializedIterator(readChan: ReadableByteChannel) : >>> AbstractIterator<MessageReader>(), AutoCloseable { >>> >>> val buffer = BufferedInputStreamWrapper(readChan) >>> >>> override fun close() { >>> buffer.close() >>> } >>> >>> override fun computeNext() { >>> try { >>> setNext(Serialize.read(buffer)) >>> } catch (e : Error) { >>> close() >>> done() >>> } >>> } >>> >>> } >>> >>> >>> >>> It seems to go fine for several million records and then I get hit with: >>> >>> >>> java.lang.NegativeArraySizeException: null >>> at org.capnproto.Serialize.read(Serialize.java:91) >>> ~[runtime-0.1.1.jar:0.1.1] >>> at org.capnproto.Serialize.read(Serialize.java:51) >>> ~[runtime-0.1.1.jar:0.1.1] >>> at SerializedIterator.computeNext(SerializedIterator.kt:18) >>> >>> >>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Cap'n Proto" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> Visit this group at https://groups.google.com/group/capnproto. >>> >> >> > -- > You received this message because you are subscribed to the Google Groups > "Cap'n Proto" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > Visit this group at https://groups.google.com/group/capnproto. > -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. Visit this group at https://groups.google.com/group/capnproto.
