Hi Farid, Does the problem happen around 2^31 bytes in? My guess is that the library is using an `int` somewhere where it should be using a `long`. Or, perhaps it's trying to map more than 2^31 bytes in a single ByteBuffer, which won't work since ByteBuffer seems to use `int`s for indexes.
David, any thoughts? -Kenton On Fri, Jul 7, 2017 at 11:55 AM, Farid Zakaria <[email protected]> wrote: > Hi everyone, > > I'm looking for some guidance on what I may be doing wrong. > I'm serializing (unpacked) multiple MessageBuilder to FileChannel via a > BufferedOutputStreamWrapper > > Here is a snippet of the code in Kotlin > > val partitionFileInputStream = FileOutputStream(filename, true).channel > val buffered = BufferedOutputStreamWrapper(partitionFileInputStream) > > val recordIterator = RecordIterator(dataSource) > recordIterator.asSequence().map { row -> > converter(row) > }.forEach { message -> > Serialize.write(buffered, message) > } > > buffered.flush() > buffered.close() > > > > I'm writing millions of records to a file which is several GB in size at the > end. > > I then try to read the file: > > > val fileChannel = RandomAccessFile(filePath.toFile(), "r").getChannel() > > for(message in SerializedIterator(fileChannel)) { > val record = message.getRoot(SomeClass.Object.factory) > } > > > here is the iterator implementation: > > > class SerializedIterator(readChan: ReadableByteChannel) : > AbstractIterator<MessageReader>(), AutoCloseable { > > val buffer = BufferedInputStreamWrapper(readChan) > > override fun close() { > buffer.close() > } > > override fun computeNext() { > try { > setNext(Serialize.read(buffer)) > } catch (e : Error) { > close() > done() > } > } > > } > > > > It seems to go fine for several million records and then I get hit with: > > > java.lang.NegativeArraySizeException: null > at org.capnproto.Serialize.read(Serialize.java:91) > ~[runtime-0.1.1.jar:0.1.1] > at org.capnproto.Serialize.read(Serialize.java:51) > ~[runtime-0.1.1.jar:0.1.1] > at SerializedIterator.computeNext(SerializedIterator.kt:18) > > > > > > -- > You received this message because you are subscribed to the Google Groups > "Cap'n Proto" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > Visit this group at https://groups.google.com/group/capnproto. > -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. Visit this group at https://groups.google.com/group/capnproto.
