Hi Farid,
Try using mmap() (disclaimer: haven't tried compiling this):
struct stat stats;
KJ_SYSCALL(fstat(fd, &stats));
size_t size = stats.st_size;
const void* data = mmap(nullptr, size, PROT_READ, MAP_PRIVATE, fd, 0);
if (data == MAP_FAILED) {
KJ_FAIL_SYSCALL("mmap");
}
KJ_DEFER(KJ_SYSCALL(munmap(data, size)) { break; });
KJ_SYSCALL(madvise(data, size, MADV_SEQUENTIAL));
kj::ArrayPtr<capnp::word> words(
reinterpret_cast<const capnp::word*>(data),
size / sizeof(capnp::word));
while (words.size() > 0) {
capnp::FlatArrayMessageReader message(words);
Message::Reader chunk = message.getRoot<Message>();
count++;
words = kj::arrayPtr(message.getEnd(), words.end());
}
(On Windows you'll need to use CreateFileMapping() and MapViewOfFile() or
whatever.)
-Kenton
On Thu, Jul 20, 2017 at 2:11 PM, Farid Zakaria <[email protected]>
wrote:
> Looking for some guidance on possibly the fastest way to read multiple
> messages in a file that has multiple MessageRoots
>
> So far I have this written:
>
> capnp::MallocMessageBuilder messageBuilder;
> //What is a good size for our words? As long as its smaller?
> capnp::word scratch[1024];
> kj::ArrayPtr<capnp::word> scratchSpace(scratch);
> kj::FdInputStream stream(fd);
> kj::BufferedInputStreamWrapper buff(stream);
>
> unsigned long count = 0;
> while (buff.tryGetReadBuffer().size() != 0) {
> capnp::InputStreamMessageReader message(buff, capnp::ReaderOptions(),
> scratchSpace);
> Message::Reader chunk = message.getRoot<Message>();
> count++;
> }
>
>
> I tried the helper methods first, but they seem too slow I think without the
> BufferedInputStreamWrapper.
>
>
> I don't do much C++ so I appreciate any help :)
>
> Btw, Is there an IRC chat or something ?
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> Visit this group at https://groups.google.com/group/capnproto.
>
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
Visit this group at https://groups.google.com/group/capnproto.
