Hi Kenton, thanks for the quick response. I was working now on this solution, the limit per blob is not too much of a problem for me. Thanks to Cap'n proto I think I'll have an elegant solution. Thanks.
El lunes, 31 de octubre de 2022 a las 20:32:30 UTC+1, [email protected] escribió: > For a single Text or Data blob there is a hard limit of 512MB. You can, > however, construct a message which contains multiple blobs, e.g. use > `List(Text)`. Such a message can be up to 2^64 bytes. > > If I were redesigning the encoding from scratch I'd probably allow for > bigger individual blobs but there's no way to introduce them now without > breaking compatibility, unfortunately. > > Protobuf theoretically supports 2GB messages but because the messages have > to be parsed upfront in O(n) time, you won't have good results with > messages of that size. Cap'n Proto, on the other hand, quite comfortably > supports multi-GB messages since you can mmap() the message and randomly > access it in O(1) time. > > -Kenton > > On Mon, Oct 31, 2022 at 2:21 PM Josemi <[email protected]> wrote: > >> Hello. >> >> I need to work with an structured data that have atributes with undefined >> lenght, some of them could have GB. >> >> I have been using Protocol Buffers for it until I see that exists a hard >> limit of 2GB per message. Then I see this Stack Overflow solution >> <https://stackoverflow.com/questions/34128872/google-protobuf-maximum-size>. >> It says that Cap'n Proto "can support messages up to 2^64 bytes (2^32 >> segments of 4GB each)" >> >> I reprogram the code and now it raise the error >> *capnp.lib.capnp.KjException: >> capnp/layout.c++:1694: failed: text blob too big * trying to set a 0.8 >> GB buffer to a Data type. >> >> On Cap'n Proto the atribute seems like that: >> *file @1 :Data; # ptr[1],* >> >> And the code, in Python, is something like: >> file = open('data_file', 'rb').read() >> >> >> Whats wrong with that. >> Cap'n Proto won't solve my problem?. >> >> Thanks. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Cap'n Proto" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/capnproto/8a23280b-b43b-483a-94db-7fd94bba93een%40googlegroups.com >> >> <https://groups.google.com/d/msgid/capnproto/8a23280b-b43b-483a-94db-7fd94bba93een%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/84bd40fa-849f-45e1-8e3d-8256fe563605n%40googlegroups.com.
