Amusingly, I was just looking at this for a protocol. Kenton's suggestion
to use List[Data] works, but it carries an unintended buffering problem. A
message can have multiple segments, and a correct capn-proto implementation
will extend its segment pool as needed to hold a large byte sequence, but
if I read the encoding spec correctly, it cannot *send* any of the segments
until *all* of the segments are available for framing. Which means that
your big blob of data sits in client memory while you are loading it, and
stays there until the message is fully transmitted . Segment release could
be optimized by releasing segments as transmission proceeds, but that isn't
required by the capn-proto specification, and it doesn't resolve the "load
big blob into memory" problem.
One alternative is to introduce a builder pattern, something like this:
interface FileBuilder {
write @0 (d : Data) -> Int32; # Returns number of bytes written
close @1 () -> (file: File)
}
interface MyService {
createFile @0 () -> (builder; FileBuilder)
myInterestingCall @1 (... file: File, ...) -> (val: InterestingResult)
}
The advantage to this, mainly, is that the byte transmission is divided
into a *sequence* of messages. On the service side, these can be stashed
aside until the Close() call is made, at which point the file object is
fabricated on the service and a File capability is returned to the client
that can be included in other messages. Because of promise pipelining, this
doesn't take as many round trips as you might think.
The main puzzle here, in my mind, is that the server needs to know when
client-side capabilities. I don't remember seeing anything like a
capability release protocol that advises the serving entity when the state
associated with an ephemeral object can be released.
On Thursday, November 3, 2022 at 10:05:56 AM UTC-7 josem...@gmail.com wrote:
> Sorry I writed it in Spanish and see it now.
>
> Translation:
>
> The problem is that the server receives all the data in a transmission
> buffer, piece by piece without knowing the structured object. So I'm
> storing it in a file because the server doesn't need to open it (it only
> needs to read small data). If I want to put all the attributes in the SQL
> tables, I will have to take all the fragments and build the structured
> object in memory, and this, as I understand it, is a problem.
> I don't know if my ideas are correct or am I missing something.
>
>
> El miércoles, 2 de noviembre de 2022 a las 18:22:04 UTC+1, Jose mi
> escribió:
>
>> El problema es que el servidor recibe todos los datos en un búfer de
>> transmisión, pieza por pieza sin conocer el objeto estructurado. Así que lo
>> estoy almacenando en un archivo porque el servidor no necesita abrirlo
>> (solo necesita leer datos pequeños). Si quiero poner todos los atributos en
>> las tablas SQL, tendré que tomar todos los fragmentos y construir el objeto
>> estructurado en la memoria, y esto, según tengo entendido, es un problema.
>> No sé si mis ideas son correctas o me estoy perdiendo algo.
>>
>> -- Josemi.
>>
>> El martes, 1 de noviembre de 2022 a las 3:45:12 UTC+1, jens....@gmail.com
>> escribió:
>>
>>>
>>> > On Oct 31, 2022, at 12:21 PM, Josemi <josem...@gmail.com> wrote:
>>> >
>>> > Hello.
>>> >
>>> > I need to work with a structured data that have atributes with
>>> undefined lenght, some of them could have GB.
>>>
>>> Most structured storage is optimized for smaller data. And huge values
>>> in-line push all the records far apart, which is bad for cache performance.
>>>
>>> SQLite supports arbitrary size blobs up to 2^64 bytes. (Even with that
>>> it’s best to put the blobs in a separate table and join your records to
>>> it.)
>>>
>>> —Jens
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/capnproto/7c56f2fc-8e73-4f61-91ce-c07917ed670cn%40googlegroups.com.
