Aleksey, I fully support the idea of a common tuple format to avoid conversions when passing data around (servers, clients). The IEP looks good to me.
> Integer Field Compression Agree with this, we can even extend the idea to float/double fields. > size of table elements might be 1, 2, 4, or 8 bytes depending on the header flags Right now we are limited by the max size of byte[] in Java (Integer.MAX_VALUE) in many places, so 8-byte offsets are probably not needed. On Thu, Jun 16, 2022 at 5:04 PM Aleksey Demakov <adema...@gmail.com> wrote: > Hi all, > > I drafted a proposal for a new data format for Apache Ignite 3.0. For > lack of a better name it is called Binary Tuple Format [1]. The idea > is to have a common representation of rows and other similar data > whenever that data is transferred and handled in binary form by > different database modules. For instance, a storage engine could > produce a batch of rows for the SQL engine, the SQL engine might do > some operations over it like filtering, projection, joining and then > pass the result further for another round of operations on a different > node, and then the final result will be delivered to the client. It > seems beneficial to use the same format everywhere as long as it is > possible. Please note that the final SQL query result may have very > little to do with any actual table schema stored in the database. So > here, in contrast with e.g. IEP-54 [2], a tuple does not necessarily > equate to a table row and has a broader meaning. > > I would ask the community for a review. Any feedback would be greatly > appreciated. > > Regards, > Aleksey > > [1] > https://cwiki.apache.org/confluence/display/IGNITE/IEP-92%3A+Binary+Tuple+Format > [2] > https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach >
