Hello igniters, while I would like to help on the calcite because H2 optimiser (or the lack thereof) is really killing us, I think that it would be wiser to start by contributing on something easier.
Therefore I will tackle another problem that we have which is the memory consumption. I stumbled upon this IEP https://cwiki.apache.org/confluence/display/IGNITE/IEP-2%3A+Binary+object+format+improvements <https://cwiki.apache.org/confluence/display/IGNITE/IEP-2%3A+Binary+object+format+improvements> that is about optimising the binary marshaller. The low hanging fruit seemed to be the null compaction so I decided to start with it. Though I am sure I do see some hidden complexity. Here a couple of questions: - Can I assign myself IGNITE-6499 and attach a patch? - Who can I contact to help with the review. In the following page https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute <https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute> there is no one assigned for marshalling. On the details: The compression is disabled by default as it is not compatible with objects previously marshalled. My approach was to go a bit beyond the JIRA. No only do I remove the indexes to null fields in the footer, I also remove the 0x65 in the objects. I did not remove them fro the collections and arrays because they are using absolute positioning. I gain between 5% to 20% depending of my test cases. Obviously the smaller the object and the higher the number of nulls, the higher the compression rate. Based on that I can quite easily add var int compression which is IGNITE-6418 and should significantly increase the compression rate with a lot of integers and longs when only using small numbers. Next step is to add JMH micro-benchmark to check the impact in terms of performances. Example on a simple object w/ null compaction Length=55 FooterPosition=50 0x67 // ValueType 0x01 // FormatVersion 0x2b 0x00 //Flags userType=true hasSchema=true offset=1 compactFooter=true 0x78 0x66 0xbe 0x44 //TypeId 0xf9 0xcd 0x07 0x57 //Hashcode 0x37 0x00 0x00 0x00 //Length 0x3d 0xa8 0x15 0xe4 //SchemaId 0x32 0x00 0x00 0x00 //Footer position = 50 0x03 0x01 0x00 0x00 0x00 0x03 0x01 0x00 0x00 0x00 0x09 0x03 0x00 0x00 0x00 0x61 0x62 0x63 0x09 0x03 0x00 0x00 0x00 0x61 0x62 0x63 Footer length=5 0x18 0x1d 0x22 0x2a 0x47 and w/o null compaction Length=60 FooterPosition=53 0x67 // ValueType 0x01 // FormatVersion 0x2b 0x00 //Flags userType=true hasSchema=true offset=1 compactFooter=true 0x78 0x66 0xbe 0x44 //TypeId 0xa4 0x43 0x0e 0xf5 //Hashcode 0x3c 0x00 0x00 0x00 //Length 0x3d 0xa8 0x15 0xe4 //SchemaId 0x35 0x00 0x00 0x00 //Footer position = 53 0x03 0x01 0x00 0x00 0x00 0x03 0x01 0x00 0x00 0x00 0x09 0x03 0x00 0x00 0x00 0x61 0x62 0x63 0x65 0x65 0x65 0x09 0x03 0x00 0x00 0x00 0x61 0x62 0x63 Footer length=7 0x18 0x1d 0x22 0x2a 0x2b 0x2c 0x2d -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
