Vladimir, Thanks for explanation... that’s true - ignite never deserialize values during query execution. The point here is why copying fields to heap that sql engine could not use (not annotated as QuerySqlField)? SQL count is a perfect example where you can benefit (heap space and gc) copying only field sql engine could use.
Regards, Igor On Wed, Oct 17, 2018 at 18:47 Vladimir Ozerov <voze...@gridgain.com> wrote: > Hi Igor, > > We never deserialize values during query execution. Instead, we copy the > row to heap and extract fields as needed. In general case it is impossible > to avoid reading the whole row because we do not know whether this is > COUNT(*) or COUNT(*) WHERE <something> or COUNT(<something>) WHERE > <something>. > We do handle plain COUNT(*) as speical case and iterate over index only, > but event this simple query cannot avoid row reads in general case when new > snapshot mode is enabled, because an entry in the index may be not visible > to current transaction. > > On Wed, Oct 17, 2018 at 7:19 PM igor.tanackovic <igor.tanacko...@gmail.com > > > wrote: > > > Hello, > > > > Seems that SQL engine always deserialize whole objects instead of using > > just > > SQL enabled fields (annotated with @QuerySqlField). This may have a huge > > impact on Ignite heap usage and GC overhead as well. > > > > For example, we have a cache holding big objects but with only two sql > > query > > fields which for each query execution (SELECT COUNT(*) FROM 'cache') > > consumes large amount on heap memory (~300MB). As a proof of concept, we > > divided the same cache to *index* cache with only sql query field and a > > *data* holding whole object for materialization. The same query (SELECT > > COUNT(*) FROM 'index-cache') consumes ~25 time less memory! The same is > > true > > for all other queries. > > > > The obvious workaround would be to always have separated regions for > > indexes > > (sql query enabled region) and a data/value region for materialization, > but > > it might be a good idea to fix this in a systematic way during off heap > > deserialization. > > > > Regards, > > Igor > > > > > > > > -- > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ > > >
