Sven, Support of materialized views sounds like a huge project. I would not expect it to appear in Ignite soon.
As far as I see you have problems with data collocation. If you can not store the original data in replicated caches, then these views will be huge as well and thus must be partitioned with some collocation. So it does not change the picture too much actually. I would suggest to go for a manual denormalization: put the same value with different affinity keys. Sergi пн, 17 сент. 2018 г. в 14:22, Sven Beauprez <sven.beaup...@theglue.com>: > Hi Dmitry, > > Yes we can use those solutions in some cases, but not always. > > Replication is indeed the simplest, but sadly enough replicated caches are > too much overhead. We have often a minimum of 12 nodes and all data must > stay in sync 12x then. We do use it for small caches that don't need a lot > of updates. > > We use colocation all over the place. Colocation based on affinity keys is > not possible though for distinct data sets with only some very specific > relationships with _some other_ dataset, well known before hand. > (eg. for example -not our exact use case which is more complex- items in > a shopping basket with items from product inventory, both are in different > caches managed on other nodes and it is not possible to denormalize such > that the shopping basket knows the amount of availble items) > > > Regards, > > Sven > > > > > SVEN BEAUPREZ > > L e a d A r c h i t e c t > > > > De Kleetlaan 5, B-1831 Diegem > > www.theglue.com <http://www.theglue.com/> > > On 17/09/2018, 10:37, "Dmitriy Setrakyan" <dsetrak...@apache.org> wrote: > > Hi Sven, > > I will let others comment on the materialized view suggestion, but I > have > another question. > > *As we all know, joins are a nightmare in a distributed system* > > > Have you considered collocation or replication? If a table is > replicated, > then it will be present on all the nodes and all joins will be fast. > If two > partitioned tables are colocated based on some affinity key, then > joins on > that affinity key will be fast as well. > > Both, colocation and replication are supported by Ignite. Will any of > these > approaches work for you? > > D. > > On Mon, Sep 17, 2018 at 11:04 AM Sven Beauprez < > sven.beaup...@theglue.com> > wrote: > > > All, > > > > > > > > We are in a situation where we have to query data over several > caches. As > > we all know, joins are a nightmare in a distributed system and I > know there > > are other means like denormalisation, but it is not sufficient > anymore in > > some cases we have and we need the joins. > > > > > > > > We mainly work in an OLTP context, where queries are known in > advance (ie > > dev time) and inpsired by following blog of several years ago, I was > > wondering if the concept of “materialized views” could make it into > Apache > > Ignite. > > > > ( > > > https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ > > ) > > > > > > > > It would work as follows: > > > > - A query must register itself in Ignite at startup time (eg. via > > configuration) or during run time (eg. API call) > > - The registered query is parsed and a new “view” cache is created > > which will ‘cache’ the resultset of the query (could take a > while, but > > intermediate status can be “warming up” and “hot” when ready) > > - All caches involved in the joins are now monitored for CUD > > operations and relevant data is stored in the new “view” cache so > the view > > gets updated in real time > > - All operations must be ACID compliant > > - The view is queried via a very trivial select statement > > > > > > > > Would that be feasible as a new feature? > > > > > > > > > > > > Regards, > > > > > > > > Sven > > > > > > > > > > > > > > > > [image: cid:image001.png@01D3007B.4D007790] > > > > > > > > SVEN BEAUPREZ > > > > > > > > L e a d A r c h i t e c t > > > > > > > > De Kleetlaan 5, B-1831 Diegem > > > > www.theglue.com > > > > >
