On September 2 Courtney will tell more about Hypi use case, join us at Virtual Meetup: https://www.meetup.com/Apache-Ignite-Virtual-Meetup/events/280030600/
Below I quote Courtney's email to the user list: The talk will look at the general architecture, playing nice with other > technologies and the future reactive-streams architecture Hypi's team > started work on with a view for being in production in 2022. We'll look at why we're transitioning to reactive streams and how Ignite is > being used to accelerate other technologies in the stack. Just like Ignite > 3, the next-gen of Hypi itself is using Apache Calcite as a data > virtualisation layer. The talk will explore a little of how this is being > done and finally will discuss some challenges and pitfalls to avoid. вт, 3 авг. 2021 г. в 12:38, Ivan Pavlukhin <vololo...@gmail.com>: > Hi Courtney, > > Statistics for query planning is a rather complex subject. > Unfortunately I am now aware of technical details how is it expected > to be implemented in Ignite 3. > > Folks driving SQL please step in. > > 2021-07-31 20:18 GMT+03:00, Courtney Robinson <courtney.robin...@hypi.io>: > > Hi Ivan, > > Atri's description of the query plan being cached is what I was thinking > of > > with my description. > > > > I lack the knowledge on how the statistics are maintained to really > comment > > constructively Atri but my first question about the problem you raise > with > > statistics would be: > > > > How/where are the stats maintained and if a query plan is cached based on > > some stats, is it not possible to invalidate the cached plan periodically > > or based on statistics changes? > > > > Regards, > > Courtney Robinson > > Founder and CEO, Hypi > > Tel: ++44 208 123 2413 (GMT+0) <https://hypi.io> > > > > <https://hypi.io> > > https://hypi.io > > > > > > On Sat, Jul 31, 2021 at 8:54 AM Atri Sharma <a...@apache.org> wrote: > > > >> Query caching works on three levels - caching results, caching blocks > and > >> caching query plans. > >> > >> Prepared queries work by caching a plan for a query and reusing that > plan > >> by changing the parameters for the incoming query. So the query remains > >> the > >> same, but input values keep changing. > >> > >> The problem with prepared queries is that query execution can go bad > very > >> fast if the underlying data distribution changes and the cached plan is > >> no > >> longer optimal for the given statistics. > >> > >> On Sat, 31 Jul 2021, 12:54 Ivan Pavlukhin, <vololo...@gmail.com> wrote: > >> > >> > Hi Courtney, > >> > > >> > Please clarify what do you mean by prepared queries and query caching? > >> > Do you mean caching query results? If so, in my mind material views > >> > are the best approach here (Ignite 2 does not support them). Do you > >> > have other good approaches in your mind? E.g. implemented in other > >> > databases. > >> > > >> > 2021-07-26 21:27 GMT+03:00, Valentin Kulichenko < > >> > valentin.kuliche...@gmail.com>: > >> > > Hi Courtney, > >> > > > >> > > Generally speaking, query caching certainly makes sense. As far as I > >> > know, > >> > > Ignite 2.x actually does that, but most likely there might be room > >> > > for > >> > > improvement as well. We will look into this. > >> > > > >> > > As for the SQL API - the answer is yes. The requirement for a dummy > >> cache > >> > > is an artifact of the current architecture. This is 100% wrong and > >> > > will > >> > be > >> > > changed in 3.0. > >> > > > >> > > -Val > >> > > > >> > > On Sun, Jul 25, 2021 at 2:51 PM Courtney Robinson > >> > > <courtney.robin...@hypi.io> > >> > > wrote: > >> > > > >> > >> Something else came to mind, are there plans to support prepared > >> > queries? > >> > >> > >> > >> I recall someone saying before that Ignite does internally cache > >> queries > >> > >> but it's not at all clear if or how it does do that. I assume a > >> > >> simple > >> > >> hash > >> > >> of the query isn't enough. > >> > >> > >> > >> We generate SQL queries based on user runtime settings and they can > >> get > >> > >> to > >> > >> hundreds of lines long, I imagine this means most of our queries > are > >> not > >> > >> being cached but there are patterns so we could generate and manage > >> > >> prepared queries ourselves. > >> > >> > >> > >> Also, will there be a dedicated API for doing SQL queries rather > >> > >> than > >> > >> having to pass a SqlFieldsQuery to a cache that has nothing to do > >> > >> with > >> > >> the > >> > >> cache being queried? When I first started with Ignite years ago, > >> > >> this > >> > was > >> > >> beyond confusing for me. I'm trying to run select x from B but I > >> > >> pass > >> > >> this > >> > >> to a cache called DUMMY or whatever arbitrary name... > >> > >> > >> > >> On Fri, Jul 23, 2021 at 4:05 PM Courtney Robinson < > >> > >> courtney.robin...@hypi.io> > >> > >> wrote: > >> > >> > >> > >> > Andrey, > >> > >> > Thanks for the response - see my comments inline. > >> > >> > > >> > >> > > >> > >> >> I've gone through the questions and have no the whole picture of > >> your > >> > >> use > >> > >> >> case. > >> > >> > > >> > >> > Would you please clarify how you exactly use the Ignite? what are > >> the > >> > >> >> integration points? > >> > >> >> > >> > >> > > >> > >> > I'll try to clarify - we have a low/no code platform. A user > >> designs a > >> > >> > model for their application and we map this model to Ignite > tables > >> and > >> > >> > other data sources. The model I'll describe is what we're > building > >> now > >> > >> and > >> > >> > expected to be in alpha some time in Q4 21. Our current > production > >> > >> > architecture is different and isn't as generic, it is heavily > tied > >> to > >> > >> > Ignite and we've redesigned to get some flexibility where Ignite > >> > >> > doesn't > >> > >> > provide what we want. Things like window functions and other > >> > >> > SQL-99 > >> > >> limits. > >> > >> > > >> > >> > In the next gen version we're working on you can create a model > >> > >> > for > >> a > >> > >> > Tweet(content, to) and we will create an Ignite table with > content > >> and > >> > >> > to > >> > >> > columns using the type the user selects. This is the simplest > >> > >> > case. > >> > >> > We are adding generic support for sources and sinks and using > >> Calcite > >> > >> > as > >> > >> a > >> > >> > data virtualisation layer. Ignite is one of the available > >> > source/sinks. > >> > >> > > >> > >> > When a user creates a model for Tweet, we also allow them to > >> > >> > specify > >> > >> > how > >> > >> > they want to index the data. We have a copy of the calcite > >> > >> > Elasticsearch > >> > >> > adapter modified for Solr. > >> > >> > > >> > >> > When a source is queried (Ignite or any other that we support), > we > >> > >> > generate SQL that Calcite executes. Calcite will push down the > >> > >> > generated > >> > >> > queries to Solr and Solr produces a list of IDs (in case of > >> > >> > Ignite) > >> > and > >> > >> we > >> > >> > do a multi-get from Ignite to produce the actual results. > >> > >> > > >> > >> > Obviously there's a lot more to this but that should give you a > >> > general > >> > >> > idea. > >> > >> > > >> > >> > and maybe share some experience with using Ignite SPIs? > >> > >> >> > >> > >> > Our evolution with Ignite started from the key value + compute > >> > >> > APIs. > >> > We > >> > >> > used the SPIs then but have since moved to using only the Ignite > >> > >> > SQL > >> > >> > API > >> > >> > (we gave up transactions for this). > >> > >> > > >> > >> > We originally used the indexing SPI to keep our own lucene index > >> > >> > of > >> > >> > data > >> > >> > in a cache. We did not use the Ignite FTS as it is very limited > >> > >> > compared > >> > >> to > >> > >> > what we allow customers to do. If I remember correctly, we were > >> using > >> > >> > an > >> > >> > affinity compute job to send queries to the right Ignite node and > >> > >> > then doing a multi-get to pull the data from caches. > >> > >> > I think we used one or two other SPIs and we found them very > >> > >> > useful > >> to > >> > >> > be > >> > >> > able to extend and customise Ignite without having to fork/change > >> > >> upstream > >> > >> > classes. We only stopped using them because we eventually > >> > >> > concluded > >> > >> > that > >> > >> > using the SQL only API was better for numerous reasons. > >> > >> > > >> > >> > > >> > >> >> We'll keep the information in mind while developing the Ignite, > >> > >> >> because this may help us to make a better product. > >> > >> >> > >> > >> >> By the way, I'll try to answer the questions. > >> > >> >> > >> > >> >> > 1. Schema change - does that include the ability to change > >> > >> >> > the > >> > >> >> > types > >> > >> >> of > >> > >> >> > fields/columns? > >> > >> >> Yes, we plan to support transparent conversion to a wider type > >> on-fly > >> > >> >> (e.g. > >> > >> >> 'int' to 'long'). > >> > >> >> This is a major point of our Live-schema concept. > >> > >> >> In fact, there is no need to convert data on all the nodes in a > >> > >> >> synchronous > >> > >> >> way as old SQL databases do (if one supports though), > >> > >> >> we are going to support multiple schema versions and convert > data > >> > >> >> on-demand > >> > >> >> on a per-row basis to the latest version, > >> > >> >> then write-back the row. > >> > >> >> > >> > >> > > >> > >> > I can understand. The auto conversion to wider type makes sense. > >> > >> > > >> > >> >> > >> > >> >> More complex things like 'String' -> 'int' are out of scope for > >> > >> >> now > >> > >> >> because > >> > >> >> it requires the execution of a user code on the critical path. > >> > >> >> > >> > >> > > >> > >> > I would argue though that executing user code on the critical > path > >> > >> > shouldn't be a blocker for custom conversions. I feel if a user > is > >> > >> > making > >> > >> > an advance enough integration to provide custom conversions they > >> would > >> > >> > be > >> > >> > aware that it impacts the system as a whole. > >> > >> > > >> > >> > The limitation here is column MUST NOT be indexed, because an > >> > >> > index > >> > >> > over > >> > >> >> the data of different kinds is impossible. > >> > >> >> > >> > >> > Understood - I'd make the case that indexing should be > pluggable. > >> > >> > I > >> > >> would > >> > >> > love for us to be able to take indexing away from Ignite in our > >> impl. > >> > - > >> > >> > I > >> > >> > think in Calcite, the Postgres adapter does this by having a > table > >> > >> > whose > >> > >> > type is "Index". The implementor would be left with the freedom > to > >> > >> > choose > >> > >> > how that table answers index lookups. From Ignite's perspective > it > >> > >> wouldn't > >> > >> > care so long as the interface's contract is met, I could use an > >> index > >> > >> that > >> > >> > does a lucene, ES, Solr or Redis lookup and the end result would > >> > >> > be > >> > the > >> > >> > same but as the implementor I'm choosing the tradeoff I want to > >> > >> > meet > >> > >> > the > >> > >> > organisation's goals. > >> > >> > > >> > >> > > >> > >> >> > >> > >> >> > 2. Will the new guaranteed consistency between APIs also > mean > >> SQL > >> > >> will > >> > >> >> > gain transaction support? > >> > >> >> Yes, we plan to have Transactional SQL. > >> > >> >> DDL will be non-transactional though, and I wonder if the one > >> > supports > >> > >> >> this. > >> > >> >> > >> > >> > I'm not sure I know of any thing that supports transactional DDL > >> > >> > so > >> > >> > don't > >> > >> > think this is an issue but I would say that a DDL statement in a > >> > >> > transaction shouldn't fail the transaction. I believe in Ignite 2 > >> > there > >> > >> is > >> > >> > a flag to turn this on or off, we should definitely keep this. > In > >> our > >> > >> > case, it's an issue with the nature of the platform we provide, > at > >> > >> > development time only about 10% of schema or other DB info is > >> > >> > known > >> - > >> > >> > we > >> > >> > generate the other 90% on the fly based on whatever customers > >> > >> > decide > >> > to > >> > >> > design from our UI. > >> > >> > > >> > >> >> > >> > >> >> Ignite 3 will operate with Rows underneath, but classic Table > API > >> and > >> > >> >> Key-value will be available to a user > >> > >> >> at the same time and with all consistency guarantees. > >> > >> > > >> > >> > Excellent! > >> > >> > > >> > >> >> > >> > >> >> > >> > >> > > >> > >> >> > 3. Has there been any decision about how much of Calcite will > >> > >> >> > be > >> > >> >> exposed > >> > >> >> > to the client? When using thick clients, it'll be hugely > >> > >> >> > beneficial > >> > >> to > >> > >> >> be > >> > >> >> > able to work with Calcite APIs directly to provide custom > >> > >> >> > rules > >> > >> >> > and > >> > >> >> > optimizations to better suit organization needs > >> > >> >> As of now, we have no plans to expose any Calcite API to a user. > >> > >> >> AFAIK, we have our custom Calcite convention, custom rules that > >> > >> >> are > >> > >> aware > >> > >> >> of distributed environment, > >> > >> >> and additional AST nodes. The rules MUST correctly propagate > >> internal > >> > >> >> information about data distribution, > >> > >> >> so I'm not sure want to give low-level access to them. > >> > >> >> > >> > >> > > >> > >> > Maybe we're an edge case but for us access to the Calcite APIs > >> > >> > would > >> > be > >> > >> > shift our current development somewhat. For us, we're treating > >> Ignite > >> > >> > as > >> > >> a > >> > >> > library that provides a good foundation and we extend and > >> > >> > customise > >> > it. > >> > >> > Again, we may be an edge case and maybe most people just want a > >> > >> > database > >> > >> to > >> > >> > put data into and get it back out without controlling some of how > >> > >> > it > >> > >> > does > >> > >> > those things. > >> > >> > > >> > >> > > >> > >> >> > We Index into Solr and use the Solr indices > >> > >> >> Ignite 1-2 has poor support for TEXT queries, which is totally > >> > >> >> unconfigurable. > >> > >> >> Also, Lucene indices underneath are NOT persistent that requires > >> too > >> > >> much > >> > >> >> effort to fix it. > >> > >> >> GeoSpatial index has the same issues, we decided to drop them > >> > >> >> along > >> > >> >> with > >> > >> >> Indexing SPI at all. > >> > >> >> > >> > >> >> However, you can find the activity on dev-list on the Index > Query > >> > >> >> topic. > >> > >> >> Guys are going to add IndexQuery (a scan query over the sorted > >> index > >> > >> which > >> > >> >> can use simple conditions) in Ignite 2. > >> > >> >> We also plan to have the same functionality, maybe it is > possible > >> to > >> > >> >> add > >> > >> >> full-text search support here. > >> > >> >> Will it work for you, what do you think? > >> > >> >> > >> > >> > Yes, we originally looked at text queries and almost immediately > >> said > >> > >> > no. > >> > >> > Nothing about it was useful for us other than the lucene > >> > >> > dependency > >> in > >> > >> > Java. In the end that also became an issue because we wanted a > >> > >> > newer > >> > >> lucene > >> > >> > version. > >> > >> > IndexQuery will be useful - we'll certainly use it but it's not > >> > enough. > >> > >> > I > >> > >> > think we customise and depend on Solr too much for IndexQuery to > >> > >> > compare > >> > >> > but it will help in some cases for simpler queries. > >> > >> > > >> > >> >> > >> > >> >> > >> > >> >> > 4. Will the unified storage model enable different versions > >> > >> >> > of > >> > >> Ignite > >> > >> >> to > >> > >> >> > be in the cluster when persistence is enabled so that > rolling > >> > >> restarts > >> > >> >> can > >> > >> >> > be done? > >> > >> >> I'm not sure a rolling upgrade (RU) will be available because > too > >> > much > >> > >> >> compatibility issues should be resolved > >> > >> >> to make RU possible under the load without downtime. > >> > >> >> > >> > >> >> Maybe it makes sense to provide some grid mode (maintenance > mode) > >> for > >> > >> >> RU > >> > >> >> purposes that will block all the user load > >> > >> >> but allow upgrade the grid. E.g. for the pure in-memory case. > >> > >> >> > >> > >> >> Persistence compatibility should be preserved as it works for > >> Ignite > >> > >> >> 2. > >> > >> >> > >> > >> > My ideal situation would be that we start a newer Ignite version, > >> > >> > it > >> > >> comes > >> > >> > online, joins the cluster and is treated as some kind of > >> > >> > maintenance > >> > >> > mode > >> > >> > as you suggested. In maintenance mode, the other nodes re-balance > >> > >> > or > >> > >> > some > >> > >> > other process to send all the data this new node will handle over > >> > >> > to > >> > >> > it. > >> > >> > The existing nodes continue serving this data until the new node > >> > >> > is > >> no > >> > >> > longer in maintenance mode and then it becomes the primary for > the > >> > data > >> > >> > that was rebalanced to it. > >> > >> > > >> > >> > The second case is if an existing node is restarted with a newer > >> > Ignite > >> > >> > version. No re-balance is needed, it joins in maintenance mode, > >> > >> > runs > >> > >> > any > >> > >> > upgrade/conversion or other task it needs to and then starts > >> accepting > >> > >> > reads and writes. Communication with lower version nodes can be > >> > >> > limited, > >> > >> > they are aware of it and sends it data and queries for which it > is > >> the > >> > >> > primary assuming they will also be upgraded. > >> > >> > > >> > >> > I guess I'm not aware of the compatibility issues this presents > >> > >> > and > >> so > >> > >> > my > >> > >> > view is narrow and perhaps naive here. > >> > >> > > >> > >> >> > >> > >> >> > >> > >> >> > 5. Will it be possible to provide a custom cache store > still > >> and > >> > >> will > >> > >> >> > these changes enable custom cache stores to be queryable > from > >> > SQL? > >> > >> >> I'm not sure I fully understand this. > >> > >> >> 1. Usually, SQL is about indices. Ignite can't perform a query > >> > >> >> over > >> > >> >> the > >> > >> >> unindexed data. > >> > >> >> > >> > >> > Yes understood > >> > >> > > >> > >> >> > >> > >> >> 2. Fullscan over the cache that contains only part of data + > scan > >> the > >> > >> >> CacheStore, then merging the results is a pain. > >> > >> >> Most likely, running a query over CacheStore directly will be a > >> > >> >> simpler > >> > >> >> way, and even more performant. > >> > >> >> Shared CacheStore (same for all nodes) will definitely kill the > >> > >> >> performance > >> > >> >> in that case. > >> > >> >> So, the preliminary loadCache() call looks like a good > >> > >> >> compromise. > >> > >> >> > >> > >> > I think the problem is largely that the CacheStore interface is > >> > >> > not > >> > >> > sufficient for being able to do this. If it had a richer > interface > >> > >> > which > >> > >> > allowed the cache store to answer index queries basically hooking > >> into > >> > >> > whatever Ignite's doing for its B+tree then this would be viable. > >> > >> > A > >> > >> > CacheStore that only implements KV API doesn't take part in SQL > >> > >> > queries. > >> > >> > > >> > >> >> > >> > >> >> 3. Splitting query into 2 parts to run on Ignite and to run on > >> > >> CacheStore > >> > >> >> looks possible with Calcite, > >> > >> >> but I think it impractical because in general, neither > CacheStore > >> nor > >> > >> >> database structure are aware of the data partitioning. > >> > >> >> > >> > >> > hmmm, maybe I missed the point but as the implementor of the > >> > CacheStore > >> > >> > you should have knowledge of the structure and partition info. or > >> have > >> > >> some > >> > >> > way of retrieving it. Again, I think the current CacheStore > >> interface > >> > >> > is > >> > >> > the problem and if it was extended to provide this information > >> > >> > then > >> > its > >> > >> up > >> > >> > to the implementation to do this whilst Ignite knows that any > >> > >> > implementation of these interfaces will meet the contract > >> > >> > necessary. > >> > >> > > >> > >> > > >> > >> >> > >> > >> >> 4. Transactions can't be supported in case of direct CacheStore > >> > >> >> access, > >> > >> >> because even if the underlying database supports 2-phase commit, > >> > which > >> > >> is > >> > >> >> a > >> > >> >> rare case, the recovery protocol looks hard. > >> > >> >> Just looks like this feature doesn't worth it. > >> > >> >> > >> > >> > I'd completely agree with this. It will be incredibly hard to get > >> this > >> > >> > done reliably > >> > >> > > >> > >> >> > >> > >> >> > >> > >> >> > 6. This question wasn't mine but I was going to ask it as > >> > >> >> > well: > >> > >> >> > What > >> > >> >> > will happen to the Indexing API since H2 is being removed? > >> > >> >> As I wrote above, Indexing SPI will be dropped, but IndexQuery > >> > >> >> will > >> > be > >> > >> >> added. > >> > >> >> > >> > >> >> > 1. As I mentioned above, we Index into Solr, in earlier > >> > >> >> > versions > >> > of > >> > >> >> > our product we used the indexing SPI to index into Lucene > >> > >> >> > on > >> > >> >> > the > >> > >> >> Ignite > >> > >> >> > nodes but this presented so many challenges we ultimately > >> > >> abandoned > >> > >> >> it and > >> > >> >> > replaced it with the current Solr solution. > >> > >> >> AFAIK, some guys developed and sell a plugin for Ignite-2 with > >> > >> persistent > >> > >> >> Lucene and Geo indices. > >> > >> >> I don't know about the capabilities and limitations of their > >> > solution, > >> > >> >> because of closed code. > >> > >> >> You can easily google it. > >> > >> >> > >> > >> >> I saw few encouraged guys who want to improve TEXT queries, > >> > >> >> but unfortunately, things weren't moved far enough. For now, > they > >> are > >> > >> >> in > >> > >> >> the middle of fixing the merging TEXT query results. > >> > >> >> So far so good. > >> > >> >> > >> > >> >> I think it is a good chance to master the skill developing of a > >> > >> >> distributed > >> > >> >> system for the one > >> > >> >> who will take a lead over the full-text search feature and add > >> native > >> > >> >> FullText index support into Ignite-3. > >> > >> >> > >> > >> > I've seen the other thread from Atri I believe about this. > >> > >> > > >> > >> >> > >> > >> >> > >> > >> >> > 7. What impact does RAFT now have on conflict resolution? > >> > >> >> RAFT is a state machine replication protocol. It guarantees all > >> > >> >> the > >> > >> nodes > >> > >> >> will see the updates in the same order. > >> > >> >> So, seems no conflicts are possible. Recovery from split-brain > is > >> > >> >> impossible in common-case. > >> > >> >> > >> > >> >> However, I think we have a conflict resolver analog in Ignite-3 > >> > >> >> as > >> it > >> > >> >> is > >> > >> >> very useful in some cases > >> > >> >> e.g datacenter replication, incremental data load from 3-rd > party > >> > >> source, > >> > >> >> recovery from 3-rd party source. > >> > >> >> > >> > >> >> > >> > >> >> > 8. CacheGroups. > >> > >> >> AFAIK, CacheGroup will be eliminated, actually, we'll keep this > >> > >> mechanic, > >> > >> >> but it will be configured in a different way, > >> > >> >> which makes Ignite configuring a bit simpler. > >> > >> >> Sorry, for now, I have no answer on your performance concerns, > >> > >> >> this > >> > >> >> part > >> > >> >> of > >> > >> >> Ignite-3 slipped from my radar. > >> > >> >> > >> > >> > No worries. I'll wait and see if anyone else suggests something. > >> > >> > Its > >> > >> > getting a lot worse, a node took 1hr to start yesterday after a > >> > >> deployment > >> > >> > and its in prod with very little visibility into what it is > doing, > >> it > >> > >> > was > >> > >> > just stopped, no logging or anything and then resumed. > >> > >> > > >> > >> > 2021-07-22 13:40:15.997 INFO [ArcOS,,,] 9 --- [orker-#40%hypi%] > >> > >> > o.a.i.i.p.cache.GridCacheProcessor [285] : Finished > recovery > >> for > >> > >> > cache [cache=hypi_01F8ZC3DGT66RNYCDZH3XNVY2E_Hue, grp=hypi, > >> > >> > startVer=AffinityTopologyVersion [topVer=79, minorTopVer=0]] > >> > >> > > >> > >> > One hour later it printed the next cache recovery message and > >> started > >> > >> > 30 > >> > >> > seconds after going through other tables. > >> > >> > > >> > >> > > >> > >> > > >> > >> >> > >> > >> >> Let's wait if someone will clarify what we could expect in > >> Ignite-3. > >> > >> >> Guys, can someone chime in and give more light on 3,4,7,8 > >> questions? > >> > >> >> > >> > >> >> > >> > >> >> On Thu, Jul 22, 2021 at 4:15 AM Courtney Robinson < > >> > >> >> courtney.robin...@hypi.io> > >> > >> >> wrote: > >> > >> >> > >> > >> >> > Hey everyone, > >> > >> >> > I attended the Alpha 2 update yesterday and was quite pleased > >> > >> >> > to > >> > see > >> > >> the > >> > >> >> > progress on things so far. So first, congratulations to > >> > >> >> > everyone > >> on > >> > >> the > >> > >> >> > work being put in and thank you to Val and Kseniya for running > >> > >> >> yesterday's > >> > >> >> > event. > >> > >> >> > > >> > >> >> > I asked a few questions after the webinar which Val had some > >> > answers > >> > >> to > >> > >> >> but > >> > >> >> > suggested posting here as some of them are not things that > have > >> > been > >> > >> >> > thought about yet or no plans exist around it at this point. > >> > >> >> > > >> > >> >> > I'll put all of them here and if necessary we can break into > >> > >> >> > different > >> > >> >> > threads after. > >> > >> >> > > >> > >> >> > 1. Schema change - does that include the ability to change > >> > >> >> > the > >> > >> types > >> > >> >> of > >> > >> >> > fields/columns? > >> > >> >> > 1. Val's answer was yes with some limitations but those > >> > >> >> > are > >> > >> >> > not > >> > >> >> well > >> > >> >> > defined yet. He did mention that something like some > kind > >> of > >> > >> >> > transformer > >> > >> >> > could be provided for doing the conversion and I would > >> second > >> > >> >> this, > >> > >> >> > even > >> > >> >> > for common types like int to long being able to do a > >> > >> >> > custom > >> > >> >> > conversion will > >> > >> >> > be immensely valuable. > >> > >> >> > 2. Will the new guaranteed consistency between APIs also > >> > >> >> > mean > >> > SQL > >> > >> >> will > >> > >> >> > gain transaction support? > >> > >> >> > 1. I believe the answer here was yes but perhaps someone > >> else > >> > >> may > >> > >> >> > want to weigh in to confirm > >> > >> >> > 3. Has there been any decision about how much of Calcite > >> > >> >> > will > >> be > >> > >> >> exposed > >> > >> >> > to the client? When using thick clients, it'll be hugely > >> > >> >> > beneficial > >> > >> >> to > >> > >> >> > be > >> > >> >> > able to work with Calcite APIs directly to provide custom > >> rules > >> > >> >> > and > >> > >> >> > optimisations to better suit organisation needs > >> > >> >> > 1. We currently use Calcite ourselves and have a lot of > >> > >> >> > custom > >> > >> rules > >> > >> >> and > >> > >> >> > optimisations and have slowly pushed more of our queries > >> > >> >> > to > >> > >> >> > Calcite that we > >> > >> >> > then push down to Ignite. > >> > >> >> > 2. We Index into Solr and use the Solr indices and > others > >> to > >> > >> >> > fulfill over all queries with Ignite just being one of > >> > >> >> > the > >> > >> >> > possible storage > >> > >> >> > targets Calcite pushes down to. If we could get to the > >> > calcite > >> > >> >> > API from an > >> > >> >> > Ignite thick client, it would enable us to remove a > layer > >> of > >> > >> >> > abstraction > >> > >> >> > and complexity and make Ignite our primary that we then > >> link > >> > >> >> > with Solr and > >> > >> >> > others to fulfill queries. > >> > >> >> > 4. Will the unified storage model enable different versions > >> > >> >> > of > >> > >> >> Ignite to > >> > >> >> > be in the cluster when persistence is enabled so that > >> > >> >> > rolling > >> > >> >> restarts > >> > >> >> > can > >> > >> >> > be done? > >> > >> >> > 1. We have to do a strange dance to perform Ignite upgrades > >> > >> >> > without > >> > >> >> > downtime because pods/nodes will fail to start on > version > >> > >> mismatch > >> > >> >> > and if > >> > >> >> > we get that dance wrong, we will corrupt a node's data. > >> > >> >> > It > >> > >> >> > will > >> > >> >> make > >> > >> >> > admin/upgrades far less brittle and error prone if this > >> > >> >> > was > >> > >> >> possible. > >> > >> >> > 5. Will it be possible to provide a custom cache store > still > >> and > >> > >> will > >> > >> >> > these changes enable custom cache stores to be queryable > >> > >> >> > from > >> > >> >> > SQL? > >> > >> >> > 1. Our Ignite usage is wide and complex because we use KV, > >> > >> >> > SQL > >> > >> >> > and > >> > >> >> other > >> > >> >> > APIs. The inconsistency of what can and can't be used > >> > >> >> > from > >> > one > >> > >> >> API to > >> > >> >> > another is a real challenge and has forced us over time > >> > >> >> > to > >> > >> >> > stick > >> > >> >> > to one API > >> > >> >> > and write alternative solutions outside of Ignite. It > >> > >> >> > will > >> > >> >> > drastically > >> > >> >> > simplify things if any CacheStore (or some new > >> > >> >> > equivalent) > >> > >> >> > could > >> > >> >> > be plugged > >> > >> >> > in and be made accessible to SQL (and in fact all other > >> APIs) > >> > >> >> without > >> > >> >> > having to load all the data from the underlying > >> > >> >> > CacheStore > >> > >> >> > first > >> > >> >> > into memory > >> > >> >> > 6. This question wasn't mine but I was going to ask it as > >> well: > >> > >> What > >> > >> >> > will happen to the Indexing API since H2 is being removed? > >> > >> >> > 1. As I mentioned above, we Index into Solr, in earlier > >> > >> >> > versions > >> > >> >> of > >> > >> >> > our product we used the indexing SPI to index into > Lucene > >> on > >> > >> >> > the > >> > >> >> > Ignite > >> > >> >> > nodes but this presented so many challenges we > ultimately > >> > >> >> > abandoned it and > >> > >> >> > replaced it with the current Solr solution. > >> > >> >> > 2. Lucene indexing was ideal because it meant we didn't > >> have > >> > >> >> > to > >> > >> >> > re-invent Solr or Elasticsearch's sharding capabilities, > >> that > >> > >> was > >> > >> >> > almost > >> > >> >> > automatic with Ignite only giving you the data that was > >> meant > >> > >> for > >> > >> >> the > >> > >> >> > current node. > >> > >> >> > 3. The Lucene API enabled more flexibility and removed a > >> > >> >> > network > >> > >> >> > round trip from our queries. > >> > >> >> > 4. Given Calcite's ability to support custom SQL > >> > >> >> > functions, > >> > >> >> > I'd > >> > >> >> love > >> > >> >> > to have the ability to define custom functions that > >> > >> >> > Lucene > >> > was > >> > >> >> > answering > >> > >> >> > 7. What impact does RAFT now have on conflict resolution, > >> > >> >> > off > >> > the > >> > >> >> top of > >> > >> >> > my head there are two cases > >> > >> >> > 1. On startup after a split brain Ignite currently takes > >> > >> >> > an > >> > >> >> "exercise > >> > >> >> > for the reader" approach and dumps a log along the lines > >> > >> >> > of > >> > >> >> > > >> > >> >> > > 1. BaselineTopology of joining node is not compatible > with > >> > >> >> > > BaselineTopology in the cluster. > >> > >> >> > > 1. Branching history of cluster BlT doesn't contain > >> branching > >> > >> point > >> > >> >> > > hash of joining node BlT. Consider cleaning persistent > >> > >> >> > > storage > >> > >> >> of > >> > >> >> > the node > >> > >> >> > > and adding it to the cluster again. > >> > >> >> > > > >> > >> >> > 1. This leaves you with no choice except to take one half > >> > >> >> > and > >> > >> >> manually > >> > >> >> > copy, write data back over to the other half then > destroy > >> the > >> > >> bad > >> > >> >> > one. > >> > >> >> > 2. The second case is conflicts on keys, I > >> > >> >> > beleive CacheVersionConflictResolver and manager are > used > >> > >> >> > by GridCacheMapEntry which just says if use old value do > >> this > >> > >> >> > otherwise use > >> > >> >> > newVal. Ideally this will be exposed in the new API so > >> > >> >> > that > >> > >> >> > one > >> > >> >> can > >> > >> >> > override this behaviour. The last writer wins approach > >> isn't > >> > >> >> always > >> > >> >> > ideal > >> > >> >> > and the semantics of the domain can mean that what is > >> > consider > >> > >> >> > "correct" in > >> > >> >> > a conflict is not so for a different domain. > >> > >> >> > 8. This is last on the list but is actually the most > >> > >> >> > important > >> > >> >> > for > >> > >> us > >> > >> >> > right now as it is an impending and growing risk. We allow > >> > >> customers > >> > >> >> to > >> > >> >> > create their own tables on demand. We're already using the > >> same > >> > >> cache > >> > >> >> > group > >> > >> >> > etc for data structures to be re-used but now that we're > >> getting > >> > >> >> > to > >> > >> >> > thousands of tables/caches our startup times are sometimes > >> > >> >> unpredictably > >> > >> >> > long - at present it seems to depend on the state of the > >> > >> cache/table > >> > >> >> > before > >> > >> >> > the restart but we're into the order of 5 - 7 mins and > >> steadily > >> > >> >> > increasing > >> > >> >> > with the growth of tables. Are there any provisions in > >> > >> >> > Ignite > >> 3 > >> > >> >> > for > >> > >> >> > ensuring startup time isn't proportional to the number of > >> > >> >> tables/caches > >> > >> >> > available? > >> > >> >> > > >> > >> >> > > >> > >> >> > Those are the key things I can think of at the moment. Val and > >> > >> >> > others > >> > >> >> I'd > >> > >> >> > love to open a conversation around these. > >> > >> >> > > >> > >> >> > Regards, > >> > >> >> > Courtney Robinson > >> > >> >> > Founder and CEO, Hypi > >> > >> >> > Tel: ++44 208 123 2413 (GMT+0) <https://hypi.io> > >> > >> >> > > >> > >> >> > <https://hypi.io> > >> > >> >> > https://hypi.io > >> > >> >> > > >> > >> >> > >> > >> >> > >> > >> >> -- > >> > >> >> Best regards, > >> > >> >> Andrey V. Mashenkov > >> > >> >> > >> > >> > > >> > >> > >> > > > >> > > >> > > >> > -- > >> > > >> > Best regards, > >> > Ivan Pavlukhin > >> > > >> > > > > > -- > > Best regards, > Ivan Pavlukhin >
