> What is the issue if third-party plugins will create «ignite-sys-cache» from the code? Great idea!
On Fri, Dec 3, 2021 at 2:15 PM Nikolay Izhikov <nizhi...@apache.org> wrote: > Vyacheslav, Val, can you please clarify - What is the issue if third-party > plugins will create «ignite-sys-cache» from the code? > > Like just replacing `Ignite#cache` with the `Ignite#getOrCreateCache`. > > > 2 дек. 2021 г., в 16:13, Вячеслав Коптилин <slava.kopti...@gmail.com> > написал(а): > > > > Hello Maxim, > > > >> I don't understand why you are using *backwards compatibility* for > > completely internal things. > >> Why you are thinking in terms of users usage when are talking about > > ignite-sys-cache but not thinking when refactoring > > First of all, we are talking about all plugin developers. It does not > > matter it is open-source or proprietary plugins. > > Honestly, I don't have a list of all possible plugins that have already > > been developed and available under the ASF license, for instance. > > Do you have such a list? Can you be sure that this change will not affect > > anyone? > > > >> I don't understand why you are using *backwards compatibility* for > > completely internal things. > >> Why you are thinking in terms of users usage when are talking about > > ignite-sys-cache but not thinking when refactoring > > The system cache was the crucial thing in the architecture of Apache > Ignite > > 1.x and 2.x (at least 2.1 - 2.11?) > > > >> All the internals have been reworked and nobody even mentioned that it > > may affect some of the plugin developers. > > Yep, perhaps, some internal parts of Apache Ignite were reworked in order > > to avoid using the system cache. > > However, it is still a part of Ignite and it works and can be used in > > plugins. > > Honestly, the mentioned alternative, I mean the distributed metastorage, > is > > the INTERNAL thing as well. > > It means that plugin developer depends on INTERNAL entities. (it does not > > matter system-cache/metastorage) > > IMHO, such questions should be CAREFULLY discussed with no rush. > > > >> I do not propose to rush with the ignite-sys-cache removal. I'd like to > > collect all the technical stuff that we depend on, fix all of them and > > after everything will be ready do a final removal. > > Good! We are on the same page! > > > >> 1. Add deprecation for the 2.12 release. I hope it is still possible. > > If I am not mistaken, the code freeze was October 29. I think it is too > > late to introduce this change. We can add deprecation for the 2.13 > release. > > > >> Apache Ignite core still have some dependencies on ignite-sys-cache that > > should fix for 2.13 > > Ok, I agree. We need to try to find out all edge cases and add new > > abilities to the metastorage in order to cover all known > > issues/requirements. > > > >> Remove the system cache in 2.14. > > It depends on our progress with improving the distributed metastorage. In > > general, I hope it will be possible. > > > > Thanks, > > S. > > > > чт, 2 дек. 2021 г. в 13:36, Maxim Muzafarov <mmu...@apache.org>: > > > >> Pavel, > >> > >> > >> I don't understand why you are using *backwards compatibility* for > >> completely internal things. Why you are thinking in terms of users > >> usage when are talking about ignite-sys-cache but not thinking when > >> refactoring, for instance, all the checkpoint classes? Take a look at > >> the [1] issue. All the internals have been reworked and nobody even > >> mentioned that it may affect some of the plugin developers. This is > >> really strange, so I can't agree with you. > >> > >> > >> I do not propose to rush with the ignite-sys-cache removal. I'd like > >> to collect all the technical stuff that we depend on, fix all of them > >> and after everything will be ready do a final removal. Slava already > >> mentioned some crucial cases, so I hope you and Val also help with the > >> rest of them. Let's be more precise in terms *what we need to fix*. > >> > >> > >> I've made some investigations related to the ignite-sys-cache and here > >> is my proposal: > >> > >> 1. Add deprecation for the 2.12 release. I hope it is still possible. > >> > >> 2. Apache Ignite core still have some dependencies on ignite-sys-cache > >> that should fix for 2.13: > >> > >> 2.1. CLUSTER_NAME property: when it's not set the `deploymentId` of > >> system cache is used. Let's move it to metastorage. > >> 2.2. When the Security is enabled each compute task name (hash to be > >> more precise) is written to the ignite-sys-cache [2]. From my point of > >> view, we can remove it in 2.13. Can anyone check? > >> 2.3. Slava mentioned some issues with the distributed metastorage that > >> might be fixed. I think this is doable. > >> > >> 3. Remove the system cache in 2.14. > >> > >> > >> > >> [1] https://issues.apache.org/jira/browse/IGNITE-13143 > >> [2] > >> > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/task/GridTaskProcessor.java#L201 > >> > >> On Thu, 2 Dec 2021 at 12:56, Pavel Tupitsyn <ptupit...@apache.org> > wrote: > >>> > >>> Maxim, > >>> > >>>> I don't think that we should wait for 3-rd party plugins to be updated > >>>> this is an awful practice when the open-source product releases depend > >>>> on some of the proprietary plugins > >>> > >>> This makes no sense, sorry. > >>> > >>> It is not that we depend on 3rd party plugins. > >>> It is that *our users depend on us to preserve backwards > compatibility*. > >>> No one likes when their app breaks suddenly because of a library > update. > >>> > >>> We know about one use case for sys-cache in GridGain, but there may be > >>> more, no one knows. > >>> Every breaking change should be carried out carefully and gradually. > >>> > >>> > >>> On Thu, Dec 2, 2021 at 12:34 PM Maxim Muzafarov <mmu...@apache.org> > >> wrote: > >>> > >>>> Slava, > >>>> > >>>> Thank you for the comments. > >>>> > >>>>> Maxim, the community must provide a reasonable time interval in > >> order to > >>>> notify all contributors and wait for updating all 3-rd party plugins. > >>>> > >>>> This is not actually true. We must notify about changes as earlier as > >>>> possible and not only users but all the developers also. However, I > >>>> don't think that we should wait for 3-rd party plugins to be updated > >>>> this is an awful practice when the open-source product releases depend > >>>> on some of the proprietary plugins. There is no dependency between > >>>> Apache Ignite releases and proprietary plugin releases. If someone > >>>> desires to upgrade to a new version of Apache Ignite it can update its > >>>> plugins any time he likes. > >>>> > >>>>> We just need to provide a window that is enough to cover all related > >>>> issues. > >>>> > >>>> Can you share all the issues you know, please? > >>>> > >>>>> distributed meta storage does not provide the ability to atomically > >>>> change several properties at the same time (there are no transactions > >> on > >>>> this API). > >>>> > >>>> Can you share an example of what kind of properties we intend to > >>>> change at the same? Will it be enough to change them through a single > >>>> thread (e.g. the discovery thread or the exchange thread)? I agree > >>>> here that distributed metastorage should provide such an ability, > >>>> however, I can't find any real examples for Apache Ignite internal > >>>> needs. Please, share the details and let's fix it. > >>>> > >>>> On Wed, 1 Dec 2021 at 23:19, Valentin Kulichenko > >>>> <valentin.kuliche...@gmail.com> wrote: > >>>>> > >>>>> Agree with Slava. Two months is not enough time, especially > >> considering > >>>>> that the system cache is not functionally equivalent to the > >> metastorage. > >>>> I > >>>>> suggest we do the following: > >>>>> > >>>>> 1. Write down the differences between the system cache and the > >>>> metastorage, > >>>>> and provide a transition guide for plugin developers. > >>>>> 2. Deprecate the system cache in 2.13. > >>>>> 3. Remove the system cache in one of the further releases. I don't > >> think > >>>>> it's reasonable to do this earlier than mid next year (even that is > >>>>> potentially too early). > >>>>> > >>>>> Removing the system cache is the right move, but let's be more > >>>> considerate > >>>>> to the users. There is absolutely no need to rush. > >>>>> > >>>>> -Val > >>>>> > >>>>> On Wed, Dec 1, 2021 at 7:28 AM Вячеслав Коптилин < > >>>> slava.kopti...@gmail.com> > >>>>> wrote: > >>>>> > >>>>>> Hi Maxim, > >>>>>> > >>>>>>> On the other hand, I don't see any valuable reason why the change > >>>> can't > >>>>>> be done and we should wait for the future that never comes. > >>>>>> Maxim, the community must provide a reasonable time interval in > >> order > >>>> to > >>>>>> notify all contributors and wait for updating all 3-rd party > >> plugins. > >>>>>> Honestly, it does not seem to me that two, three months (the > >> current > >>>>>> timeline - end of March is the release date, so the end of Feb is > >> code > >>>>>> freeze) are quite enough. > >>>>>> > >>>>>>> I don't see any reasons why we should keep something in the CORE > >>>> module > >>>>>> that is being used at PLUGINS only. This is a lack of architecture > >>>> design > >>>>>> that must be fixed for sure; > >>>>>> I agree with you. We just need to provide a window that is enough > >> to > >>>> cover > >>>>>> all related issues. > >>>>>> For example, distributed meta storage does not provide the ability > >> to > >>>>>> atomically change several properties at the same time (there are no > >>>>>> transactions on this API). > >>>>>> > >>>>>> Perhaps, it would be nice to plan to remove the system cache on > >> 2.14 or > >>>>>> even 2.15. What do you think? > >>>>>> > >>>>>> Thanks, > >>>>>> S. > >>>>>> > >>>>>> > >>>>>> вт, 30 нояб. 2021 г. в 13:58, Maxim Muzafarov <mmu...@apache.org>: > >>>>>> > >>>>>>> Hello Val, > >>>>>>> > >>>>>>> Thank you for sharing the details. On the one hand, I agree with > >> you > >>>>>>> that there is no need to haste with this change and it must be > >>>>>>> prepared carefully. On the other hand, I don't see any valuable > >>>> reason > >>>>>>> why the change can't be done and we should wait for the future > >> that > >>>>>>> never comes. > >>>>>>> > >>>>>>> Please, consider the following my considerations: > >>>>>>> > >>>>>>> - The release 2.13 is not started yet. It will take months to be > >>>>>>> released (e.g. the end of March as a possible release date). The > >> time > >>>>>>> is more than enough; > >>>>>>> - The 2.13 release has breaking changes, so it is a good > >> opportunity > >>>>>>> to fix some gaps; > >>>>>>> - I don't see any reasons why we should keep something in the > >> CORE > >>>>>>> module that is being used at PLUGINS only. This is a lack of > >>>>>>> architecture design that must be fixed for sure; > >>>>>>> - The cache affects cluster stability and require additional > >> human > >>>>>>> resources for the code maintenance; > >>>>>>> - As a straightforward solution plugins can create their own > >> internal > >>>>>>> caches and use them ever they like (or use the metastorage as I > >>>>>>> mentioned earlier). Moving system cache to plugin doesn't look so > >>>>>>> complicated and harmful; > >>>>>>> > >>>>>>> On Mon, 29 Nov 2021 at 23:07, Valentin Kulichenko > >>>>>>> <valentin.kuliche...@gmail.com> wrote: > >>>>>>>> > >>>>>>>> Maxim, > >>>>>>>> > >>>>>>>> You're right that the system cache is still likely to be used > >> by > >>>>>> plugins. > >>>>>>>> We at GridGain use it for security features, for example. As > >> far > >>>> as I > >>>>>>> know, > >>>>>>>> that's not the only case. > >>>>>>>> > >>>>>>>> I also agree that the metastorage should be the preferred way > >> for > >>>> this > >>>>>>> kind > >>>>>>>> of purposes, but is there any harm in keeping the system cache > >> for > >>>> now? > >>>>>>>> Unlike the legacy Service Grid, this is not a public feature > >> that > >>>> can > >>>>>> be > >>>>>>>> used directly by a regular user, so I'm not sure why the rush. > >>>>>>>> > >>>>>>>> How about we instead deprecate the system cache, clearly > >> document > >>>> how > >>>>>> to > >>>>>>>> use the metastorage as an alternative, and then complete the > >>>> removal > >>>>>>>> sometime in the future? > >>>>>>>> > >>>>>>>> -Val > >>>>>>>> > >>>>>>>> On Sun, Nov 28, 2021 at 6:49 AM Maxim Muzafarov < > >> mmu...@apache.org > >>>>> > >>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Igniters, > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Since the legacy service grid implementation was removed [2] > >> I'd > >>>> like > >>>>>>>>> to remove the ignite-sys-cache also. It is still possible > >> that > >>>> some > >>>>>> of > >>>>>>>>> Ignite plugins (e.g. security plugin) are still using this > >> cache, > >>>>>>>>> however, AFAIK this is not the reason to keep the outdated > >> system > >>>>>>>>> cache and these plugins must be migrated to the metastorage > >>>> instead. > >>>>>>>>> > >>>>>>>>> I've filed the issue [1] for removal. Any objections? > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-16008 > >>>>>>>>> [2] https://issues.apache.org/jira/browse/IGNITE-15758 > >>>>>>>>> > >>>>>>> > >>>>>> > >>>> > >> > >
