Valentin > plugin framework allows to implement internal components and use the > internal API.
Can you please, point out to documentation or place in Ignite code that describe this behaviour? AFAIK plugin can only use public API, internal API can be changed any time we want. > Folks, can anyone explain the rush? No rush from my side. Just want to understand your vision of integration points between Ignite and plugins. > Is there any specific reason for it? Intention of IEP-80 is to improve Ignite codebase by removing abandoned features. > But the fact is that there are users that can depend on the system cache via > the plugin framework. Why this dependency can’t be changed to any regular cache? Just replace `Ignite#cache` to `Ignite#getOrCreateCache` and that’s it. Do we have some special guarantees or logic besides system cache? > 4 дек. 2021 г., в 02:14, Valentin Kulichenko <valentin.kuliche...@gmail.com> > написал(а): > > Nikolay, that's right, plugin framework allows to implement internal > components and use the internal API. There is a difference between a plugin > and an extension that uses only public API > > Folks, can anyone explain the rush? Is there any specific reason for it? > > I think we all agree that this is a good change - system cache has to go. I > guess technically it's not even a breaking change, because system cache is > not a public API feature. But the fact is that there are users that can > depend on the system cache via the plugin framework. Let's be respectful to > those users, provide a reasonable documented alternative and give time. > > -Val > > On Fri, Dec 3, 2021 at 8:18 AM Nikolay Izhikov <nizhi...@apache.org> wrote: > >> I don’t understand how it possible. >> >> Are you talking about plugin that uses some kind of internal API? >> >>> 3 дек. 2021 г., в 18:50, Вячеслав Коптилин <slava.kopti...@gmail.com> >> написал(а): >>> >>> Hello Nikolay, >>> >>> If I am not mistaken, the method you mentioned, allows to create a "user" >>> cache that is available through public api for the user. This does not >>> cover the case when the plugin developer wants to hide such cache and >>> protect it form the end user (at least, via public api). >>> >>> Thanks, >>> S. >>> >>> пт, 3 дек. 2021 г., 14:15 Nikolay Izhikov <nizhi...@apache.org>: >>> >>>> Vyacheslav, Val, can you please clarify - What is the issue if >> third-party >>>> plugins will create «ignite-sys-cache» from the code? >>>> >>>> Like just replacing `Ignite#cache` with the `Ignite#getOrCreateCache`. >>>> >>>>> 2 дек. 2021 г., в 16:13, Вячеслав Коптилин <slava.kopti...@gmail.com> >>>> написал(а): >>>>> >>>>> Hello Maxim, >>>>> >>>>>> I don't understand why you are using *backwards compatibility* for >>>>> completely internal things. >>>>>> Why you are thinking in terms of users usage when are talking about >>>>> ignite-sys-cache but not thinking when refactoring >>>>> First of all, we are talking about all plugin developers. It does not >>>>> matter it is open-source or proprietary plugins. >>>>> Honestly, I don't have a list of all possible plugins that have already >>>>> been developed and available under the ASF license, for instance. >>>>> Do you have such a list? Can you be sure that this change will not >> affect >>>>> anyone? >>>>> >>>>>> I don't understand why you are using *backwards compatibility* for >>>>> completely internal things. >>>>>> Why you are thinking in terms of users usage when are talking about >>>>> ignite-sys-cache but not thinking when refactoring >>>>> The system cache was the crucial thing in the architecture of Apache >>>> Ignite >>>>> 1.x and 2.x (at least 2.1 - 2.11?) >>>>> >>>>>> All the internals have been reworked and nobody even mentioned that it >>>>> may affect some of the plugin developers. >>>>> Yep, perhaps, some internal parts of Apache Ignite were reworked in >> order >>>>> to avoid using the system cache. >>>>> However, it is still a part of Ignite and it works and can be used in >>>>> plugins. >>>>> Honestly, the mentioned alternative, I mean the distributed >> metastorage, >>>> is >>>>> the INTERNAL thing as well. >>>>> It means that plugin developer depends on INTERNAL entities. (it does >> not >>>>> matter system-cache/metastorage) >>>>> IMHO, such questions should be CAREFULLY discussed with no rush. >>>>> >>>>>> I do not propose to rush with the ignite-sys-cache removal. I'd like >> to >>>>> collect all the technical stuff that we depend on, fix all of them and >>>>> after everything will be ready do a final removal. >>>>> Good! We are on the same page! >>>>> >>>>>> 1. Add deprecation for the 2.12 release. I hope it is still possible. >>>>> If I am not mistaken, the code freeze was October 29. I think it is too >>>>> late to introduce this change. We can add deprecation for the 2.13 >>>> release. >>>>> >>>>>> Apache Ignite core still have some dependencies on ignite-sys-cache >> that >>>>> should fix for 2.13 >>>>> Ok, I agree. We need to try to find out all edge cases and add new >>>>> abilities to the metastorage in order to cover all known >>>>> issues/requirements. >>>>> >>>>>> Remove the system cache in 2.14. >>>>> It depends on our progress with improving the distributed metastorage. >> In >>>>> general, I hope it will be possible. >>>>> >>>>> Thanks, >>>>> S. >>>>> >>>>> чт, 2 дек. 2021 г. в 13:36, Maxim Muzafarov <mmu...@apache.org>: >>>>> >>>>>> Pavel, >>>>>> >>>>>> >>>>>> I don't understand why you are using *backwards compatibility* for >>>>>> completely internal things. Why you are thinking in terms of users >>>>>> usage when are talking about ignite-sys-cache but not thinking when >>>>>> refactoring, for instance, all the checkpoint classes? Take a look at >>>>>> the [1] issue. All the internals have been reworked and nobody even >>>>>> mentioned that it may affect some of the plugin developers. This is >>>>>> really strange, so I can't agree with you. >>>>>> >>>>>> >>>>>> I do not propose to rush with the ignite-sys-cache removal. I'd like >>>>>> to collect all the technical stuff that we depend on, fix all of them >>>>>> and after everything will be ready do a final removal. Slava already >>>>>> mentioned some crucial cases, so I hope you and Val also help with the >>>>>> rest of them. Let's be more precise in terms *what we need to fix*. >>>>>> >>>>>> >>>>>> I've made some investigations related to the ignite-sys-cache and here >>>>>> is my proposal: >>>>>> >>>>>> 1. Add deprecation for the 2.12 release. I hope it is still possible. >>>>>> >>>>>> 2. Apache Ignite core still have some dependencies on ignite-sys-cache >>>>>> that should fix for 2.13: >>>>>> >>>>>> 2.1. CLUSTER_NAME property: when it's not set the `deploymentId` of >>>>>> system cache is used. Let's move it to metastorage. >>>>>> 2.2. When the Security is enabled each compute task name (hash to be >>>>>> more precise) is written to the ignite-sys-cache [2]. From my point of >>>>>> view, we can remove it in 2.13. Can anyone check? >>>>>> 2.3. Slava mentioned some issues with the distributed metastorage that >>>>>> might be fixed. I think this is doable. >>>>>> >>>>>> 3. Remove the system cache in 2.14. >>>>>> >>>>>> >>>>>> >>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13143 >>>>>> [2] >>>>>> >>>> >> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/task/GridTaskProcessor.java#L201 >>>>>> >>>>>> On Thu, 2 Dec 2021 at 12:56, Pavel Tupitsyn <ptupit...@apache.org> >>>> wrote: >>>>>>> >>>>>>> Maxim, >>>>>>> >>>>>>>> I don't think that we should wait for 3-rd party plugins to be >> updated >>>>>>>> this is an awful practice when the open-source product releases >> depend >>>>>>>> on some of the proprietary plugins >>>>>>> >>>>>>> This makes no sense, sorry. >>>>>>> >>>>>>> It is not that we depend on 3rd party plugins. >>>>>>> It is that *our users depend on us to preserve backwards >>>> compatibility*. >>>>>>> No one likes when their app breaks suddenly because of a library >>>> update. >>>>>>> >>>>>>> We know about one use case for sys-cache in GridGain, but there may >> be >>>>>>> more, no one knows. >>>>>>> Every breaking change should be carried out carefully and gradually. >>>>>>> >>>>>>> >>>>>>> On Thu, Dec 2, 2021 at 12:34 PM Maxim Muzafarov <mmu...@apache.org> >>>>>> wrote: >>>>>>> >>>>>>>> Slava, >>>>>>>> >>>>>>>> Thank you for the comments. >>>>>>>> >>>>>>>>> Maxim, the community must provide a reasonable time interval in >>>>>> order to >>>>>>>> notify all contributors and wait for updating all 3-rd party >> plugins. >>>>>>>> >>>>>>>> This is not actually true. We must notify about changes as earlier >> as >>>>>>>> possible and not only users but all the developers also. However, I >>>>>>>> don't think that we should wait for 3-rd party plugins to be updated >>>>>>>> this is an awful practice when the open-source product releases >> depend >>>>>>>> on some of the proprietary plugins. There is no dependency between >>>>>>>> Apache Ignite releases and proprietary plugin releases. If someone >>>>>>>> desires to upgrade to a new version of Apache Ignite it can update >> its >>>>>>>> plugins any time he likes. >>>>>>>> >>>>>>>>> We just need to provide a window that is enough to cover all >> related >>>>>>>> issues. >>>>>>>> >>>>>>>> Can you share all the issues you know, please? >>>>>>>> >>>>>>>>> distributed meta storage does not provide the ability to atomically >>>>>>>> change several properties at the same time (there are no >> transactions >>>>>> on >>>>>>>> this API). >>>>>>>> >>>>>>>> Can you share an example of what kind of properties we intend to >>>>>>>> change at the same? Will it be enough to change them through a >> single >>>>>>>> thread (e.g. the discovery thread or the exchange thread)? I agree >>>>>>>> here that distributed metastorage should provide such an ability, >>>>>>>> however, I can't find any real examples for Apache Ignite internal >>>>>>>> needs. Please, share the details and let's fix it. >>>>>>>> >>>>>>>> On Wed, 1 Dec 2021 at 23:19, Valentin Kulichenko >>>>>>>> <valentin.kuliche...@gmail.com> wrote: >>>>>>>>> >>>>>>>>> Agree with Slava. Two months is not enough time, especially >>>>>> considering >>>>>>>>> that the system cache is not functionally equivalent to the >>>>>> metastorage. >>>>>>>> I >>>>>>>>> suggest we do the following: >>>>>>>>> >>>>>>>>> 1. Write down the differences between the system cache and the >>>>>>>> metastorage, >>>>>>>>> and provide a transition guide for plugin developers. >>>>>>>>> 2. Deprecate the system cache in 2.13. >>>>>>>>> 3. Remove the system cache in one of the further releases. I don't >>>>>> think >>>>>>>>> it's reasonable to do this earlier than mid next year (even that is >>>>>>>>> potentially too early). >>>>>>>>> >>>>>>>>> Removing the system cache is the right move, but let's be more >>>>>>>> considerate >>>>>>>>> to the users. There is absolutely no need to rush. >>>>>>>>> >>>>>>>>> -Val >>>>>>>>> >>>>>>>>> On Wed, Dec 1, 2021 at 7:28 AM Вячеслав Коптилин < >>>>>>>> slava.kopti...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Maxim, >>>>>>>>>> >>>>>>>>>>> On the other hand, I don't see any valuable reason why the change >>>>>>>> can't >>>>>>>>>> be done and we should wait for the future that never comes. >>>>>>>>>> Maxim, the community must provide a reasonable time interval in >>>>>> order >>>>>>>> to >>>>>>>>>> notify all contributors and wait for updating all 3-rd party >>>>>> plugins. >>>>>>>>>> Honestly, it does not seem to me that two, three months (the >>>>>> current >>>>>>>>>> timeline - end of March is the release date, so the end of Feb is >>>>>> code >>>>>>>>>> freeze) are quite enough. >>>>>>>>>> >>>>>>>>>>> I don't see any reasons why we should keep something in the CORE >>>>>>>> module >>>>>>>>>> that is being used at PLUGINS only. This is a lack of architecture >>>>>>>> design >>>>>>>>>> that must be fixed for sure; >>>>>>>>>> I agree with you. We just need to provide a window that is enough >>>>>> to >>>>>>>> cover >>>>>>>>>> all related issues. >>>>>>>>>> For example, distributed meta storage does not provide the ability >>>>>> to >>>>>>>>>> atomically change several properties at the same time (there are >> no >>>>>>>>>> transactions on this API). >>>>>>>>>> >>>>>>>>>> Perhaps, it would be nice to plan to remove the system cache on >>>>>> 2.14 or >>>>>>>>>> even 2.15. What do you think? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> S. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> вт, 30 нояб. 2021 г. в 13:58, Maxim Muzafarov <mmu...@apache.org >>> : >>>>>>>>>> >>>>>>>>>>> Hello Val, >>>>>>>>>>> >>>>>>>>>>> Thank you for sharing the details. On the one hand, I agree with >>>>>> you >>>>>>>>>>> that there is no need to haste with this change and it must be >>>>>>>>>>> prepared carefully. On the other hand, I don't see any valuable >>>>>>>> reason >>>>>>>>>>> why the change can't be done and we should wait for the future >>>>>> that >>>>>>>>>>> never comes. >>>>>>>>>>> >>>>>>>>>>> Please, consider the following my considerations: >>>>>>>>>>> >>>>>>>>>>> - The release 2.13 is not started yet. It will take months to be >>>>>>>>>>> released (e.g. the end of March as a possible release date). The >>>>>> time >>>>>>>>>>> is more than enough; >>>>>>>>>>> - The 2.13 release has breaking changes, so it is a good >>>>>> opportunity >>>>>>>>>>> to fix some gaps; >>>>>>>>>>> - I don't see any reasons why we should keep something in the >>>>>> CORE >>>>>>>>>>> module that is being used at PLUGINS only. This is a lack of >>>>>>>>>>> architecture design that must be fixed for sure; >>>>>>>>>>> - The cache affects cluster stability and require additional >>>>>> human >>>>>>>>>>> resources for the code maintenance; >>>>>>>>>>> - As a straightforward solution plugins can create their own >>>>>> internal >>>>>>>>>>> caches and use them ever they like (or use the metastorage as I >>>>>>>>>>> mentioned earlier). Moving system cache to plugin doesn't look so >>>>>>>>>>> complicated and harmful; >>>>>>>>>>> >>>>>>>>>>> On Mon, 29 Nov 2021 at 23:07, Valentin Kulichenko >>>>>>>>>>> <valentin.kuliche...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Maxim, >>>>>>>>>>>> >>>>>>>>>>>> You're right that the system cache is still likely to be used >>>>>> by >>>>>>>>>> plugins. >>>>>>>>>>>> We at GridGain use it for security features, for example. As >>>>>> far >>>>>>>> as I >>>>>>>>>>> know, >>>>>>>>>>>> that's not the only case. >>>>>>>>>>>> >>>>>>>>>>>> I also agree that the metastorage should be the preferred way >>>>>> for >>>>>>>> this >>>>>>>>>>> kind >>>>>>>>>>>> of purposes, but is there any harm in keeping the system cache >>>>>> for >>>>>>>> now? >>>>>>>>>>>> Unlike the legacy Service Grid, this is not a public feature >>>>>> that >>>>>>>> can >>>>>>>>>> be >>>>>>>>>>>> used directly by a regular user, so I'm not sure why the rush. >>>>>>>>>>>> >>>>>>>>>>>> How about we instead deprecate the system cache, clearly >>>>>> document >>>>>>>> how >>>>>>>>>> to >>>>>>>>>>>> use the metastorage as an alternative, and then complete the >>>>>>>> removal >>>>>>>>>>>> sometime in the future? >>>>>>>>>>>> >>>>>>>>>>>> -Val >>>>>>>>>>>> >>>>>>>>>>>> On Sun, Nov 28, 2021 at 6:49 AM Maxim Muzafarov < >>>>>> mmu...@apache.org >>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Igniters, >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Since the legacy service grid implementation was removed [2] >>>>>> I'd >>>>>>>> like >>>>>>>>>>>>> to remove the ignite-sys-cache also. It is still possible >>>>>> that >>>>>>>> some >>>>>>>>>> of >>>>>>>>>>>>> Ignite plugins (e.g. security plugin) are still using this >>>>>> cache, >>>>>>>>>>>>> however, AFAIK this is not the reason to keep the outdated >>>>>> system >>>>>>>>>>>>> cache and these plugins must be migrated to the metastorage >>>>>>>> instead. >>>>>>>>>>>>> >>>>>>>>>>>>> I've filed the issue [1] for removal. Any objections? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-16008 >>>>>>>>>>>>> [2] https://issues.apache.org/jira/browse/IGNITE-15758 >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>> >>>> >>>> >> >>
