Slava, I don’t get it. > Plugins have access to different internal Ignite components, such as security > processor and others, and can extend the programmatic API of Ignite.
Where docs say that we, as a community, take responsibility to keep internals in the way plugin expect? > 6 дек. 2021 г., в 14:48, Вячеслав Коптилин <slava.kopti...@gmail.com> > написал(а): > > Hello Nikolay, > >>> plugin framework allows to implement internal components and use the > internal API. >> Can you please, point out to documentation or place in Ignite code that > describe this behaviour? > > Looks like it states here https://ignite.apache.org/docs/latest/plugins > >> The Ignite plugin system allows you to extend the core functionality of >> Ignite. Plugins have access to different internal Ignite components, such >> as security processor and others, and can extend the programmatic API of >> Ignite. > > > Thanks, > S. > > > сб, 4 дек. 2021 г. в 14:54, Nikolay Izhikov <nizhi...@apache.org>: > >> Valentin >> >>> plugin framework allows to implement internal components and use the >> internal API. >> >> Can you please, point out to documentation or place in Ignite code that >> describe this behaviour? >> AFAIK plugin can only use public API, internal API can be changed any time >> we want. >> >>> Folks, can anyone explain the rush? >> >> No rush from my side. >> >> Just want to understand your vision of integration points between Ignite >> and plugins. >> >>> Is there any specific reason for it? >> >> Intention of IEP-80 is to improve Ignite codebase by removing abandoned >> features. >> >>> But the fact is that there are users that can depend on the system cache >> via the plugin framework. >> >> Why this dependency can’t be changed to any regular cache? >> Just replace `Ignite#cache` to `Ignite#getOrCreateCache` and that’s it. >> >> Do we have some special guarantees or logic besides system cache? >> >> >>> 4 дек. 2021 г., в 02:14, Valentin Kulichenko < >> valentin.kuliche...@gmail.com> написал(а): >>> >>> Nikolay, that's right, plugin framework allows to implement internal >>> components and use the internal API. There is a difference between a >> plugin >>> and an extension that uses only public API >>> >>> Folks, can anyone explain the rush? Is there any specific reason for it? >>> >>> I think we all agree that this is a good change - system cache has to >> go. I >>> guess technically it's not even a breaking change, because system cache >> is >>> not a public API feature. But the fact is that there are users that can >>> depend on the system cache via the plugin framework. Let's be respectful >> to >>> those users, provide a reasonable documented alternative and give time. >>> >>> -Val >>> >>> On Fri, Dec 3, 2021 at 8:18 AM Nikolay Izhikov <nizhi...@apache.org> >> wrote: >>> >>>> I don’t understand how it possible. >>>> >>>> Are you talking about plugin that uses some kind of internal API? >>>> >>>>> 3 дек. 2021 г., в 18:50, Вячеслав Коптилин <slava.kopti...@gmail.com> >>>> написал(а): >>>>> >>>>> Hello Nikolay, >>>>> >>>>> If I am not mistaken, the method you mentioned, allows to create a >> "user" >>>>> cache that is available through public api for the user. This does not >>>>> cover the case when the plugin developer wants to hide such cache and >>>>> protect it form the end user (at least, via public api). >>>>> >>>>> Thanks, >>>>> S. >>>>> >>>>> пт, 3 дек. 2021 г., 14:15 Nikolay Izhikov <nizhi...@apache.org>: >>>>> >>>>>> Vyacheslav, Val, can you please clarify - What is the issue if >>>> third-party >>>>>> plugins will create «ignite-sys-cache» from the code? >>>>>> >>>>>> Like just replacing `Ignite#cache` with the `Ignite#getOrCreateCache`. >>>>>> >>>>>>> 2 дек. 2021 г., в 16:13, Вячеслав Коптилин <slava.kopti...@gmail.com >>> >>>>>> написал(а): >>>>>>> >>>>>>> Hello Maxim, >>>>>>> >>>>>>>> I don't understand why you are using *backwards compatibility* for >>>>>>> completely internal things. >>>>>>>> Why you are thinking in terms of users usage when are talking about >>>>>>> ignite-sys-cache but not thinking when refactoring >>>>>>> First of all, we are talking about all plugin developers. It does not >>>>>>> matter it is open-source or proprietary plugins. >>>>>>> Honestly, I don't have a list of all possible plugins that have >> already >>>>>>> been developed and available under the ASF license, for instance. >>>>>>> Do you have such a list? Can you be sure that this change will not >>>> affect >>>>>>> anyone? >>>>>>> >>>>>>>> I don't understand why you are using *backwards compatibility* for >>>>>>> completely internal things. >>>>>>>> Why you are thinking in terms of users usage when are talking about >>>>>>> ignite-sys-cache but not thinking when refactoring >>>>>>> The system cache was the crucial thing in the architecture of Apache >>>>>> Ignite >>>>>>> 1.x and 2.x (at least 2.1 - 2.11?) >>>>>>> >>>>>>>> All the internals have been reworked and nobody even mentioned that >> it >>>>>>> may affect some of the plugin developers. >>>>>>> Yep, perhaps, some internal parts of Apache Ignite were reworked in >>>> order >>>>>>> to avoid using the system cache. >>>>>>> However, it is still a part of Ignite and it works and can be used in >>>>>>> plugins. >>>>>>> Honestly, the mentioned alternative, I mean the distributed >>>> metastorage, >>>>>> is >>>>>>> the INTERNAL thing as well. >>>>>>> It means that plugin developer depends on INTERNAL entities. (it does >>>> not >>>>>>> matter system-cache/metastorage) >>>>>>> IMHO, such questions should be CAREFULLY discussed with no rush. >>>>>>> >>>>>>>> I do not propose to rush with the ignite-sys-cache removal. I'd like >>>> to >>>>>>> collect all the technical stuff that we depend on, fix all of them >> and >>>>>>> after everything will be ready do a final removal. >>>>>>> Good! We are on the same page! >>>>>>> >>>>>>>> 1. Add deprecation for the 2.12 release. I hope it is still >> possible. >>>>>>> If I am not mistaken, the code freeze was October 29. I think it is >> too >>>>>>> late to introduce this change. We can add deprecation for the 2.13 >>>>>> release. >>>>>>> >>>>>>>> Apache Ignite core still have some dependencies on ignite-sys-cache >>>> that >>>>>>> should fix for 2.13 >>>>>>> Ok, I agree. We need to try to find out all edge cases and add new >>>>>>> abilities to the metastorage in order to cover all known >>>>>>> issues/requirements. >>>>>>> >>>>>>>> Remove the system cache in 2.14. >>>>>>> It depends on our progress with improving the distributed >> metastorage. >>>> In >>>>>>> general, I hope it will be possible. >>>>>>> >>>>>>> Thanks, >>>>>>> S. >>>>>>> >>>>>>> чт, 2 дек. 2021 г. в 13:36, Maxim Muzafarov <mmu...@apache.org>: >>>>>>> >>>>>>>> Pavel, >>>>>>>> >>>>>>>> >>>>>>>> I don't understand why you are using *backwards compatibility* for >>>>>>>> completely internal things. Why you are thinking in terms of users >>>>>>>> usage when are talking about ignite-sys-cache but not thinking when >>>>>>>> refactoring, for instance, all the checkpoint classes? Take a look >> at >>>>>>>> the [1] issue. All the internals have been reworked and nobody even >>>>>>>> mentioned that it may affect some of the plugin developers. This is >>>>>>>> really strange, so I can't agree with you. >>>>>>>> >>>>>>>> >>>>>>>> I do not propose to rush with the ignite-sys-cache removal. I'd like >>>>>>>> to collect all the technical stuff that we depend on, fix all of >> them >>>>>>>> and after everything will be ready do a final removal. Slava already >>>>>>>> mentioned some crucial cases, so I hope you and Val also help with >> the >>>>>>>> rest of them. Let's be more precise in terms *what we need to fix*. >>>>>>>> >>>>>>>> >>>>>>>> I've made some investigations related to the ignite-sys-cache and >> here >>>>>>>> is my proposal: >>>>>>>> >>>>>>>> 1. Add deprecation for the 2.12 release. I hope it is still >> possible. >>>>>>>> >>>>>>>> 2. Apache Ignite core still have some dependencies on >> ignite-sys-cache >>>>>>>> that should fix for 2.13: >>>>>>>> >>>>>>>> 2.1. CLUSTER_NAME property: when it's not set the `deploymentId` of >>>>>>>> system cache is used. Let's move it to metastorage. >>>>>>>> 2.2. When the Security is enabled each compute task name (hash to be >>>>>>>> more precise) is written to the ignite-sys-cache [2]. From my point >> of >>>>>>>> view, we can remove it in 2.13. Can anyone check? >>>>>>>> 2.3. Slava mentioned some issues with the distributed metastorage >> that >>>>>>>> might be fixed. I think this is doable. >>>>>>>> >>>>>>>> 3. Remove the system cache in 2.14. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13143 >>>>>>>> [2] >>>>>>>> >>>>>> >>>> >> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/task/GridTaskProcessor.java#L201 >>>>>>>> >>>>>>>> On Thu, 2 Dec 2021 at 12:56, Pavel Tupitsyn <ptupit...@apache.org> >>>>>> wrote: >>>>>>>>> >>>>>>>>> Maxim, >>>>>>>>> >>>>>>>>>> I don't think that we should wait for 3-rd party plugins to be >>>> updated >>>>>>>>>> this is an awful practice when the open-source product releases >>>> depend >>>>>>>>>> on some of the proprietary plugins >>>>>>>>> >>>>>>>>> This makes no sense, sorry. >>>>>>>>> >>>>>>>>> It is not that we depend on 3rd party plugins. >>>>>>>>> It is that *our users depend on us to preserve backwards >>>>>> compatibility*. >>>>>>>>> No one likes when their app breaks suddenly because of a library >>>>>> update. >>>>>>>>> >>>>>>>>> We know about one use case for sys-cache in GridGain, but there may >>>> be >>>>>>>>> more, no one knows. >>>>>>>>> Every breaking change should be carried out carefully and >> gradually. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Dec 2, 2021 at 12:34 PM Maxim Muzafarov <mmu...@apache.org >>> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Slava, >>>>>>>>>> >>>>>>>>>> Thank you for the comments. >>>>>>>>>> >>>>>>>>>>> Maxim, the community must provide a reasonable time interval in >>>>>>>> order to >>>>>>>>>> notify all contributors and wait for updating all 3-rd party >>>> plugins. >>>>>>>>>> >>>>>>>>>> This is not actually true. We must notify about changes as earlier >>>> as >>>>>>>>>> possible and not only users but all the developers also. However, >> I >>>>>>>>>> don't think that we should wait for 3-rd party plugins to be >> updated >>>>>>>>>> this is an awful practice when the open-source product releases >>>> depend >>>>>>>>>> on some of the proprietary plugins. There is no dependency between >>>>>>>>>> Apache Ignite releases and proprietary plugin releases. If someone >>>>>>>>>> desires to upgrade to a new version of Apache Ignite it can update >>>> its >>>>>>>>>> plugins any time he likes. >>>>>>>>>> >>>>>>>>>>> We just need to provide a window that is enough to cover all >>>> related >>>>>>>>>> issues. >>>>>>>>>> >>>>>>>>>> Can you share all the issues you know, please? >>>>>>>>>> >>>>>>>>>>> distributed meta storage does not provide the ability to >> atomically >>>>>>>>>> change several properties at the same time (there are no >>>> transactions >>>>>>>> on >>>>>>>>>> this API). >>>>>>>>>> >>>>>>>>>> Can you share an example of what kind of properties we intend to >>>>>>>>>> change at the same? Will it be enough to change them through a >>>> single >>>>>>>>>> thread (e.g. the discovery thread or the exchange thread)? I agree >>>>>>>>>> here that distributed metastorage should provide such an ability, >>>>>>>>>> however, I can't find any real examples for Apache Ignite internal >>>>>>>>>> needs. Please, share the details and let's fix it. >>>>>>>>>> >>>>>>>>>> On Wed, 1 Dec 2021 at 23:19, Valentin Kulichenko >>>>>>>>>> <valentin.kuliche...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>> Agree with Slava. Two months is not enough time, especially >>>>>>>> considering >>>>>>>>>>> that the system cache is not functionally equivalent to the >>>>>>>> metastorage. >>>>>>>>>> I >>>>>>>>>>> suggest we do the following: >>>>>>>>>>> >>>>>>>>>>> 1. Write down the differences between the system cache and the >>>>>>>>>> metastorage, >>>>>>>>>>> and provide a transition guide for plugin developers. >>>>>>>>>>> 2. Deprecate the system cache in 2.13. >>>>>>>>>>> 3. Remove the system cache in one of the further releases. I >> don't >>>>>>>> think >>>>>>>>>>> it's reasonable to do this earlier than mid next year (even that >> is >>>>>>>>>>> potentially too early). >>>>>>>>>>> >>>>>>>>>>> Removing the system cache is the right move, but let's be more >>>>>>>>>> considerate >>>>>>>>>>> to the users. There is absolutely no need to rush. >>>>>>>>>>> >>>>>>>>>>> -Val >>>>>>>>>>> >>>>>>>>>>> On Wed, Dec 1, 2021 at 7:28 AM Вячеслав Коптилин < >>>>>>>>>> slava.kopti...@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Maxim, >>>>>>>>>>>> >>>>>>>>>>>>> On the other hand, I don't see any valuable reason why the >> change >>>>>>>>>> can't >>>>>>>>>>>> be done and we should wait for the future that never comes. >>>>>>>>>>>> Maxim, the community must provide a reasonable time interval in >>>>>>>> order >>>>>>>>>> to >>>>>>>>>>>> notify all contributors and wait for updating all 3-rd party >>>>>>>> plugins. >>>>>>>>>>>> Honestly, it does not seem to me that two, three months (the >>>>>>>> current >>>>>>>>>>>> timeline - end of March is the release date, so the end of Feb >> is >>>>>>>> code >>>>>>>>>>>> freeze) are quite enough. >>>>>>>>>>>> >>>>>>>>>>>>> I don't see any reasons why we should keep something in the >> CORE >>>>>>>>>> module >>>>>>>>>>>> that is being used at PLUGINS only. This is a lack of >> architecture >>>>>>>>>> design >>>>>>>>>>>> that must be fixed for sure; >>>>>>>>>>>> I agree with you. We just need to provide a window that is >> enough >>>>>>>> to >>>>>>>>>> cover >>>>>>>>>>>> all related issues. >>>>>>>>>>>> For example, distributed meta storage does not provide the >> ability >>>>>>>> to >>>>>>>>>>>> atomically change several properties at the same time (there are >>>> no >>>>>>>>>>>> transactions on this API). >>>>>>>>>>>> >>>>>>>>>>>> Perhaps, it would be nice to plan to remove the system cache on >>>>>>>> 2.14 or >>>>>>>>>>>> even 2.15. What do you think? >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> S. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> вт, 30 нояб. 2021 г. в 13:58, Maxim Muzafarov < >> mmu...@apache.org >>>>> : >>>>>>>>>>>> >>>>>>>>>>>>> Hello Val, >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you for sharing the details. On the one hand, I agree >> with >>>>>>>> you >>>>>>>>>>>>> that there is no need to haste with this change and it must be >>>>>>>>>>>>> prepared carefully. On the other hand, I don't see any valuable >>>>>>>>>> reason >>>>>>>>>>>>> why the change can't be done and we should wait for the future >>>>>>>> that >>>>>>>>>>>>> never comes. >>>>>>>>>>>>> >>>>>>>>>>>>> Please, consider the following my considerations: >>>>>>>>>>>>> >>>>>>>>>>>>> - The release 2.13 is not started yet. It will take months to >> be >>>>>>>>>>>>> released (e.g. the end of March as a possible release date). >> The >>>>>>>> time >>>>>>>>>>>>> is more than enough; >>>>>>>>>>>>> - The 2.13 release has breaking changes, so it is a good >>>>>>>> opportunity >>>>>>>>>>>>> to fix some gaps; >>>>>>>>>>>>> - I don't see any reasons why we should keep something in the >>>>>>>> CORE >>>>>>>>>>>>> module that is being used at PLUGINS only. This is a lack of >>>>>>>>>>>>> architecture design that must be fixed for sure; >>>>>>>>>>>>> - The cache affects cluster stability and require additional >>>>>>>> human >>>>>>>>>>>>> resources for the code maintenance; >>>>>>>>>>>>> - As a straightforward solution plugins can create their own >>>>>>>> internal >>>>>>>>>>>>> caches and use them ever they like (or use the metastorage as I >>>>>>>>>>>>> mentioned earlier). Moving system cache to plugin doesn't look >> so >>>>>>>>>>>>> complicated and harmful; >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, 29 Nov 2021 at 23:07, Valentin Kulichenko >>>>>>>>>>>>> <valentin.kuliche...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Maxim, >>>>>>>>>>>>>> >>>>>>>>>>>>>> You're right that the system cache is still likely to be used >>>>>>>> by >>>>>>>>>>>> plugins. >>>>>>>>>>>>>> We at GridGain use it for security features, for example. As >>>>>>>> far >>>>>>>>>> as I >>>>>>>>>>>>> know, >>>>>>>>>>>>>> that's not the only case. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I also agree that the metastorage should be the preferred way >>>>>>>> for >>>>>>>>>> this >>>>>>>>>>>>> kind >>>>>>>>>>>>>> of purposes, but is there any harm in keeping the system cache >>>>>>>> for >>>>>>>>>> now? >>>>>>>>>>>>>> Unlike the legacy Service Grid, this is not a public feature >>>>>>>> that >>>>>>>>>> can >>>>>>>>>>>> be >>>>>>>>>>>>>> used directly by a regular user, so I'm not sure why the rush. >>>>>>>>>>>>>> >>>>>>>>>>>>>> How about we instead deprecate the system cache, clearly >>>>>>>> document >>>>>>>>>> how >>>>>>>>>>>> to >>>>>>>>>>>>>> use the metastorage as an alternative, and then complete the >>>>>>>>>> removal >>>>>>>>>>>>>> sometime in the future? >>>>>>>>>>>>>> >>>>>>>>>>>>>> -Val >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sun, Nov 28, 2021 at 6:49 AM Maxim Muzafarov < >>>>>>>> mmu...@apache.org >>>>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Igniters, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Since the legacy service grid implementation was removed [2] >>>>>>>> I'd >>>>>>>>>> like >>>>>>>>>>>>>>> to remove the ignite-sys-cache also. It is still possible >>>>>>>> that >>>>>>>>>> some >>>>>>>>>>>> of >>>>>>>>>>>>>>> Ignite plugins (e.g. security plugin) are still using this >>>>>>>> cache, >>>>>>>>>>>>>>> however, AFAIK this is not the reason to keep the outdated >>>>>>>> system >>>>>>>>>>>>>>> cache and these plugins must be migrated to the metastorage >>>>>>>>>> instead. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I've filed the issue [1] for removal. Any objections? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-16008 >>>>>>>>>>>>>>> [2] https://issues.apache.org/jira/browse/IGNITE-15758 >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>> >>>> >> >>
