Re: Apache Ignite 2.12 RELEASE [Time, Scope, Manager]
Hello, I want to include the issue [1] to the 2.12 scope. It fixes some metrics according to the JSR107. Any objections? [1] https://issues.apache.org/jira/browse/IGNITE-16002 The remove cache method should update statistics if the method returns true чт, 25 нояб. 2021 г. в 17:12, Nikita Amelchev : > > Hello Ivan, > > +1, the issue can be cherry-picked to the 2.12. > > чт, 25 нояб. 2021 г. в 17:07, Ivan Bessonov : > > > > Hello everyone, > > > > is there a chance to add this issue to the release scope? [1] > > Currently, defragmentation of certain types of indexes can corrupt them. > > Fix is straightforward and should not cause any problems. > > Thank you! > > > > [1] https://issues.apache.org/jira/browse/IGNITE-15968 > > > > вт, 23 нояб. 2021 г. в 20:44, Nikita Amelchev : > > > > > Ivan, > > > > > > I have cherry-picked the issue: > > > https://issues.apache.org/jira/browse/IGNITE-15767 > > > > > > вт, 23 нояб. 2021 г. в 20:31, Nikolay Izhikov : > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-15951 > > > > > > > > Cherry-picked to 2.12 > > > > > > > > > 23 нояб. 2021 г., в 16:38, Nikita Amelchev > > > написал(а): > > > > > > > > > > Hi, +1 to cherry-pick. > > > > > > > > > > вт, 23 нояб. 2021 г. в 15:25, Ivan Daschinsky : > > > > >> > > > > >> Hi! Lets add one simple bug fix, but very useful > > > > >> > > > > >> https://issues.apache.org/jira/browse/IGNITE-15767 > > > > >> > > > > >> пн, 22 нояб. 2021 г. в 15:22, Pavel Tupitsyn : > > > > >> > > > > >>> Yes, let's fix those auth issues, there are so many complaints on > > > the user > > > > >>> list. > > > > >>> > > > > >>> On Mon, Nov 22, 2021 at 1:03 PM Mikhail Petrov < > > > pmgheap@gmail.com> > > > > >>> wrote: > > > > >>> > > > > Igniters, > > > > it seems that issues [1] and [2] that were discovered recently are > > > > blockers for the 2.12 release. > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-15951 > > > > [2] https://issues.apache.org/jira/browse/IGNITE-15966 > > > > > > > > On 22.11.2021 11:04, Nikolay Izhikov wrote: > > > > > Hello. > > > > > > > > > > One more tiny fix that will improve java thin client performance > > > > > in > > > > certain cases [1] > > > > > I think it worth to cherry-pick it to 2.12 release. > > > > > > > > > > Any objections? > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-15924 > > > > > > > > > >> 19 нояб. 2021 г., в 14:56, Nikita Amelchev > > > > написал(а): > > > > >> > > > > >> Vladimir, +1. > > > > >> > > > > >> I have cherry-picked the issue. > > > > >> > > > > >> пт, 19 нояб. 2021 г. в 14:53, Steshin Vladimir < > > > vlads...@gmail.com>: > > > > >>> Hi all. > > > > >>> > > > > >>> > > > > >>> Let's add simple and useful thin-client-pool monitoring in > > > 2.12: > > > > >>> > > > > >>> https://issues.apache.org/jira/browse/IGNITE-15183 > > > > >>> > > > > >>> https://github.com/apache/ignite/pull/9525 > > > > >>> > > > > >>> > > > > >>> 19.11.2021 09:50, Maksim Timonin пишет: > > > > Hi, I want to add to 2.12 a fix [1] of a bug [2]. It affects > > > users > > > > >>> of > > > > metrics in case there are too many partitions (> 2), we > > > observed > > > > extensive heap usage. > > > > > > > > [1] https://github.com/apache/ignite/pull/9518/files > > > > [2] https://issues.apache.org/jira/browse/IGNITE-15781 > > > > > > > > On Thu, Nov 18, 2021 at 12:59 PM Maksim Timonin < > > > > timonin.ma...@gmail.com> > > > > wrote: > > > > > > > > > Hi, guys, we have a blocker [1]. NPE for some SQL queries. I > > > will > > > > fix it > > > > > in 1-2 days. > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-15943 > > > > > > > > > > On Mon, Nov 15, 2021 at 2:57 PM Maxim Muzafarov < > > > mmu...@apache.org > > > > > > > > wrote: > > > > > > > > > >> Folks, > > > > >> > > > > >> I've fixed almost all the review comments [1]. Hopefully, the > > > > changes > > > > >> will be merged by the end of the week. > > > > >> > > > > >> [1] https://issues.apache.org/jira/browse/IGNITE-14744 > > > > >> > > > > >> On Mon, 8 Nov 2021 at 15:42, Nikita Amelchev < > > > > >>> namelc...@apache.org> > > > > >> wrote: > > > > >>> Hello, Pavel. > > > > >>> > > > > >>> The code freeze is blocked by two issues discussed above [1] > > > [2]. > > > > The > > > > >>> issues are in the final review state and should be merged > > > soon. > > > > >>> > > > > >>> The bugfix is useful and can be cherry-picked, thank you. > > > > >>> > > > > >>> [1] https://issues.apache.org/jira/browse/IGNITE-1553
[DISCUSSION] [IEP-81] New Cluster Management API (Ignite 2.x)
Igniters, I'd like to propose a new internal cluster management API that will simplify new cluster management commands creation and expose new commands to CLI, REST, JMX Ignite interfaces without any additional actions from the engineer adding a new command. This main goal of the IEP is supposed to be available after the implementation of the 1-st phase. From my point of view, the implementation will also provide some additional features like auto ASCII-docs generation which can be achieved with minor code changes. Please, take a look at the IEP-81 [1] with a detailed explanation of my proposal. Below you can find some crucial points from it. = Current Limitations = - The same commands through CLI, REST, JMX APIs don't have the same input arguments and use different subsystems for command execution; - A new command that is added must be manually exposed to all the Ignite APIs (new implementation required for each new command being added); - New commands can't be added via Ignite Plugins and exposed to API; - The own binary protocol is used (GridClient) for command executions instead of the ignite thin client (IgniteClient); - Security and role model: a user have to add compute tasks permissions the same time as adding permissions for the process he intended to use. - New commands can't be added or executed at runtime = Crutial Impelemntation Notes = 1. Create a one for all proxy compute task gate that will accept a map of parameters for preparing management command based on input parameters and executing it on ignite nodes. For instance: IgniteClient.compute().execute(ProxyManagementTask.class.getName(), attrs); Map: baseline.add=execute baseline.add.projection=SINGLE baseline.add.nodes=[consistendtId3, consistendeId4] 2. Create a CommandRegisty that will contain all available commanded on the local ignite node. Commands will be added by command scanners e.g. AnnotationCommandScanner, PackageCommandScanner, URICommandScanner as well as registered manually at runtime by calling `add` method on command registry. The CommandRegistry will also be available for the thin clients in a standalone mode. 3. Prepare a command parsers for REST, JMX, CLI interfaces that will use command registry and calling the proxy management task right away with correct task input parameters. 4. Check the API [2] for commands that may be executed only on a single node (the same as the VisorOneNodeTask) or on all nodes e.g. collecting some information from each node and reducing it on a originating node. [1] https://cwiki.apache.org/confluence/display/IGNITE/IEP-81+Cluster+Management+API [2] https://cwiki.apache.org/confluence/display/IGNITE/IEP-81+Cluster+Management+API#IEP81ClusterManagementAPI-CommandInterface
Re: [DISCUSSION][IEP-80] Breaking changes in Ignite-2.x
Folks, This is a friendly reminder :-) PR [1] is ready for review. Will anyone take a look? [1] https://github.com/apache/ignite/pull/9577/files On Sat, 20 Nov 2021 at 03:17, Maxim Muzafarov wrote: > > Folks, > > > I've prepared the changes related to the legacy Service Grid removal > for the 2.13 release. Here is the issue [1] and the PR [2] of these > changes. > > Some important notes: > > - Removed GridServiceProcessor legacy implementation (internal) > - The IGNITE_EVENT_DRIVEN_SERVICE_PROCESSOR_ENABLED system property > was removed (this is a breaking change) > - The ATTR_EVENT_DRIVEN_SERVICE_PROCESSOR_ENABLED node attribute was > removed (this is a breaking change) > - The node with 2.13 won't be able to connect to 2.12 (assume Ignite > services are used) > > The legacy test suite [3] will be removed after the [1] issue will be merged. > > > Who can take a look at this PR? > > > [1] https://issues.apache.org/jira/browse/IGNITE-15758 > [2] https://github.com/apache/ignite/pull/9577/files > [3] > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ServiceGridLegacyMode > > On Fri, 22 Oct 2021 at 12:43, Nikolay Izhikov wrote: > > > > Hello. > > > > In the scope of IEP-80 Breaking changes in Ignite-2.x PR for the following > > API are prepared: > > > > 1. MVCC > > - https://issues.apache.org/jira/browse/IGNITE-15757 > > - https://github.com/apache/ignite/pull/9516 > > > > 2. LOCAL caches > > - https://issues.apache.org/jira/browse/IGNITE-15756 > > - https://github.com/apache/ignite/pull/9515 > > > > 3. CacheConfiguration#rebalanceDelay > > - https://issues.apache.org/jira/browse/IGNITE-15764 > > - https://github.com/apache/ignite/pull/9515 > > > > Please, review. > > > > > 15 окт. 2021 г., в 16:23, Nikolay Izhikov > > > написал(а): > > > > > > THanks, Maksim. > > > > > > Tickets included in IEP scope and marked for d&r in 2.12-2.13 > > > > > >> 15 окт. 2021 г., в 16:03, Maxim Muzafarov написал(а): > > >> > > >> Let's deprecate RebalanceDelay and RebalanceMode=NONE also. > > >> > > >> [1] https://issues.apache.org/jira/browse/IGNITE-12662 > > >> [2] https://issues.apache.org/jira/browse/IGNITE-14613 > > >> > > >> On Fri, 15 Oct 2021 at 15:46, Anton Vinogradov wrote: > > >>> > > >>> +1 > > >>> > > >>> On Fri, Oct 15, 2021 at 3:41 PM Nikita Amelchev > > >>> wrote: > > >>> > > +1 for deprecation in the 2.12 release > > > > пт, 15 окт. 2021 г. в 15:35, Nikolay Izhikov : > > > > > > Hello, Igniters. > > > > > > I’ve prepared IEP-80 [1] to track breaking changes that should be done > > in Ignite 2.x. > > > > > > We agreed on the following algorithm to deal with breaking changes: > > > > > > - Ignite 2.[x] - deprecate the API + notification of the users. > > > - Ignite 2.[x+1] - API removal. > > > > > > > > > I propose to start these improvements with the d&r (depuration & > > removal) of the following features: > > > > > > - LOCAL caches > > > - MVCC caches > > > - legacy service grid implementation > > > - legacy JMX metrics beans. > > > > > > Deprecation in 2.12 > > > Removal in 2.13 > > > > > > Please, share your opinion. > > > > > > [1] > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191334475 > > > > > > > > -- > > Best wishes, > > Amelchev Nikita > > > > > > >
Re: [DISCUSSION] [IEP-81] New Cluster Management API (Ignite 2.x)
Hello, Maxim.
Thanks for the IEP.
I support the intentions and design of the IEP in general but have some
comments:
> AnnotationCommandScanner, PackageCommandScanner, URICommandScanner
Don’t think we should rely on auto scan class path capabilities.
1. How this automatic scanner will distinguish ignite class from user
class if they both in node class path and same package like
«org.apache.ignite.internal»?
2. It seems hard to debug any errors with auto class path scanner. How
we ensure correct order of scanning in case different jar order in class path?
I think we should go with some kind of hardcoded «on startup» command
registration.
> execute(ProxyManagementTask.class, Map atts)
1. Do we have some well-defined place to validate `atts`?
2. Do we really need to rely on `java.lang.Class` parameter?
This means executor (thin client) should have all class definitions
which can be hard for plugins provided class.
Can we rely on some kind of «path array» like `String[] { «baseline»,
«set» }` or `String[] { «user», «add» }`
So the API usage will looks like:
```
String execute(String[] path, Map params)
Map params = new HashMap();
params.put(«—login», «admin»)
params.put(«—password», «mypassw0rd»)
String res = execute(new String[] {«user», «add»}, params);
```
> Design issues … it is not possible to add and run new management tasks
> at the cluster runtime (e.g. REPL mode for CLI tools can't be used);
It seems to me as a good thing?
We shouldn’t have ability to add management tasks in runtime from the user side
because of security reasons.
But, we should be able to run any existing task dynamically in some scripting
environment - REPL, CLI as you mentioned.
> Features
I think we should focus on:
1. Subsystem then stores all available management tasks
2. Unified execution flow that guarantees all required things like
authorization, authentication, logging and event generation.
3. Creation of the specific protocol implementation that will expose all
existing commands **in the same way** through all supported protocols
**without any additional development**.
4. Simplification of new command development and improvement of existing.
> Compatibility
IEP doesn’t clearly define compatibility.
Do we keep all existing commands as is?
Is new subsystem a replacement for current API?
Is new subsystem will co-exist with current API? For how long?
> 26 нояб. 2021 г., в 11:44, Maxim Muzafarov написал(а):
>
> Igniters,
>
>
> I'd like to propose a new internal cluster management API that will
> simplify new cluster management commands creation and expose new
> commands to CLI, REST, JMX Ignite interfaces without any additional
> actions from the engineer adding a new command. This main goal of the
> IEP is supposed to be available after the implementation of the 1-st
> phase. From my point of view, the implementation will also provide
> some additional features like auto ASCII-docs generation which can be
> achieved with minor code changes.
>
> Please, take a look at the IEP-81 [1] with a detailed explanation of
> my proposal. Below you can find some crucial points from it.
>
>
> = Current Limitations =
>
> - The same commands through CLI, REST, JMX APIs don't have the same
> input arguments and use different subsystems for command execution;
> - A new command that is added must be manually exposed to all the
> Ignite APIs (new implementation required for each new command being
> added);
> - New commands can't be added via Ignite Plugins and exposed to API;
> - The own binary protocol is used (GridClient) for command executions
> instead of the ignite thin client (IgniteClient);
> - Security and role model: a user have to add compute tasks
> permissions the same time as adding permissions for the process he
> intended to use.
> - New commands can't be added or executed at runtime
>
>
> = Crutial Impelemntation Notes =
>
> 1. Create a one for all proxy compute task gate that will accept a map
> of parameters for preparing management command based on input
> parameters and executing it on ignite nodes.
> For instance:
>
> IgniteClient.compute().execute(ProxyManagementTask.class.getName(), attrs);
>
> Map:
> baseline.add=execute
> baseline.add.projection=SINGLE
> baseline.add.nodes=[consistendtId3, consistendeId4]
>
> 2. Create a CommandRegisty that will contain all available commanded
> on the local ignite node. Commands will be added by command scanners
> e.g. AnnotationCommandScanner, PackageCommandScanner,
> URICommandScanner as well as registered manually at runtime by calling
> `add` method on command registry. The CommandRegistry will also be
> available for the thin clients in a standalone mode.
>
> 3. Prepare a command parsers for REST, JMX, CLI interfaces that will
> use command registry and calling the proxy management task right away
> with correct task input parameters.
>
> 4. Che
Re: [DISCUSSION] [IEP-81] New Cluster Management API (Ignite 2.x)
Hi Maxim,
I like the idea, the IEP looks very useful.
However, I agree with Nikolay on this
>
> Don’t think we should rely on auto scan class path capabilities.
> 1. How this automatic scanner will distinguish ignite class from
> user class if they both in node class path and same package like
> «org.apache.ignite.internal»?
> 2. It seems hard to debug any errors with auto class path scanner.
> How we ensure correct order of scanning in case different jar order in
> class path?
I think we should go with some kind of hardcoded «on startup» command
> registration.
Maybe we could use Java ServiceLoader [1] for this?
You can add a list of command classes to e.g.
META-INF/services/o.a.i.CommandHandlerProvider,
then register all mentioned classes that are found via ServiceLoader.load(
CommandHandlerProvider.class)
Service loader can return the iterator for all found providers.
This will prevent scanning all the classes in classpath and allow to load
classes from 3-rd party extensions.
[1] https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html
On Fri, Nov 26, 2021 at 12:25 PM Nikolay Izhikov
wrote:
> Hello, Maxim.
>
> Thanks for the IEP.
> I support the intentions and design of the IEP in general but have some
> comments:
>
> > AnnotationCommandScanner, PackageCommandScanner, URICommandScanner
>
> Don’t think we should rely on auto scan class path capabilities.
> 1. How this automatic scanner will distinguish ignite class from
> user class if they both in node class path and same package like
> «org.apache.ignite.internal»?
> 2. It seems hard to debug any errors with auto class path scanner.
> How we ensure correct order of scanning in case different jar order in
> class path?
>
> I think we should go with some kind of hardcoded «on startup» command
> registration.
>
>
> > execute(ProxyManagementTask.class, Map atts)
>
> 1. Do we have some well-defined place to validate `atts`?
>
> 2. Do we really need to rely on `java.lang.Class` parameter?
> This means executor (thin client) should have all class
> definitions which can be hard for plugins provided class.
> Can we rely on some kind of «path array» like `String[] {
> «baseline», «set» }` or `String[] { «user», «add» }`
> So the API usage will looks like:
>
> ```
> String execute(String[] path, Map params)
>
> Map params = new HashMap();
>
> params.put(«—login», «admin»)
> params.put(«—password», «mypassw0rd»)
>
> String res = execute(new String[] {«user», «add»}, params);
> ```
>
>
> > Design issues … it is not possible to add and run new management
> tasks at the cluster runtime (e.g. REPL mode for CLI tools can't be used);
>
> It seems to me as a good thing?
> We shouldn’t have ability to add management tasks in runtime from the user
> side because of security reasons.
>
> But, we should be able to run any existing task dynamically in some
> scripting environment - REPL, CLI as you mentioned.
>
>
> > Features
>
> I think we should focus on:
>
> 1. Subsystem then stores all available management tasks
> 2. Unified execution flow that guarantees all required things like
> authorization, authentication, logging and event generation.
> 3. Creation of the specific protocol implementation that will expose all
> existing commands **in the same way** through all supported protocols
> **without any additional development**.
> 4. Simplification of new command development and improvement of existing.
>
>
> > Compatibility
>
> IEP doesn’t clearly define compatibility.
> Do we keep all existing commands as is?
> Is new subsystem a replacement for current API?
>
> Is new subsystem will co-exist with current API? For how long?
>
> > 26 нояб. 2021 г., в 11:44, Maxim Muzafarov
> написал(а):
> >
> > Igniters,
> >
> >
> > I'd like to propose a new internal cluster management API that will
> > simplify new cluster management commands creation and expose new
> > commands to CLI, REST, JMX Ignite interfaces without any additional
> > actions from the engineer adding a new command. This main goal of the
> > IEP is supposed to be available after the implementation of the 1-st
> > phase. From my point of view, the implementation will also provide
> > some additional features like auto ASCII-docs generation which can be
> > achieved with minor code changes.
> >
> > Please, take a look at the IEP-81 [1] with a detailed explanation of
> > my proposal. Below you can find some crucial points from it.
> >
> >
> > = Current Limitations =
> >
> > - The same commands through CLI, REST, JMX APIs don't have the same
> > input arguments and use different subsystems for command execution;
> > - A new command that is added must be manually exposed to all the
> > Ignite APIs (new implementation required for each new command being
> > added);
> > - New commands can't be added via Ignite Plugins and exposed to API;
> > - The own binary protocol is used (GridClient) for c
