Re: Adding metrics of using WAL archive

[Nikolay Izhikov]

Hello, Kirill.

Can you, please, clarify - What question about WAL user have in mind?
And what answers he(or she) gets with these new metrics?

> 12 февр. 2021 г., в 14:26, ткаленко кирилл  написал(а):
> 
> Hi everyone!
> At the moment, I have not found an opportunity to estimate how many WAL 
> segments fall into the archive, say per day.
> So I created a ticket https://issues.apache.org/jira/browse/IGNITE-14170 to 
> add a couple of new metrics.

.NET naming convention + RegisterSameJavaType mode

[Nikolay Izhikov]

Hello, Igniters.

Currently, BasicBinaryNameMapper[isSimpleName=false] assumes that type names 
equals both in Java and .Net.

• Java type - ru.company.Model
• .Net type - ‘ru.company.Model`

However, .net naming conventions [1] differs from java [2]
So, if a user follows both conventions then names will be

• Java type - ru.company.Model
• .Net type - 'Ru.Company.Model`

User can implement this behaviour in custom name mapper, but custom name mapper 
will disable RegisterSameJavaType mode.

To solve this issue I prepared a PR[3] and ticked [4].
PR introduces new BasicBinaryNameMapper#ForceJavaNamingConventions flag that 
enables mapping from .Net style to the java naming style.

WDYT?

[1] 
https://docs.microsoft.com/en-us/dotnet/standard/design-guidelines/names-of-namespaces
[2] 
https://www.oracle.com/java/technologies/javase/codeconventions-namingconventions.html
[3] https://github.com/apache/ignite/pull/8795
[4] https://issues.apache.org/jira/browse/IGNITE-14169

Re: Adding metrics of using WAL archive

[Nikolay Izhikov]

Hello, Kirill.

> It is not possible for us to estimate how much space a user will need in the 
> archive so as not to overflow it under its load

It still not clear for me why do we need those metrics.
Can you please, write down specific scenario - how user will use these metrics 
to estimate required WAL volume?

> 12 февр. 2021 г., в 19:35, ткаленко кирилл  написал(а):
> 
> Hi, Nikolay!
> 
> It is not possible for us to estimate how much space a user will need in the 
> archive so as not to overflow it under its load. And the proposed metrics 
> will allow you to make a rough estimate.
> 
> 
> 12.02.2021, 17:23, "Nikolay Izhikov" :
>> Hello, Kirill.
>> 
>> Can you, please, clarify - What question about WAL user have in mind?
>> And what answers he(or she) gets with these new metrics?
>> 
>>>  12 февр. 2021 г., в 14:26, ткаленко кирилл  
>>> написал(а):
>>> 
>>>  Hi everyone!
>>>  At the moment, I have not found an opportunity to estimate how many WAL 
>>> segments fall into the archive, say per day.
>>>  So I created a ticket https://issues.apache.org/jira/browse/IGNITE-14170 
>>> to add a couple of new metrics.

Re: [DISCUSSION] Moving tracing feature to GA

[Nikolay Izhikov]

Hello, Valentin.

I’m -1 to make Tracing API GA

>  the API is stabilized, and there are no known performance issues associated 
> with it.

For now, API works with strings only and this leads to huge performance 
degradation when tracing enabled.
Our measurement shows about 50% drop.

Moreover, as we discussed earlier, there is a plan to merge internals of 
tracing and performance profiling 
To make it, we should enhance Tracing API and learn it to work with primitives 
in some king of lazy manner.

So, I think we can’t stabilise Tracing API, for now.

You can find more details in meetup talk (in Russian) [1]

[1] https://www.youtube.com/watch?v=68QoOBD574g


> 13 февр. 2021 г., в 04:23, Valentin Kulichenko 
>  написал(а):
> 
> Igniters,
> 
> As far as I know, the tracing feature [1] is still marked as experimental.
> However, It seems that we've had it for a while, it's being used, the API
> is stabilized, and there are no known performance issues associated with it.
> 
> That said, should we remove the "experimental" mark and move the feature to
> the GA status? Please share your thoughts.
> 
> [1] https://ignite.apache.org/docs/latest/monitoring-metrics/tracing
> 
> -Val

Re: Adding metrics of using WAL archive

[Nikolay Izhikov]

Hello, Kirill

Your conclusions still not clear for me.

>  It is not possible for us to estimate how much space a user will need in the 
> archive so as not to overflow it under its load
>  We take the maximum 44  and multiply it by a 
> DataStorageConfiguration#getWalSegmentSize

Why you take a single day (24h) for a standard period? Is there any rationale 
behind this?

1. We have `walAutoArchiveAfterInactivity` property. So WAL segment can have a 
size less than the maximum.
2. For CDC feature I want to introduce «WAL force rollover timeout» to make 
data available for a consumer in a guaranteed period [1].

Why does the user want to estimate those numbers in the first place?
Are we talking about some kind of capacity planning?

If yes, then maybe it will be better to have a metric for a count of bytes 
written in the WAL?
With it, we will have an exact number of space we need for WAL.

How user should estimate capacity for a page memory and indexes?

[1] https://issues.apache.org/jira/browse/IGNITE-13582

> 14 февр. 2021 г., в 09:48, ткаленко кирилл  написал(а):
> 
> Hi, Nikolay!
> 
> The user will be able to take the getLastArchivedSegmentIndex every day and 
> remember it and do it, say, for several days. 
> 
> For example, when starting the application, the getLastArchivedSegmentIndex 
> is 0, then at the end of the first day the value will be 30 at the end of the 
> second 55 and at the end of the third 99.
> It turns out that 30 segments were used for the first day, 25 for the second 
> and 44 for the third. We take the maximum 44 and multiply it by a 
> DataStorageConfiguration#getWalSegmentSize, and we get the possible maximum 
> that the archive overflow was the least likely. If the user uses compression, 
> then it can be subtracted from the result (result * 
> getMaxSizeCompressedArchivedSegment).
> 
> 13.02.2021, 10:47, "Nikolay Izhikov" :
>> Hello, Kirill.
>> 
>>>  It is not possible for us to estimate how much space a user will need in 
>>> the archive so as not to overflow it under its load
>> 
>> It still not clear for me why do we need those metrics.
>> Can you please, write down specific scenario - how user will use these 
>> metrics to estimate required WAL volume?
>> 
>>>  12 февр. 2021 г., в 19:35, ткаленко кирилл  
>>> написал(а):
>>> 
>>>  Hi, Nikolay!
>>> 
>>>  It is not possible for us to estimate how much space a user will need in 
>>> the archive so as not to overflow it under its load. And the proposed 
>>> metrics will allow you to make a rough estimate.
>>> 
>>>  12.02.2021, 17:23, "Nikolay Izhikov" :
>>>>  Hello, Kirill.
>>>> 
>>>>  Can you, please, clarify - What question about WAL user have in mind?
>>>>  And what answers he(or she) gets with these new metrics?
>>>> 
>>>>>   12 февр. 2021 г., в 14:26, ткаленко кирилл  
>>>>> написал(а):
>>>>> 
>>>>>   Hi everyone!
>>>>>   At the moment, I have not found an opportunity to estimate how many WAL 
>>>>> segments fall into the archive, say per day.
>>>>>   So I created a ticket 
>>>>> https://issues.apache.org/jira/browse/IGNITE-14170 to add a couple of new 
>>>>> metrics.

Re: .NET naming convention + RegisterSameJavaType mode

[Nikolay Izhikov]

Hello, Pavel.

Thanks, for the feedback

> We can create a derived class so that existing _registerSameJavaType logic 
> works.

For now, BinaryBasicNameMapper#GetTypeName is not virtual.
So we can’t override it.

And if we make GetTypeName virtual then any user can make extension of 
BinaryBasicNameMapper and use it.
I thought we want to avoid this.

Anyway, I’m OK with the built-in extension of BinaryBasicNameMapper.
Will do those changes.

> 13 февр. 2021 г., в 13:07, Pavel Tupitsyn  написал(а):
> 
> Nikolay,
> 
> 1) What about the org./com. prefix that is present in all Java package names, 
> e.g:
> Java: org.apache.ignite.foo.Bar
> .NET: Apache.Ignite.Foo.Bar
> 
> I think we should handle this too.
> 
> 2) Can we create a separate name mapper instead of adding a property to the 
> existing one?
> We can create a derived class so that existing _registerSameJavaType logic 
> works.
> 
> On Fri, Feb 12, 2021 at 7:33 PM Nikolay Izhikov  wrote:
> Hello, Igniters.
> 
> Currently, BasicBinaryNameMapper[isSimpleName=false] assumes that type names 
> equals both in Java and .Net.
> 
> • Java type - ru.company.Model
> • .Net type - ‘ru.company.Model`
> 
> However, .net naming conventions [1] differs from java [2]
> So, if a user follows both conventions then names will be
> 
> • Java type - ru.company.Model
> • .Net type - 'Ru.Company.Model`
> 
> User can implement this behaviour in custom name mapper, but custom name 
> mapper will disable RegisterSameJavaType mode.
> 
> To solve this issue I prepared a PR[3] and ticked [4].
> PR introduces new BasicBinaryNameMapper#ForceJavaNamingConventions flag that 
> enables mapping from .Net style to the java naming style.
> 
> WDYT?
> 
> [1] 
> https://docs.microsoft.com/en-us/dotnet/standard/design-guidelines/names-of-namespaces
> [2] 
> https://www.oracle.com/java/technologies/javase/codeconventions-namingconventions.html
> [3] https://github.com/apache/ignite/pull/8795
> [4] https://issues.apache.org/jira/browse/IGNITE-14169
> 

Re: Adding metrics of using WAL archive

[Nikolay Izhikov]

Hello, Kirill.

Thanks for an answers.
Now, I understand your intentions.

> t also seems that it will be more natural to operate not just bytes but 
> multiples of a segment.

Can’t agree here.
From my point of view - it’s better to know exact number, not just «count of 
segments».

> 15 февр. 2021 г., в 13:00, ткаленко кирилл  написал(а):
> 
> Hello, Nikolay!
> 
> The period of one day (24h) seems more natural, you can take more or less, I 
> think that one day may not be enough, and it is worth getting the metric for 
> several days (collect statistics) for example a week. Yes, the total size of 
> the segments may not be DataStorageConfiguration#getMaxWalArchiveSize, but 
> for capacity planning, accuracy is not so important to us, since the load can 
> always change, it will hurt users more if we overflow the archive and it will 
> not be able to start the node. So to say that more is better than less, it 
> also seems that it will be more natural to operate not just bytes but 
> multiples of a segment.
> 
> In separate threads, you can discuss the metric that you propose about page 
> memory and indexes estimates.
> 
> 
> 14.02.2021, 11:54, "Nikolay Izhikov" :
>> Hello, Kirill
>> 
>> Your conclusions still not clear for me.
>> 
>>>   It is not possible for us to estimate how much space a user will need in 
>>> the archive so as not to overflow it under its load
>>>   We take the maximum 44 and multiply it by a 
>>> DataStorageConfiguration#getWalSegmentSize
>> 
>> Why you take a single day (24h) for a standard period? Is there any 
>> rationale behind this?
>> 
>> 1. We have `walAutoArchiveAfterInactivity` property. So WAL segment can have 
>> a size less than the maximum.
>> 2. For CDC feature I want to introduce «WAL force rollover timeout» to make 
>> data available for a consumer in a guaranteed period [1].
>> 
>> Why does the user want to estimate those numbers in the first place?
>> Are we talking about some kind of capacity planning?
>> 
>> If yes, then maybe it will be better to have a metric for a count of bytes 
>> written in the WAL?
>> With it, we will have an exact number of space we need for WAL.
>> 
>> How user should estimate capacity for a page memory and indexes?
>> 
>> [1] https://issues.apache.org/jira/browse/IGNITE-13582
>> 
>>>  14 февр. 2021 г., в 09:48, ткаленко кирилл  
>>> написал(а):
>>> 
>>>  Hi, Nikolay!
>>> 
>>>  The user will be able to take the getLastArchivedSegmentIndex every day 
>>> and remember it and do it, say, for several days.
>>> 
>>>  For example, when starting the application, the 
>>> getLastArchivedSegmentIndex is 0, then at the end of the first day the 
>>> value will be 30 at the end of the second 55 and at the end of the third 99.
>>>  It turns out that 30 segments were used for the first day, 25 for the 
>>> second and 44 for the third. We take the maximum 44 and multiply it by a 
>>> DataStorageConfiguration#getWalSegmentSize, and we get the possible maximum 
>>> that the archive overflow was the least likely. If the user uses 
>>> compression, then it can be subtracted from the result (result * 
>>> getMaxSizeCompressedArchivedSegment).
>>> 
>>>  13.02.2021, 10:47, "Nikolay Izhikov" :
>>>>  Hello, Kirill.
>>>> 
>>>>>   It is not possible for us to estimate how much space a user will need 
>>>>> in the archive so as not to overflow it under its load
>>>> 
>>>>  It still not clear for me why do we need those metrics.
>>>>  Can you please, write down specific scenario - how user will use these 
>>>> metrics to estimate required WAL volume?
>>>> 
>>>>>   12 февр. 2021 г., в 19:35, ткаленко кирилл  
>>>>> написал(а):
>>>>> 
>>>>>   Hi, Nikolay!
>>>>> 
>>>>>   It is not possible for us to estimate how much space a user will need 
>>>>> in the archive so as not to overflow it under its load. And the proposed 
>>>>> metrics will allow you to make a rough estimate.
>>>>> 
>>>>>   12.02.2021, 17:23, "Nikolay Izhikov" :
>>>>>>   Hello, Kirill.
>>>>>> 
>>>>>>   Can you, please, clarify - What question about WAL user have in mind?
>>>>>>   And what answers he(or she) gets with these new metrics?
>>>>>> 
>>>>>>>12 февр. 2021 г., в 14:26, ткаленко кирилл  
>>>>>>> написал(а):
>>>>>>> 
>>>>>>>Hi everyone!
>>>>>>>At the moment, I have not found an opportunity to estimate how many 
>>>>>>> WAL segments fall into the archive, say per day.
>>>>>>>So I created a ticket 
>>>>>>> https://issues.apache.org/jira/browse/IGNITE-14170 to add a couple of 
>>>>>>> new metrics.

Re: Adding metrics of using WAL archive

[Nikolay Izhikov]

My suggestion that «count of files» is meaningless number.
And «count of bytes written to the files» is useful number to know and use for 
capacity planning..

> 15 февр. 2021 г., в 15:59, ткаленко кирилл  написал(а):
> 
> Hi, Nikolay!
> 
> There may be a number (count of segments * segment size) or there may be a 
> count of segments, whichever is more convenient for the user.
> 
> 15.02.2021, 13:14, "Nikolay Izhikov" :
>> Hello, Kirill.
>> 
>> Thanks for an answers.
>> Now, I understand your intentions.
>> 
>>>  t also seems that it will be more natural to operate not just bytes but 
>>> multiples of a segment.
>> 
>> Can’t agree here.
>> From my point of view - it’s better to know exact number, not just «count of 
>> segments».
>> 
>>>  15 февр. 2021 г., в 13:00, ткаленко кирилл  
>>> написал(а):
>>> 
>>>  Hello, Nikolay!
>>> 
>>>  The period of one day (24h) seems more natural, you can take more or less, 
>>> I think that one day may not be enough, and it is worth getting the metric 
>>> for several days (collect statistics) for example a week. Yes, the total 
>>> size of the segments may not be 
>>> DataStorageConfiguration#getMaxWalArchiveSize, but for capacity planning, 
>>> accuracy is not so important to us, since the load can always change, it 
>>> will hurt users more if we overflow the archive and it will not be able to 
>>> start the node. So to say that more is better than less, it also seems that 
>>> it will be more natural to operate not just bytes but multiples of a 
>>> segment.
>>> 
>>>  In separate threads, you can discuss the metric that you propose about 
>>> page memory and indexes estimates.
>>> 
>>>  14.02.2021, 11:54, "Nikolay Izhikov" :
>>>>  Hello, Kirill
>>>> 
>>>>  Your conclusions still not clear for me.
>>>> 
>>>>>It is not possible for us to estimate how much space a user will need 
>>>>> in the archive so as not to overflow it under its load
>>>>>We take the maximum 44 and multiply it by a 
>>>>> DataStorageConfiguration#getWalSegmentSize
>>>> 
>>>>  Why you take a single day (24h) for a standard period? Is there any 
>>>> rationale behind this?
>>>> 
>>>>  1. We have `walAutoArchiveAfterInactivity` property. So WAL segment can 
>>>> have a size less than the maximum.
>>>>  2. For CDC feature I want to introduce «WAL force rollover timeout» to 
>>>> make data available for a consumer in a guaranteed period [1].
>>>> 
>>>>  Why does the user want to estimate those numbers in the first place?
>>>>  Are we talking about some kind of capacity planning?
>>>> 
>>>>  If yes, then maybe it will be better to have a metric for a count of 
>>>> bytes written in the WAL?
>>>>  With it, we will have an exact number of space we need for WAL.
>>>> 
>>>>  How user should estimate capacity for a page memory and indexes?
>>>> 
>>>>  [1] https://issues.apache.org/jira/browse/IGNITE-13582
>>>> 
>>>>>   14 февр. 2021 г., в 09:48, ткаленко кирилл  
>>>>> написал(а):
>>>>> 
>>>>>   Hi, Nikolay!
>>>>> 
>>>>>   The user will be able to take the getLastArchivedSegmentIndex every day 
>>>>> and remember it and do it, say, for several days.
>>>>> 
>>>>>   For example, when starting the application, the 
>>>>> getLastArchivedSegmentIndex is 0, then at the end of the first day the 
>>>>> value will be 30 at the end of the second 55 and at the end of the third 
>>>>> 99.
>>>>>   It turns out that 30 segments were used for the first day, 25 for the 
>>>>> second and 44 for the third. We take the maximum 44 and multiply it by a 
>>>>> DataStorageConfiguration#getWalSegmentSize, and we get the possible 
>>>>> maximum that the archive overflow was the least likely. If the user uses 
>>>>> compression, then it can be subtracted from the result (result * 
>>>>> getMaxSizeCompressedArchivedSegment).
>>>>> 
>>>>>   13.02.2021, 10:47, "Nikolay Izhikov" :
>>>>>>   Hello, Kirill.
>>>>>> 
>>>>>>>It is not possible for us to estimate how much space a user will 
>>>>>>> need 

Re: Adding metrics of using WAL archive

[Nikolay Izhikov]

Kirill.

«Count of segments» is a very internal thing for a regular user.
Regular user don’t want to know about such things.

You suggest to calculate the number (space required to store WAL) with some 
kind of rough calculation, and with the «Count of bytes written in WAL» we can 
have exact number without any suggestions or calculations.

Moreover, «Count of bytes written in WAL» is independent on internal WAL 
implementation.

So, I think exact number is always better to have then some approximation.

What do you think?


> 15 февр. 2021 г., в 20:45, ткаленко кирилл  написал(а):
> 
> Hi, Nikolay!
> 
> We set the number of segments in the working directory, we also delete by 
> segment, it seems that this is a matter of usability. I prefer to dwell on my 
> own version, this is a simple metric that does not hurt and you can add more 
> as needed.
> 
> 15.02.2021, 17:10, "Nikolay Izhikov" :
>> My suggestion that «count of files» is meaningless number.
>> And «count of bytes written to the files» is useful number to know and use 
>> for capacity planning..
>> 
>>>  15 февр. 2021 г., в 15:59, ткаленко кирилл  
>>> написал(а):
>>> 
>>>  Hi, Nikolay!
>>> 
>>>  There may be a number (count of segments * segment size) or there may be a 
>>> count of segments, whichever is more convenient for the user.
>>> 
>>>  15.02.2021, 13:14, "Nikolay Izhikov" :
>>>>  Hello, Kirill.
>>>> 
>>>>  Thanks for an answers.
>>>>  Now, I understand your intentions.
>>>> 
>>>>>   t also seems that it will be more natural to operate not just bytes but 
>>>>> multiples of a segment.
>>>> 
>>>>  Can’t agree here.
>>>>  From my point of view - it’s better to know exact number, not just «count 
>>>> of segments».
>>>> 
>>>>>   15 февр. 2021 г., в 13:00, ткаленко кирилл  
>>>>> написал(а):
>>>>> 
>>>>>   Hello, Nikolay!
>>>>> 
>>>>>   The period of one day (24h) seems more natural, you can take more or 
>>>>> less, I think that one day may not be enough, and it is worth getting the 
>>>>> metric for several days (collect statistics) for example a week. Yes, the 
>>>>> total size of the segments may not be 
>>>>> DataStorageConfiguration#getMaxWalArchiveSize, but for capacity planning, 
>>>>> accuracy is not so important to us, since the load can always change, it 
>>>>> will hurt users more if we overflow the archive and it will not be able 
>>>>> to start the node. So to say that more is better than less, it also seems 
>>>>> that it will be more natural to operate not just bytes but multiples of a 
>>>>> segment.
>>>>> 
>>>>>   In separate threads, you can discuss the metric that you propose about 
>>>>> page memory and indexes estimates.
>>>>> 
>>>>>   14.02.2021, 11:54, "Nikolay Izhikov" :
>>>>>>   Hello, Kirill
>>>>>> 
>>>>>>   Your conclusions still not clear for me.
>>>>>> 
>>>>>>> It is not possible for us to estimate how much space a user will 
>>>>>>> need in the archive so as not to overflow it under its load
>>>>>>> We take the maximum 44 and multiply it by a 
>>>>>>> DataStorageConfiguration#getWalSegmentSize
>>>>>> 
>>>>>>   Why you take a single day (24h) for a standard period? Is there any 
>>>>>> rationale behind this?
>>>>>> 
>>>>>>   1. We have `walAutoArchiveAfterInactivity` property. So WAL segment 
>>>>>> can have a size less than the maximum.
>>>>>>   2. For CDC feature I want to introduce «WAL force rollover timeout» to 
>>>>>> make data available for a consumer in a guaranteed period [1].
>>>>>> 
>>>>>>   Why does the user want to estimate those numbers in the first place?
>>>>>>   Are we talking about some kind of capacity planning?
>>>>>> 
>>>>>>   If yes, then maybe it will be better to have a metric for a count of 
>>>>>> bytes written in the WAL?
>>>>>>   With it, we will have an exact number of space we need for WAL.
>>>>>> 
>>>>>>   How user should estimate capacity for a page memory and indexes?
>>>>>> 
>>

Re: Adding metrics of using WAL archive

[Nikolay Izhikov]

Hello, Kirill.

Can you, please, write down your proposal?
What metrics you want to add in the Ignite?

> 18 февр. 2021 г., в 14:11, ткаленко кирилл  написал(а):
> 
> Hi, Nikolay!
> 
> Have we reached a consensus?
> 
> 16.02.2021, 17:09, "ткаленко кирилл" :
>> Hi, Zhenya!
>> 
>> Users can also use it, I see nothing wrong with the presence of two metrics.
>> 
>> 16.02.2021, 16:50, "Zhenya Stanilovsky" :
>>>  Kirill, is it good practice to have a metrics for internal use? Don`t 
>>> think so.
>>>  +1 witk Nikolay size is more readable than abstract segments count.
>>> 
>>>>  Hi, Nikolay!
>>>> 
>>>>  For internal use, leave the metric that I propose and also add the 
>>>> metric: Count of bytes logged in WAL. Why not "written" because for the 
>>>> mmap we cannot track when the physical writting will occur.
>>>> 
>>>>  16.02.2021, 15:42, "Nikolay Izhikov" < nizhi...@apache.org >:
>>>>>   Kirill.
>>>>> 
>>>>>   «Count of segments» is a very internal thing for a regular user.
>>>>>   Regular user don’t want to know about such things.
>>>>> 
>>>>>   You suggest to calculate the number (space required to store WAL) with 
>>>>> some kind of rough calculation, and with the «Count of bytes written in 
>>>>> WAL» we can have exact number without any suggestions or calculations.
>>>>> 
>>>>>   Moreover, «Count of bytes written in WAL» is independent on internal 
>>>>> WAL implementation.
>>>>> 
>>>>>   So, I think exact number is always better to have then some 
>>>>> approximation.
>>>>> 
>>>>>   What do you think?
>>>>> 
>>>>>>15 февр. 2021 г., в 20:45, ткаленко кирилл < tkalkir...@yandex.ru > 
>>>>>> написал(а):
>>>>>> 
>>>>>>Hi, Nikolay!
>>>>>> 
>>>>>>We set the number of segments in the working directory, we also 
>>>>>> delete by segment, it seems that this is a matter of usability. I prefer 
>>>>>> to dwell on my own version, this is a simple metric that does not hurt 
>>>>>> and you can add more as needed.
>>>>>> 
>>>>>>15.02.2021, 17:10, "Nikolay Izhikov" < nizhi...@apache.org >:
>>>>>>>My suggestion that «count of files» is meaningless number.
>>>>>>>And «count of bytes written to the files» is useful number to know 
>>>>>>> and use for capacity planning..
>>>>>>> 
>>>>>>>> 15 февр. 2021 г., в 15:59, ткаленко кирилл < tkalkir...@yandex.ru 
>>>>>>>> > написал(а):
>>>>>>>> 
>>>>>>>> Hi, Nikolay!
>>>>>>>> 
>>>>>>>> There may be a number (count of segments * segment size) or there 
>>>>>>>> may be a count of segments, whichever is more convenient for the user.
>>>>>>>> 
>>>>>>>> 15.02.2021, 13:14, "Nikolay Izhikov" < nizhi...@apache.org >:
>>>>>>>>> Hello, Kirill.
>>>>>>>>> 
>>>>>>>>> Thanks for an answers.
>>>>>>>>> Now, I understand your intentions.
>>>>>>>>> 
>>>>>>>>>>  t also seems that it will be more natural to operate not just 
>>>>>>>>>> bytes but multiples of a segment.
>>>>>>>>> 
>>>>>>>>> Can’t agree here.
>>>>>>>>> From my point of view - it’s better to know exact number, not 
>>>>>>>>> just «count of segments».
>>>>>>>>> 
>>>>>>>>>>  15 февр. 2021 г., в 13:00, ткаленко кирилл < 
>>>>>>>>>> tkalkir...@yandex.ru > написал(а):
>>>>>>>>>> 
>>>>>>>>>>  Hello, Nikolay!
>>>>>>>>>> 
>>>>>>>>>>  The period of one day (24h) seems more natural, you can take 
>>>>>>>>>> more or less, I think that one day may not be enough, and it is 
>>>>>>>>>> worth getting the metric 

Re: Adding metrics of using WAL archive

[Nikolay Izhikov]

+1 For this.

> 2 марта 2021 г., в 18:32, ткаленко кирилл  написал(а):
> 
> Hi, Nikolay!
> 
> I thought about your proposal and offer the following two metrics:
> 
> 1)The number of bytes logged to the WAL;
> 2)The number of compressed bytes in the WAL.
> 
> Monotonically increasing long.
> 
> WDYT?

Re: Adding metrics of using WAL archive

[Nikolay Izhikov]

Yes, definitely.

> 5 марта 2021 г., в 16:31, ткаленко кирилл  написал(а):
> 
> Hi Nikolay, can you do a review?
> 
> 02.03.2021, 18:59, "Nikolay Izhikov" :
>> +1 For this.
>> 
>>>  2 марта 2021 г., в 18:32, ткаленко кирилл  
>>> написал(а):
>>> 
>>>  Hi, Nikolay!
>>> 
>>>  I thought about your proposal and offer the following two metrics:
>>> 
>>>  1)The number of bytes logged to the WAL;
>>>  2)The number of compressed bytes in the WAL.
>>> 
>>>  Monotonically increasing long.
>>> 
>>>  WDYT?

Re: [DISCUSSION] IEP-69 The evolutionary release process

[Nikolay Izhikov]

Hello, Maxim.

I’m +1 to follow your proposal.

> 5 марта 2021 г., в 22:08, Maxim Muzafarov  написал(а):
> 
> Ignites,
> 
> 
> I've created the IEP-69 [1] which describes the evolutionary release
> process for the Apache Ignite 2.x version. You can find all the
> details of my suggestion there, but here you can find the crucial
> points:
> 
> 0. Versioning - grand.major.bug-fix[-rc_number]
> 
> 1. Prepare the next 3.0 release based on 2.x with some breaking
> compatibility changes. The same things happen from time to time with
> other Apache projects like Hadoop, Spark.
> 
> 2. Discuss with the whole Community and assign the right release
> version to the activities related to the development of the new Ignite
> architecture (currently all the changes you can find in the ignite-3
> branch).
> I see no 3.0 discussions on the dev-list and I see no-activity with
> the 3.0 version currently. So,  it's better to remove the `lock` from
> the 3.0 version and allow the removal of obsolete features.
> 
> 3. Guarantee the PDS compatibility between the `grand` versions of the
> Apache Ignite for the next year.
> 
> 4. Guarantee the bug-fix release for the last 2.x Apache Ignite
> version for the next year.
> 
> 5. Perform some improvements which break the backward compatibility,
> for instance: removing @deprecated API (except metrics), removing
> obsolete modules, changing the cluster defaults. You can find
> additional details on the IEP-69 page [1].
> 
> 
> Please, share your thoughts.
> 
> 
> [1] 
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-69%3A+The+evolutionary+release+process

Re: [DISCUSSION] IEP-69 The evolutionary release process

[Nikolay Izhikov]

Hello, Alexei

Thanks for feedback.

> Removing some deprecated features and releasing it as Ignite 3.0 doesn’t make 
> sense to me.
> This should be done in Ignite 2.X, but mostly (except MVCC) looks like a 
> waste of time to me.

But we have a big wish list  that we want to remove from Ignite in the next 
major version.
Do you believe it useless for Ignite and Ignite users?

> Such a release has no value for a user, because actually it's 2.X.

I think we can provide more stability and easy to use if remove obsolete parts 
of Ignite code.

> Because 3.0 is started from scratch

This statement is controversial with statements we discussed in the Ignite-3 
thread [2]

Alexey Goncharuk > First of all, I wanted to stress that I do not intend to 
rewrite everything from scratch

[1] 
https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist
[2] 
http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSS-Ignite-3-0-development-approach-td49922.html

> 9 марта 2021 г., в 20:47, Maxim Muzafarov  написал(а):
> 
> Alexei,
> 
> 
> Thank you for sharing details with the progress in the ignite-3
> development with the Community.
> 
> I would like to believe in the best with the distribution database
> development, but please do not forget our previous experience with the
> 2.x version:
> - it took years to make the Ignite production-ready and finally, it
> became like that
> - please note that bad fame is very hard to fix, so the developed but
> not well-tested source code may scare away some users
> - as a developer, I also really enjoy working on breakthrough
> technologies, but It's very sad to hear reviews about instability and
> data loss
> - take into account the resource management - some developers may or
> may not be switched to different projects (you also know examples of
> this)
> - take into account the MVCC and Calcite features wich much smaller
> than the changes submitted to ignite-3 and still not finished
> completely
> 
> According to all of these points above, I can't share your optimism
> and propose to go through my suggested `evolutionary changes` with the
> next release.
> 
> On Tue, 9 Mar 2021 at 20:14, Alexei Scherbakov
>  wrote:
>> 
>>  -1.
>> 
>> Removing some deprecated features and releasing it as Ignite 3.0 doesn't
>> make sense to me.
>> This should be done in Ignite 2.X, but mostly (except MVCC) looks like a
>> waste of time to me.
>> Such a release has no value for a user, because actually it's 2.X.
>> Because 3.0 is started from scratch, it will not contain deprecated
>> features by definition.
>> 
>> Releasing Ignite 4.0, 5.0 etc doesn't make any sense to me as well.
>> Upgrading to a "grand" release is always a big trouble and we shouldn't
>> make such releases as pies.
>> We have already discussed and agreed on a list of release driver IEPs like
>> IEP-54, IEP-55, IEP-61 and should stick to it for 3.0.
>> 
>> Moveover, there is already a big progress on raft protocol implementation
>> in 3.0 (IEP-61), as well as other features, and I'm going to make a
>> public update on this topic in the next few days.
>> 
>> The estimation in years to finish 3.0 looks too huge to me, actually it
>> should be finished by the end of the year.
>> 
>> вт, 9 мар. 2021 г. в 19:53, Maxim Muzafarov :
>> 
>>> Pavel,
>>> 
 We have agreed on a direction for 3.0 [1], no need to change it.
>>> 
>>> Thank you for sharing the link, but there is no agreement on that
>>> thread. The Community even not vote in that direction, so I think we
>>> can consider another option here.
>>> 
>>> On Tue, 9 Mar 2021 at 19:46, Pavel Tupitsyn  wrote:
 
 -1
 
 We have agreed on a direction for 3.0 [1], no need to change it.
 
 [1]
 
>>> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSS-Ignite-3-0-development-approach-td49922.html
 
 On Tue, Mar 9, 2021 at 7:42 PM Maxim Muzafarov 
>>> wrote:
 
> Pavel,
> 
>> 2) Much later, release what is being worked on in ignite-3 as Ignite
>>> 4.0
> or 5.0
> 
> Yes, you're right.
> 
> On Tue, 9 Mar 2021 at 19:41, Maxim Muzafarov 
>>> wrote:
>> 
>> Ilya,
>> 
>>> 0. I am accustomed with major.minor.maintenance schema. Does it
>>> make
> any difference?
>> 
>> There is no difference without a small note that from my point of
>>> view
>> minor releases 2.7 > 2.8 > 2.9 by the amount of changes are not so
>> 'minor'.
>> 
>>> 2. What's `lock'?
>> 
>> I'm talking about some public and marketing activities with 3.0
>> version which happened some time ago [1]. I don't think they can
>> really block the proposed release but at least it should be discussed
>> how we should promote the new release.
>> 
>>> 3. I don't see why there would be implicit PDS compatibility
>>> between
> any X.0.0 and Y.0.0, X != Y.
>> 
>> Yes, in general, we can't guarantee the PDS compatibility. I propose
>> the following steps:
>>>

Thin client implementation removal

[Nikolay Izhikov]

Hello, Igor.

I’ve noticed that Python, NodeJS, PHP thin client implementations were removed 
from master by you.
Can you, please, comment?
Why do you make those decision?

Re: Thin client implementation removal

[Nikolay Izhikov]

Thanks.

> 15 марта 2021 г., в 14:35, Igor Sapego  написал(а):
> 
> Nikolay,
> 
> That's because we now have separate repos for them: [1], [2] and [3].
> Actually, active development is moved to those repos some time ago and
> those directories in main repo are not actual anymore anyway.
> 
> Decision of moving those clients to separate repos was discussed in [4]
> 
> [1] - https://github.com/apache/ignite-python-thin-client
> [2] - https://github.com/apache/ignite-nodejs-thin-client
> [3] - https://github.com/apache/ignite-php-thin-client
> [4] -
> http://apache-ignite-developers.2346864.n4.nabble.com/Moving-python-php-and-node-js-in-separate-repos-and-release-cycles-td47030.html
> 
> Best Regards,
> Igor
> 
> 
> On Mon, Mar 15, 2021 at 2:20 PM Nikolay Izhikov  wrote:
> 
>> Hello, Igor.
>> 
>> I’ve noticed that Python, NodeJS, PHP thin client implementations were
>> removed from master by you.
>> Can you, please, comment?
>> Why do you make those decision?

Re: Adding metrics of using WAL archive

[Nikolay Izhikov]

LGTM.

Please, proceed with the merge.

> 17 марта 2021 г., в 12:20, ткаленко кирилл  написал(а):
> 
> Hi Nikolay! Can you do an additional review or can we merge?
> 
> 15.03.2021, 08:48, "ткаленко кирилл" :
>> Hi Nikolay! Can you do an additional review or can we merge?
>> 
>> 05.03.2021, 16:33, "Nikolay Izhikov" :
>>>  Yes, definitely.
>>> 
>>>>   5 марта 2021 г., в 16:31, ткаленко кирилл  
>>>> написал(а):
>>>> 
>>>>   Hi Nikolay, can you do a review?
>>>> 
>>>>   02.03.2021, 18:59, "Nikolay Izhikov" :
>>>>>   +1 For this.
>>>>> 
>>>>>>2 марта 2021 г., в 18:32, ткаленко кирилл  
>>>>>> написал(а):
>>>>>> 
>>>>>>Hi, Nikolay!
>>>>>> 
>>>>>>I thought about your proposal and offer the following two metrics:
>>>>>> 
>>>>>>1)The number of bytes logged to the WAL;
>>>>>>2)The number of compressed bytes in the WAL.
>>>>>> 
>>>>>>Monotonically increasing long.
>>>>>> 
>>>>>>WDYT?

Re: [DISCUSSION] Merge APIs of IgniteAuthenticationProcessor and IgniteSecurity

[Nikolay Izhikov]

Hello, Mikhail.

I'm +1 to follow your suggestion.

чт, 18 мар. 2021 г. в 17:53, Mikhail Petrov :

> Hello, Igniters.
>
> As of now, there are two independent APIs related to security:
> 1. IgniteSecurity - handle node/client authentication and authorize all
> operations.
> 2. IgniteAuthenticationProcessor - handle authentication of thin clients
> only.
>
> The main purpose of creating the IgniteAuthenticationProcessor was to
> bring default security implementation in Ignite (see [1]) because
> IgniteSecurity has always had a single implementation that delegates
> authorization and authentication operation to an external security plugin.
>
> But two different APIs that are related to the same leads to security
> checks duplication in code. As of now, it's possible to configure both
> security approaches at the same time, and that is confusing for the
> user. E.g., the user can provide a security plugin and execute ALTER /
> DROP / CREATE commands successfully. In this case, mentioned commands
> will do nothing because they only work with the authentication processor
>
> I propose to merge the two mentioned above security APIs into one based
> on the IgniteSecurity interface.
>
> For this it is proposed:
> 1. Remove an IgniteAuthenticationProcessor that is now treated as an
> independent processor.
> 2. Move the logic of IgniteAuthenticationProcessor into the
> implementation of the security plugin that will be used if
> authentication is enabled via configuration.
> 3. Remove duplication of security checks and leave IgniteSecurity as a
> single security API. As of now, authentication operations are not
> delegated to IgniteAuthenticationProcessor if a security plugin is
> specified. So the overall security behavior from the user side will
> remain intact.
> 4. Remove the AuthorizationContext completely as IgniteSecurity provides
> an API for storing and managing the security contexts.
> 5. Extend GridSecurityPlugin interface with methods that provide the
> ability to manage security users to support existing commands available
> for authentication processor - alter/drop/create through SQL and REST.
>
> As a result, we will make the security-related code more consistent and
> simpler.
>
> Proposed signatures of GridSecurityPlugin methods that provide the
> ability to manage security users:
>
> public void createUser(String login, UserOptions opts) throws
> IgniteCheckedException  Â
> Â
> public void alterUser(String login, UserOptions opts) throws
> IgniteCheckedException
>Â
> public void dropUser(String login) throws IgniteCheckedException
>
> The UserOptions class is a wrapper over EnumMap that maps option values
> ​​to their aliases. This allows the class to be used for both create
> and alter user operations and doesn't break backward compatibility in
> case new options are declared.
>
> Â
> The proposed changes lead to the following compatibility issues:
>
> 1. When a user provides a security plugin and enables authentication -
> in this case, the user will face exceptions during the node start while
> now node starts smoothly. This case makes a little sense because now
> authentication operations are not delegated to
> IgniteAuthenticationProcessor at all if a security plugin is specified.
> 2. The current implementation of IgniteAuthenticationProcessor can
> enable authentication itself if the current node connects to the cluster
> with authentication enabled - this functionality will not be supported.
> The user can easily overcome this by explicitly enabling authentication
> in the configuration on all nodes.
>
>
> The remaining implementation of the IgniteAuthenticationProcessor and
> its general behavior will remain intact. I also propose to keep the
> current callbacks for the IgniteAuthenticationProcessor (e.g.
> IgniteAuthenticationProcessor#cacheProcessorStarted) that are called
> from other managers intact, just skip these calls if the authentication
> is disabled.
>
> WDYT?
>
> Ticket - https://issues.apache.org/jira/browse/IGNITE-14335
> Draft PR - https://github.com/apache/ignite/pull/8892
>
> [1] -
>
> http://apache-ignite-developers.2346864.n4.nabble.com/Username-password-authentication-for-thin-clients-td26058.html
>
> Regards,
> Mikhail.
>
>

Re: [ANNOUNCE] Welcome Ivan Daschinsky as a new committer

[Nikolay Izhikov]

Congrats! Well deserved.

> 13 апр. 2021 г., в 09:34, Zhenya Stanilovsky  
> написал(а):
> 
> 
> Big deal ! Ivan, ignite it !) 
> 
> 
>  
>> The Project Management Committee (PMC) for Apache Ignite has invited
>> Ivan Daschinsky to become a committer and we are pleased to announce that
>> he has accepted.
>> 
>> Ivan made a lot of contributions to Apache Ignite.
>> He helped a lot to improve our Python Thin Client fixing a lot of different
>> bugs and contributing major feature such as asyncio support and C-extension
>> which improved performance significantly for many cases. Thanks to Ivan
>> Python
>> Thin client has become much more stable and production-ready. He also
>> introduced the CMake building system for Ignite C++, and has made a number
>> of
>> other important improvements. Besides the code contributions, Ivan is also
>> an
>> active community member.
>> 
>> Being a committer enables easier contribution to the project since there is
>> no need to go via the patch submission process. This should enable better
>> productivity.
>> 
>> Please join me in welcoming Ivan, and congratulating him on the new role in
>> the Apache Ignite Community.
>> 
>> Best Regards,
>> Igor 
>  
>  
>  
>  

[DISCUSSION] MaxLineLength checkstyle rule

[Nikolay Izhikov]

Hello, Igniters.

Right now, we have a code style rule [1] - the line should fit in 120 
characters.
But, this rule violated in many and many places through code.
I have a plan to add a check style rule to force maximum line length.

For me, personally, 120 characters a bit old-fashioned restriction.
Should we increase the maximum line length to 150 or even 180 characters?

[1] https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines

Re: [DISCUSSION] MaxLineLength checkstyle rule

[Nikolay Izhikov]

Hello, Ilya.

Thanks for the feedback.

140 characters is fine for me.

> 15 апр. 2021 г., в 12:25, Ilya Kasnacheev  
> написал(а):
> 
> Hello!
> 
> Please find attached the distribution of line lengths in the project, in the 
> form of (count, line length).
> 
> I think that we can enforce a hard limit of 140 chars per line. I think that 
> having longer lines is excessive and does not benefit readability.
> 
>  Having a limit of 150 or 180 does not give us much since there's still a 
> long tail which has to be fixed.
> 
> Regards,
> -- 
> Ilya Kasnacheev
> 
> 
> чт, 15 апр. 2021 г. в 11:30, Nikolay Izhikov :
> Hello, Igniters.
> 
> Right now, we have a code style rule [1] - the line should fit in 120 
> characters.
> But, this rule violated in many and many places through code.
> I have a plan to add a check style rule to force maximum line length.
> 
> For me, personally, 120 characters a bit old-fashioned restriction.
> Should we increase the maximum line length to 150 or even 180 characters?
> 
> [1] https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines
> 

Re: [DISCUSSION] MaxLineLength checkstyle rule

[Nikolay Izhikov]

> I think that we should keep the "soft" 120 chars limit in CC, but introduce a 
> hard limit of 140 in checkstyle

Works for me.

> 15 апр. 2021 г., в 14:21, Ilya Kasnacheev  
> написал(а):
> 
> Hello!
> 
> Let me clarify: I think that we should keep the "soft" 120 chars limit in
> CC, but introduce a hard limit of 140 in checkstyle, since it should not be
> too much work or annoy too much.
> 
> In the future we may wish to harmonize the two.
> 
> Regards,
> -- 
> Ilya Kasnacheev
> 
> 
> чт, 15 апр. 2021 г. в 12:37, Ivan Daschinsky :
> 
>> Hi!
>> Personally, I suppose that 120 chars per line is OK. Moreover, many
>> codestyles suggests less chars per line.
>> For example PEP8 recommends 80 (but we use 120 in pyignite and flake8
>> codestyle checks it). Google java codestyle insists on 100.
>> 
>> More than 120 chars is too long as for me and is not convenient for 3-way
>> merges.
>> 
>> чт, 15 апр. 2021 г. в 12:28, Nikolay Izhikov :
>> 
>>> Hello, Ilya.
>>> 
>>> Thanks for the feedback.
>>> 
>>> 140 characters is fine for me.
>>> 
>>>> 15 апр. 2021 г., в 12:25, Ilya Kasnacheev 
>>> написал(а):
>>>> 
>>>> Hello!
>>>> 
>>>> Please find attached the distribution of line lengths in the project,
>> in
>>> the form of (count, line length).
>>>> 
>>>> I think that we can enforce a hard limit of 140 chars per line. I think
>>> that having longer lines is excessive and does not benefit readability.
>>>> 
>>>> Having a limit of 150 or 180 does not give us much since there's still
>>> a long tail which has to be fixed.
>>>> 
>>>> Regards,
>>>> --
>>>> Ilya Kasnacheev
>>>> 
>>>> 
>>>> чт, 15 апр. 2021 г. в 11:30, Nikolay Izhikov :
>>>> Hello, Igniters.
>>>> 
>>>> Right now, we have a code style rule [1] - the line should fit in 120
>>> characters.
>>>> But, this rule violated in many and many places through code.
>>>> I have a plan to add a check style rule to force maximum line length.
>>>> 
>>>> For me, personally, 120 characters a bit old-fashioned restriction.
>>>> Should we increase the maximum line length to 150 or even 180
>> characters?
>>>> 
>>>> [1]
>> https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines
>>>> 
>>> 
>>> 
>> 
>> --
>> Sincerely yours, Ivan Daschinskiy
>> 

[DISCUSSION] Array to BinaryObject serialization

[Nikolay Izhikov]

Hello, Igniters.

Currently, binary marshaller works as follows(Say, we have a class `User` then):

IgniteBinary#toBinary(User)` -> BinaryObject
IgniteBinary#toBinary(User[])` -> Object[]
IgniteBinary#toBinary(Object[])` -> Object[]

This means, that we lose array component type information during binary 
serialization.
AFAIK, it’s a design choice made during binary infrastructure development.

This lead to the following disadvantages:

1. `IgniteBinary` can’t be used as a universal SerDe mechanism.
2. Ignite internals(service grid, .Net calls) contains many tweaks and hacks to 
deal with custom user array and still has many issues [1]

I propose to make breaking changes and fix the custom user array SeDe as 
follows:

1. Implement binary serialization that correctly Ser and Deser array 
using some kind of the wrapper (BinaryArrayWrapper).

IgniteBinary#toBinary(User)` -> BinaryObject 
IgniteBinary#toBinary(User[])` -> BinaryObject
IgniteBinary#toBinary(Object[])` -> BinaryObject

2. Introduce system flag `IGNITE_USE_BINARY_ARRAY` that enables correct 
SerDe of arrays. The default value is false to keep backward compatibility in 
the next Ignite release(2.11).

3. Set  `IGNITE_USE_BINARY_ARRAY` to `true` in the ongoing Ignite 
releases (2.12).

WDYT?

[1] https://issues.apache.org/jira/browse/IGNITE-14299

Re: [DISCUSSION] Array to BinaryObject serialization

[Nikolay Izhikov]

Igniters, 

Want to clarify my proposal about new array store format.
I think we should store array in special binary wrapper that will keep original 
component type

```
public class BinaryArrayWrapper implements BinaryObjectEx, Externalizable {
/** Type ID. */
private int compTypeId;

/** Raw data. */
private String compClsName;

/** Value. */
private Object[] arr;

// Further implementation.
}
```


> 30 апр. 2021 г., в 16:31, Nikolay Izhikov  написал(а):
> 
> Hello, Igniters.
> 
> Currently, binary marshaller works as follows(Say, we have a class `User` 
> then):
> 
> IgniteBinary#toBinary(User)` -> BinaryObject
> IgniteBinary#toBinary(User[])` -> Object[]
> IgniteBinary#toBinary(Object[])` -> Object[]
> 
> This means, that we lose array component type information during binary 
> serialization.
> AFAIK, it’s a design choice made during binary infrastructure development.
> 
> This lead to the following disadvantages:
> 
> 1. `IgniteBinary` can’t be used as a universal SerDe mechanism.
> 2. Ignite internals(service grid, .Net calls) contains many tweaks and hacks 
> to deal with custom user array and still has many issues [1]
> 
> I propose to make breaking changes and fix the custom user array SeDe as 
> follows:
> 
>   1. Implement binary serialization that correctly Ser and Deser array 
> using some kind of the wrapper (BinaryArrayWrapper).
> 
>   IgniteBinary#toBinary(User)` -> BinaryObject 
>   IgniteBinary#toBinary(User[])` -> BinaryObject
>   IgniteBinary#toBinary(Object[])` -> BinaryObject
> 
>   2. Introduce system flag `IGNITE_USE_BINARY_ARRAY` that enables correct 
> SerDe of arrays. The default value is false to keep backward compatibility in 
> the next Ignite release(2.11).
> 
>   3. Set  `IGNITE_USE_BINARY_ARRAY` to `true` in the ongoing Ignite 
> releases (2.12).
> 
> WDYT?
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-14299

Re: [DISCUSSION] Array to BinaryObject serialization

[Nikolay Izhikov]

Hello, Ilya.

Thanks for the feedback!

> For me it sounds like something we would like to do in 3.0

Ignite 3 is a very long way to go, so I prefer to target this fix in Ignite 2.x.

> I think making it default "true" is a breaking change and is not possible in 
> a minor release

Yes, you are correct it is a breaking change.
It seems for me, we all agreed that breaking changes are possible in minor 
releases.

Anyway, if we will decide do not to enable this feature by default it’s OK for 
me.
We still can implement it and improve the binary SerDe mechanism.


> 30 апр. 2021 г., в 17:23, Ilya Kasnacheev  
> написал(а):
> 
> Hello!
> 
> For me it sounds like something we would like to do in 3.0 (if indeed it
> will have arrays as possible value (or key) type), but doing it in 2.x
> raises concerns whether it has enough time left to stabilize.
> 
> Also, I think making it default "true" is a breaking change and is not
> possible in a minor release, case in point,
> IGNITE_BINARY_SORT_OBJECT_FIELDS is still waiting for 3.0 and it is less
> destructive.
> 
> Of course I would also like to hear what other community members think.
> 
> Regards,
> -- 
> Ilya Kasnacheev
> 
> 
> пт, 30 апр. 2021 г. в 17:16, Nikolay Izhikov :
> 
>> Igniters,
>> 
>> Want to clarify my proposal about new array store format.
>> I think we should store array in special binary wrapper that will keep
>> original component type
>> 
>> ```
>> public class BinaryArrayWrapper implements BinaryObjectEx, Externalizable {
>>/** Type ID. */
>>private int compTypeId;
>> 
>>/** Raw data. */
>>    private String compClsName;
>> 
>>/** Value. */
>>private Object[] arr;
>> 
>>// Further implementation.
>> }
>> ```
>> 
>> 
>>> 30 апр. 2021 г., в 16:31, Nikolay Izhikov 
>> написал(а):
>>> 
>>> Hello, Igniters.
>>> 
>>> Currently, binary marshaller works as follows(Say, we have a class
>> `User` then):
>>> 
>>> IgniteBinary#toBinary(User)` -> BinaryObject
>>> IgniteBinary#toBinary(User[])` -> Object[]
>>> IgniteBinary#toBinary(Object[])` -> Object[]
>>> 
>>> This means, that we lose array component type information during binary
>> serialization.
>>> AFAIK, it’s a design choice made during binary infrastructure
>> development.
>>> 
>>> This lead to the following disadvantages:
>>> 
>>> 1. `IgniteBinary` can’t be used as a universal SerDe mechanism.
>>> 2. Ignite internals(service grid, .Net calls) contains many tweaks and
>> hacks to deal with custom user array and still has many issues [1]
>>> 
>>> I propose to make breaking changes and fix the custom user array SeDe as
>> follows:
>>> 
>>>  1. Implement binary serialization that correctly Ser and Deser
>> array using some kind of the wrapper (BinaryArrayWrapper).
>>> 
>>>  IgniteBinary#toBinary(User)` -> BinaryObject
>>>  IgniteBinary#toBinary(User[])` -> BinaryObject
>>>  IgniteBinary#toBinary(Object[])` -> BinaryObject
>>> 
>>>  2. Introduce system flag `IGNITE_USE_BINARY_ARRAY` that enables
>> correct SerDe of arrays. The default value is false to keep backward
>> compatibility in the next Ignite release(2.11).
>>> 
>>>  3. Set  `IGNITE_USE_BINARY_ARRAY` to `true` in the ongoing Ignite
>> releases (2.12).
>>> 
>>> WDYT?
>>> 
>>> [1] https://issues.apache.org/jira/browse/IGNITE-14299
>> 
>> 

Re: [DISCUSSION] MaxLineLength checkstyle rule

[Nikolay Izhikov]

Hello.

> BTW, was there any progress with this?

Patch is ready, please, join the review - 
https://github.com/apache/ignite/pull/9106

> 10 мая 2021 г., в 09:47, Ivan Pavlukhin  написал(а):
> 
> +1 for 140 lines compromise.
> +1 if someone is ready to fix everything to fit 120.
> 
> BTW, was there any progress with this?
> 
> 2021-04-15 21:41 GMT+03:00, Maxim Muzafarov :
>> Folks,
>> 
>> I've briefly checked the total amount of the max length violations:
>> 
>> 120 line length - 5540 violations
>> 130 line length - 1891 violations
>> 140 line length - 895 violations
>> 150 line length - 478 violations
>> 
>> 
>> I think the 140 max line length might be the best option for us.
>> 
>> On Thu, 15 Apr 2021 at 14:51, Ivan Daschinsky  wrote:
>>> 
>>> But super long lines are a real problem while merging. It is super
>>> inconvenient.
>>> 120 chars is a good compromise.
>>> 
>>> чт, 15 апр. 2021 г. в 14:39, Zhenya Stanilovsky
>>> >>> :
>>> 
>>>> 
>>>> Python is not so verbose as java )
>>>> +1 for 140
>>>> 
>>>>> Hi!
>>>>> Personally, I suppose that 120 chars per line is OK. Moreover, many
>>>>> codestyles suggests less chars per line.
>>>>> For example PEP8 recommends 80 (but we use 120 in pyignite and flake8
>>>>> codestyle checks it). Google java codestyle insists on 100.
>>>>> 
>>>>> More than 120 chars is too long as for me and is not convenient for
>>>>> 3-way
>>>>> merges.
>>>>> 
>>>>> чт, 15 апр. 2021 г. в 12:28, Nikolay Izhikov < nizhi...@apache.org >:
>>>>> 
>>>>>> Hello, Ilya.
>>>>>> 
>>>>>> Thanks for the feedback.
>>>>>> 
>>>>>> 140 characters is fine for me.
>>>>>> 
>>>>>>> 15 апр. 2021 г., в 12:25, Ilya Kasnacheev <
>>>>>>> ilya.kasnach...@gmail.com
>>>>> 
>>>>>> написал(а):
>>>>>>> 
>>>>>>> Hello!
>>>>>>> 
>>>>>>> Please find attached the distribution of line lengths in the
>>>>>>> project,
>>>> in
>>>>>> the form of (count, line length).
>>>>>>> 
>>>>>>> I think that we can enforce a hard limit of 140 chars per line. I
>>>> think
>>>>>> that having longer lines is excessive and does not benefit
>>>>>> readability.
>>>>>>> 
>>>>>>> Having a limit of 150 or 180 does not give us much since there's
>>>>>>> still
>>>>>> a long tail which has to be fixed.
>>>>>>> 
>>>>>>> Regards,
>>>>>>> --
>>>>>>> Ilya Kasnacheev
>>>>>>> 
>>>>>>> 
>>>>>>> чт, 15 апр. 2021 г. в 11:30, Nikolay Izhikov < nizhi...@apache.org
>>>>>>>> :
>>>>>>> Hello, Igniters.
>>>>>>> 
>>>>>>> Right now, we have a code style rule [1] - the line should fit in
>>>>>>> 120
>>>>>> characters.
>>>>>>> But, this rule violated in many and many places through code.
>>>>>>> I have a plan to add a check style rule to force maximum line
>>>>>>> length.
>>>>>>> 
>>>>>>> For me, personally, 120 characters a bit old-fashioned
>>>>>>> restriction.
>>>>>>> Should we increase the maximum line length to 150 or even 180
>>>> characters?
>>>>>>> 
>>>>>>> [1]
>>>> https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> --
>>>>> Sincerely yours, Ivan Daschinskiy
>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Sincerely yours, Ivan Daschinskiy
>> 
> 
> 
> -- 
> 
> Best regards,
> Ivan Pavlukhin

Re: [DISCUSSION] Array to BinaryObject serialization

[Nikolay Izhikov]

Hello, 

> However, for internal platform and services implementations we should fix the 
> root cause:
> avoid extra deserialization->serialization pass completely.
> This will also improve performance.

Pavel, thanks for the feedback.
If I understand correctly, your suggestion is to know data size at the start of 
reading service parameters.
Is it correct?

Right now, when the service method invoked we pass an array of parameters 
through platform reader/writer machinery.
On java side parameters read one by one and we don’t know the size of the data 
on the read start.

AFAICU, this will require x2 memory on the .Net or thin client-side.

https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/services/PlatformServices.java#L289


> if we are to break compatibility, I would like to see it done for some really 
> common pain point.

Ilya, can you, please, provide a list of common issues with Ignite that can be 
resolved
only with compatibility breakage?

> 4 мая 2021 г., в 12:58, Ilya Kasnacheev  
> написал(а):
> 
> Hello!
> 
> If we really decide to break some compatibility then Array to BinaryObject
> serialization will be very, very low on my personal list.
> 
> I just don't see how this issue is relevant. I have been reading and
> answering user list for a few years now, and I don't remember a single
> question about storing ConcreteType[] as value and complaining about type
> information loss.
> 
> If you have a good scenario how do we keep persistent store binary
> compatibility here, without adding a lot of new code and still checking for
> both old and new approaches - you can go forward for sure.
> 
> However, it does seem questionable where we have a new wrapper class
> specifically for top level arrays. You can have this wrapper in your own
> client code and it should work OK.
> 
> Bottom line, if we are to break compatibility, I would like to see it done
> for some really common pain point.
> 
> Regards,
> -- 
> Ilya Kasnacheev
> 
> 
> пт, 30 апр. 2021 г. в 17:34, Nikolay Izhikov :
> 
>> Hello, Ilya.
>> 
>> Thanks for the feedback!
>> 
>>> For me it sounds like something we would like to do in 3.0
>> 
>> Ignite 3 is a very long way to go, so I prefer to target this fix in
>> Ignite 2.x.
>> 
>>> I think making it default "true" is a breaking change and is not
>> possible in a minor release
>> 
>> Yes, you are correct it is a breaking change.
>> It seems for me, we all agreed that breaking changes are possible in minor
>> releases.
>> 
>> Anyway, if we will decide do not to enable this feature by default it’s OK
>> for me.
>> We still can implement it and improve the binary SerDe mechanism.
>> 
>> 
>>> 30 апр. 2021 г., в 17:23, Ilya Kasnacheev 
>> написал(а):
>>> 
>>> Hello!
>>> 
>>> For me it sounds like something we would like to do in 3.0 (if indeed it
>>> will have arrays as possible value (or key) type), but doing it in 2.x
>>> raises concerns whether it has enough time left to stabilize.
>>> 
>>> Also, I think making it default "true" is a breaking change and is not
>>> possible in a minor release, case in point,
>>> IGNITE_BINARY_SORT_OBJECT_FIELDS is still waiting for 3.0 and it is less
>>> destructive.
>>> 
>>> Of course I would also like to hear what other community members think.
>>> 
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>> 
>>> 
>>> пт, 30 апр. 2021 г. в 17:16, Nikolay Izhikov :
>>> 
>>>> Igniters,
>>>> 
>>>> Want to clarify my proposal about new array store format.
>>>> I think we should store array in special binary wrapper that will keep
>>>> original component type
>>>> 
>>>> ```
>>>> public class BinaryArrayWrapper implements BinaryObjectEx,
>> Externalizable {
>>>>   /** Type ID. */
>>>>   private int compTypeId;
>>>> 
>>>>   /** Raw data. */
>>>>   private String compClsName;
>>>> 
>>>>   /** Value. */
>>>>   private Object[] arr;
>>>> 
>>>>   // Further implementation.
>>>> }
>>>> ```
>>>> 
>>>> 
>>>>> 30 апр. 2021 г., в 16:31, Nikolay Izhikov 
>>>> написал(а):
>>>>> 
>>>>> Hello, Igniters.
>>>>> 
>>>>> Currently, binary marshaller works as follows(Say, we have a class
>>>>

Re: [DISCUSSION] Array to BinaryObject serialization

[Nikolay Izhikov]

Thanks, Ilya.

Can you put more context on this? 
I don’t familiar with these issues.

> 19 мая 2021 г., в 13:02, Ilya Kasnacheev  
> написал(а):
> 
> Hello!
> 
> Obvious issues are Lazy SQL, Event Driven Services, Sort Binary Object
> Fields.
> 
> Regards,
> -- 
> Ilya Kasnacheev
> 
> 
> ср, 19 мая 2021 г. в 12:56, Nikolay Izhikov :
> 
>> Hello,
>> 
>>> However, for internal platform and services implementations we should
>> fix the root cause:
>>> avoid extra deserialization->serialization pass completely.
>>> This will also improve performance.
>> 
>> Pavel, thanks for the feedback.
>> If I understand correctly, your suggestion is to know data size at the
>> start of reading service parameters.
>> Is it correct?
>> 
>> Right now, when the service method invoked we pass an array of parameters
>> through platform reader/writer machinery.
>> On java side parameters read one by one and we don’t know the size of the
>> data on the read start.
>> 
>> AFAICU, this will require x2 memory on the .Net or thin client-side.
>> 
>> 
>> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/services/PlatformServices.java#L289
>> 
>> 
>>> if we are to break compatibility, I would like to see it done for some
>> really common pain point.
>> 
>> Ilya, can you, please, provide a list of common issues with Ignite that
>> can be resolved
>> only with compatibility breakage?
>> 
>>> 4 мая 2021 г., в 12:58, Ilya Kasnacheev 
>> написал(а):
>>> 
>>> Hello!
>>> 
>>> If we really decide to break some compatibility then Array to
>> BinaryObject
>>> serialization will be very, very low on my personal list.
>>> 
>>> I just don't see how this issue is relevant. I have been reading and
>>> answering user list for a few years now, and I don't remember a single
>>> question about storing ConcreteType[] as value and complaining about type
>>> information loss.
>>> 
>>> If you have a good scenario how do we keep persistent store binary
>>> compatibility here, without adding a lot of new code and still checking
>> for
>>> both old and new approaches - you can go forward for sure.
>>> 
>>> However, it does seem questionable where we have a new wrapper class
>>> specifically for top level arrays. You can have this wrapper in your own
>>> client code and it should work OK.
>>> 
>>> Bottom line, if we are to break compatibility, I would like to see it
>> done
>>> for some really common pain point.
>>> 
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>> 
>>> 
>>> пт, 30 апр. 2021 г. в 17:34, Nikolay Izhikov :
>>> 
>>>> Hello, Ilya.
>>>> 
>>>> Thanks for the feedback!
>>>> 
>>>>> For me it sounds like something we would like to do in 3.0
>>>> 
>>>> Ignite 3 is a very long way to go, so I prefer to target this fix in
>>>> Ignite 2.x.
>>>> 
>>>>> I think making it default "true" is a breaking change and is not
>>>> possible in a minor release
>>>> 
>>>> Yes, you are correct it is a breaking change.
>>>> It seems for me, we all agreed that breaking changes are possible in
>> minor
>>>> releases.
>>>> 
>>>> Anyway, if we will decide do not to enable this feature by default it’s
>> OK
>>>> for me.
>>>> We still can implement it and improve the binary SerDe mechanism.
>>>> 
>>>> 
>>>>> 30 апр. 2021 г., в 17:23, Ilya Kasnacheev 
>>>> написал(а):
>>>>> 
>>>>> Hello!
>>>>> 
>>>>> For me it sounds like something we would like to do in 3.0 (if indeed
>> it
>>>>> will have arrays as possible value (or key) type), but doing it in 2.x
>>>>> raises concerns whether it has enough time left to stabilize.
>>>>> 
>>>>> Also, I think making it default "true" is a breaking change and is not
>>>>> possible in a minor release, case in point,
>>>>> IGNITE_BINARY_SORT_OBJECT_FIELDS is still waiting for 3.0 and it is
>> less
>>>>> destructive.
>>>>> 
>>>>> Of course I would also like to hear what other community members think.
>>>>> 
>>>>

Re: [DISCUSSION] Array to BinaryObject serialization

[Nikolay Izhikov]

Igniters.

Just to clarify the issue:

```
public class BinaryObjectTest extends GridCommonAbstractTest {
/** */
@Test
public void testArray() throws Exception {
Ignite ign = startGrid();

IgniteCache cache = ign.createCache("my-cache");

cache.put(1, new TestClass1[] {new TestClass1(), new TestClass1()});
TestClass1[] obj = cache.get(1); //This will fail with 
ClassCastException.

assertEquals(TestClass1[].class, obj.getClass());
}
}
```

> 19 мая 2021 г., в 13:04, Nikolay Izhikov  написал(а):
> 
> Thanks, Ilya.
> 
> Can you put more context on this? 
> I don’t familiar with these issues.
> 
>> 19 мая 2021 г., в 13:02, Ilya Kasnacheev  
>> написал(а):
>> 
>> Hello!
>> 
>> Obvious issues are Lazy SQL, Event Driven Services, Sort Binary Object
>> Fields.
>> 
>> Regards,
>> -- 
>> Ilya Kasnacheev
>> 
>> 
>> ср, 19 мая 2021 г. в 12:56, Nikolay Izhikov :
>> 
>>> Hello,
>>> 
>>>> However, for internal platform and services implementations we should
>>> fix the root cause:
>>>> avoid extra deserialization->serialization pass completely.
>>>> This will also improve performance.
>>> 
>>> Pavel, thanks for the feedback.
>>> If I understand correctly, your suggestion is to know data size at the
>>> start of reading service parameters.
>>> Is it correct?
>>> 
>>> Right now, when the service method invoked we pass an array of parameters
>>> through platform reader/writer machinery.
>>> On java side parameters read one by one and we don’t know the size of the
>>> data on the read start.
>>> 
>>> AFAICU, this will require x2 memory on the .Net or thin client-side.
>>> 
>>> 
>>> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/services/PlatformServices.java#L289
>>> 
>>> 
>>>> if we are to break compatibility, I would like to see it done for some
>>> really common pain point.
>>> 
>>> Ilya, can you, please, provide a list of common issues with Ignite that
>>> can be resolved
>>> only with compatibility breakage?
>>> 
>>>> 4 мая 2021 г., в 12:58, Ilya Kasnacheev 
>>> написал(а):
>>>> 
>>>> Hello!
>>>> 
>>>> If we really decide to break some compatibility then Array to
>>> BinaryObject
>>>> serialization will be very, very low on my personal list.
>>>> 
>>>> I just don't see how this issue is relevant. I have been reading and
>>>> answering user list for a few years now, and I don't remember a single
>>>> question about storing ConcreteType[] as value and complaining about type
>>>> information loss.
>>>> 
>>>> If you have a good scenario how do we keep persistent store binary
>>>> compatibility here, without adding a lot of new code and still checking
>>> for
>>>> both old and new approaches - you can go forward for sure.
>>>> 
>>>> However, it does seem questionable where we have a new wrapper class
>>>> specifically for top level arrays. You can have this wrapper in your own
>>>> client code and it should work OK.
>>>> 
>>>> Bottom line, if we are to break compatibility, I would like to see it
>>> done
>>>> for some really common pain point.
>>>> 
>>>> Regards,
>>>> --
>>>> Ilya Kasnacheev
>>>> 
>>>> 
>>>> пт, 30 апр. 2021 г. в 17:34, Nikolay Izhikov :
>>>> 
>>>>> Hello, Ilya.
>>>>> 
>>>>> Thanks for the feedback!
>>>>> 
>>>>>> For me it sounds like something we would like to do in 3.0
>>>>> 
>>>>> Ignite 3 is a very long way to go, so I prefer to target this fix in
>>>>> Ignite 2.x.
>>>>> 
>>>>>> I think making it default "true" is a breaking change and is not
>>>>> possible in a minor release
>>>>> 
>>>>> Yes, you are correct it is a breaking change.
>>>>> It seems for me, we all agreed that breaking changes are possible in
>>> minor
>>>>> releases.
>>>>> 
>>>>> Anyway, if we will decide do not to enable this feature by default it’s
>>> OK
>>>>> for me.
>>>>> We still can imp

Re: [DISCUSSION] Array to BinaryObject serialization

[Nikolay Izhikov]

Ilya,

> Maybe we should just automate that, e.g., whenever user stores Type[], always 
> store one-field ArrayWrapper object, and automatically unwrap it on get()

Yes, I’m talking about this opportunity from the beginning of the thread.

>  1. Implement binary serialization that correctly Ser and Deser array using 
> some kind of the wrapper (BinaryArrayWrapper).


> 19 мая 2021 г., в 14:05, Ilya Kasnacheev  
> написал(а):
> 
> Hello!
> 
> Yes, it does not look pretty, I agree. But I'm saying that I've not
> encountered this issue on the user list before, and it can probably be
> easily countered by storing some one-field ArrayWrapper object in the cache
> as value.
> 
> Maybe we should just automate that, e.g., whenever user stores Type[],
> always store one-field ArrayWrapper object, and automatically unwrap it on
> get()
> This way we may not even need any changes to storage format, if binary
> marshaller does not mind.
> 
> Regards,
> 
> Regards,
> -- 
> Ilya Kasnacheev
> 
> 
> ср, 19 мая 2021 г. в 13:33, Nikolay Izhikov :
> 
>> Igniters.
>> 
>> Just to clarify the issue:
>> 
>> ```
>> public class BinaryObjectTest extends GridCommonAbstractTest {
>>/** */
>>@Test
>>public void testArray() throws Exception {
>>Ignite ign = startGrid();
>> 
>>IgniteCache cache =
>> ign.createCache("my-cache");
>> 
>>cache.put(1, new TestClass1[] {new TestClass1(), new
>> TestClass1()});
>>    TestClass1[] obj = cache.get(1); //This will fail with
>> ClassCastException.
>> 
>>assertEquals(TestClass1[].class, obj.getClass());
>>}
>> }
>> ```
>> 
>>> 19 мая 2021 г., в 13:04, Nikolay Izhikov 
>> написал(а):
>>> 
>>> Thanks, Ilya.
>>> 
>>> Can you put more context on this?
>>> I don’t familiar with these issues.
>>> 
>>>> 19 мая 2021 г., в 13:02, Ilya Kasnacheev 
>> написал(а):
>>>> 
>>>> Hello!
>>>> 
>>>> Obvious issues are Lazy SQL, Event Driven Services, Sort Binary Object
>>>> Fields.
>>>> 
>>>> Regards,
>>>> --
>>>> Ilya Kasnacheev
>>>> 
>>>> 
>>>> ср, 19 мая 2021 г. в 12:56, Nikolay Izhikov :
>>>> 
>>>>> Hello,
>>>>> 
>>>>>> However, for internal platform and services implementations we should
>>>>> fix the root cause:
>>>>>> avoid extra deserialization->serialization pass completely.
>>>>>> This will also improve performance.
>>>>> 
>>>>> Pavel, thanks for the feedback.
>>>>> If I understand correctly, your suggestion is to know data size at the
>>>>> start of reading service parameters.
>>>>> Is it correct?
>>>>> 
>>>>> Right now, when the service method invoked we pass an array of
>> parameters
>>>>> through platform reader/writer machinery.
>>>>> On java side parameters read one by one and we don’t know the size of
>> the
>>>>> data on the read start.
>>>>> 
>>>>> AFAICU, this will require x2 memory on the .Net or thin client-side.
>>>>> 
>>>>> 
>>>>> 
>> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/platform/services/PlatformServices.java#L289
>>>>> 
>>>>> 
>>>>>> if we are to break compatibility, I would like to see it done for some
>>>>> really common pain point.
>>>>> 
>>>>> Ilya, can you, please, provide a list of common issues with Ignite that
>>>>> can be resolved
>>>>> only with compatibility breakage?
>>>>> 
>>>>>> 4 мая 2021 г., в 12:58, Ilya Kasnacheev 
>>>>> написал(а):
>>>>>> 
>>>>>> Hello!
>>>>>> 
>>>>>> If we really decide to break some compatibility then Array to
>>>>> BinaryObject
>>>>>> serialization will be very, very low on my personal list.
>>>>>> 
>>>>>> I just don't see how this issue is relevant. I have been reading and
>>>>>> answering user list for a few years now, and I don't remember a single
>>>>>> question about storing ConcreteType[] as value and complaining about
>> type
>>>>

Re: [DISCUSSION] Array to BinaryObject serialization

[Nikolay Izhikov]

Ilya.

Actually, current behaviour described even in documentation [1]


> Note that not all objects are converted to the binary object format. The 
> following classes are never converted (e.g., the toBinary(Object) method 
> returns the original object, and instances of these classes are stored 
> without changes):
...
> ... arrays of objects (but the objects inside them are reconverted if they 
> are binary)

[1] 
https://ignite.apache.org/docs/latest/key-value-api/binary-objects#enabling-binary-mode-for-caches

> 19 мая 2021 г., в 14:12, Ilya Kasnacheev  
> написал(а):
> 
> Hello!
> 
> Why do you need to take compatibility into account here at all?
> You can start writing entries to cache in new format while reading both old
> and new format.
> 
> I consider returning Object[] instead of ConcreteType[] a bug so it's
> totally OK to start returning the "better" ConcreteType[] instead.
> 
> I think you can just fix the bug while preserving accessibility of old
> (pre-bugfix) data as it was pre-2.11.
> 
> Regards,
> -- 
> Ilya Kasnacheev
> 
> 
> ср, 19 мая 2021 г. в 14:08, Nikolay Izhikov :
> 
>> Ilya,
>> 
>>> Maybe we should just automate that, e.g., whenever user stores Type[],
>> always store one-field ArrayWrapper object, and automatically unwrap it on
>> get()
>> 
>> Yes, I’m talking about this opportunity from the beginning of the thread.
>> 
>>> 1. Implement binary serialization that correctly Ser and Deser array
>> using some kind of the wrapper (BinaryArrayWrapper).
>> 
>> 
>>> 19 мая 2021 г., в 14:05, Ilya Kasnacheev 
>> написал(а):
>>> 
>>> Hello!
>>> 
>>> Yes, it does not look pretty, I agree. But I'm saying that I've not
>>> encountered this issue on the user list before, and it can probably be
>>> easily countered by storing some one-field ArrayWrapper object in the
>> cache
>>> as value.
>>> 
>>> Maybe we should just automate that, e.g., whenever user stores Type[],
>>> always store one-field ArrayWrapper object, and automatically unwrap it
>> on
>>> get()
>>> This way we may not even need any changes to storage format, if binary
>>> marshaller does not mind.
>>> 
>>> Regards,
>>> 
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>> 
>>> 
>>> ср, 19 мая 2021 г. в 13:33, Nikolay Izhikov :
>>> 
>>>> Igniters.
>>>> 
>>>> Just to clarify the issue:
>>>> 
>>>> ```
>>>> public class BinaryObjectTest extends GridCommonAbstractTest {
>>>>   /** */
>>>>   @Test
>>>>   public void testArray() throws Exception {
>>>>   Ignite ign = startGrid();
>>>> 
>>>>   IgniteCache cache =
>>>> ign.createCache("my-cache");
>>>> 
>>>>   cache.put(1, new TestClass1[] {new TestClass1(), new
>>>> TestClass1()});
>>>>   TestClass1[] obj = cache.get(1); //This will fail with
>>>> ClassCastException.
>>>> 
>>>>   assertEquals(TestClass1[].class, obj.getClass());
>>>>   }
>>>> }
>>>> ```
>>>> 
>>>>> 19 мая 2021 г., в 13:04, Nikolay Izhikov 
>>>> написал(а):
>>>>> 
>>>>> Thanks, Ilya.
>>>>> 
>>>>> Can you put more context on this?
>>>>> I don’t familiar with these issues.
>>>>> 
>>>>>> 19 мая 2021 г., в 13:02, Ilya Kasnacheev 
>>>> написал(а):
>>>>>> 
>>>>>> Hello!
>>>>>> 
>>>>>> Obvious issues are Lazy SQL, Event Driven Services, Sort Binary Object
>>>>>> Fields.
>>>>>> 
>>>>>> Regards,
>>>>>> --
>>>>>> Ilya Kasnacheev
>>>>>> 
>>>>>> 
>>>>>> ср, 19 мая 2021 г. в 12:56, Nikolay Izhikov :
>>>>>> 
>>>>>>> Hello,
>>>>>>> 
>>>>>>>> However, for internal platform and services implementations we
>> should
>>>>>>> fix the root cause:
>>>>>>>> avoid extra deserialization->serialization pass completely.
>>>>>>>> This will also improve performance.
>>>>>>> 
>>>>>>> Pavel, thanks for the feedback.
>>>>>>> If I understand correctly, your

[DISCUSSION] Code style. Variable abbrevations

[Nikolay Izhikov]

Hello, Igniters.

Right now, we have the rule to use some predefined list of abbrevation for 
variable names [1].
Some of the reviewers ask to follow this rule strictly.

> It is required to use abbreviated form for code consistency.

I tried to implement this rule in form of checkstyle check [2] and it seems 
like many of use doesn’t follow this requirement.
My check found 4124 violation in core module.

Should we make this rule optional in documentation or should we remove it 
completely?
Or should someone proceed and fix all the violations?

WDYT?


Example of output:
```
[ERROR] 
/Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsWithTtlTest.java:94:
 Abbrevation should be used for CACHE_NAME_LOCAL_ATOMIC! Please, use loc, 
instead of LOCAL [IgniteAbbrevationsRule]
[ERROR] 
/Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsWithTtlTest.java:97:
 Abbrevation should be used for CACHE_NAME_LOCAL_TX! Please, use loc, instead 
of LOCAL [IgniteAbbrevationsRule]
[ERROR] 
/Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsWithTtlTest.java:264:
 Abbrevation should be used for checkpointManager! Please, use mgr, instead of 
Manager [IgniteAbbrevationsRule]
[ERROR] 
/Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsPartitionPreloadTest.java:63:
 Abbrevation should be used for DEFAULT_REGION! Please, use dflt, instead of 
DEFAULT [IgniteAbbrevationsRule]
[ERROR] 
/Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsWholeClusterRestartTest.java:47:
 Abbrevation should be used for ENTRIES_COUNT! Please, use cnt, instead of 
COUNT [IgniteAbbrevationsRule]
[ERROR] 
/Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsRebalancingOnNotStableTopologyTest.java:49:
 Abbrevation should be used for CHECKPOINT_FREQUENCY! Please, use freq, instead 
of FREQUENCY [IgniteAbbrevationsRule]
[ERROR] 
/Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsTransactionsHangTest.java:75:
 Abbrevation should be used for MAX_KEY_COUNT! Please, use cnt, instead of 
COUNT [IgniteAbbrevationsRule]
[ERROR] 
/Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsTransactionsHangTest.java:78:
 Abbrevation should be used for CHECKPOINT_FREQUENCY! Please, use freq, instead 
of FREQUENCY [IgniteAbbrevationsRule]
[ERROR] 
/Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/SlowHistoricalRebalanceSmallHistoryTest.java:57:
 Abbrevation should be used for SUPPLY_MESSAGE_LATCH! Please, use msg, instead 
of MESSAGE [IgniteAbbrevationsRule]
[ERROR] 
/Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgniteLogicalRecoveryTest.java:74:
 Abbrevation should be used for SHARED_GROUP_NAME! Please, use grp, instead of 
GROUP [IgniteAbbrevationsRule]
[ERROR] 
/Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgniteLogicalRecoveryTest.java:200:
 Abbrevation should be used for cacheLoader! Please, use ldr, instead of Loader 
[IgniteAbbrevationsRule]
```

[1] 
https://cwiki.apache.org/confluence/display/IGNITE/Abbreviation+Rules#AbbreviationRules-VariableAbbreviation
[2] https://github.com/apache/ignite/pull/9153

Re: [DISCUSSION] Code style. Variable abbrevations

[Nikolay Izhikov]

Hello, guys.

Thanks for the feedback. 

Dmitry,

> Manual rule enforcement saves a couple of symbols in code

I’m talking about automatic check.
We can implement it easily.
So, if we decide to keep this rule all we need is to fix current violations 
(several thousand!).
After it, checkstyle will automatically enforce this rule.
As you may know, currently, any PR checked according to our checkstyle rules.
Please, take a look at little green check sign after PR name.


> 5 июня 2021 г., в 00:57, Dmitry Pavlov  написал(а):
> 
> +1 for removal. Cnt and count both seem to be fine.
> 
> Manual rule enforcement saves a couple of symbols in code, but requires some 
> time from both contributor and reviewer.
> 
> Sincerely,
> Dmitriy Pavlov
> 
> On 2021/06/04 19:18:37, Pavel Tupitsyn  wrote: 
>> In my opinion, we should remove this rule.
>> Looks like a waste of time. freq or frequency, cnt or count, it is fine
>> either way.
>> 
>> On Fri, Jun 4, 2021 at 7:39 PM Nikolay Izhikov  wrote:
>> 
>>> Hello, Igniters.
>>> 
>>> Right now, we have the rule to use some predefined list of abbrevation for
>>> variable names [1].
>>> Some of the reviewers ask to follow this rule strictly.
>>> 
>>>> It is required to use abbreviated form for code consistency.
>>> 
>>> I tried to implement this rule in form of checkstyle check [2] and it
>>> seems like many of use doesn’t follow this requirement.
>>> My check found 4124 violation in core module.
>>> 
>>> Should we make this rule optional in documentation or should we remove it
>>> completely?
>>> Or should someone proceed and fix all the violations?
>>> 
>>> WDYT?
>>> 
>>> 
>>> Example of output:
>>> ```
>>> [ERROR]
>>> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsWithTtlTest.java:94:
>>> Abbrevation should be used for CACHE_NAME_LOCAL_ATOMIC! Please, use loc,
>>> instead of LOCAL [IgniteAbbrevationsRule]
>>> [ERROR]
>>> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsWithTtlTest.java:97:
>>> Abbrevation should be used for CACHE_NAME_LOCAL_TX! Please, use loc,
>>> instead of LOCAL [IgniteAbbrevationsRule]
>>> [ERROR]
>>> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsWithTtlTest.java:264:
>>> Abbrevation should be used for checkpointManager! Please, use mgr, instead
>>> of Manager [IgniteAbbrevationsRule]
>>> [ERROR]
>>> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsPartitionPreloadTest.java:63:
>>> Abbrevation should be used for DEFAULT_REGION! Please, use dflt, instead of
>>> DEFAULT [IgniteAbbrevationsRule]
>>> [ERROR]
>>> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsWholeClusterRestartTest.java:47:
>>> Abbrevation should be used for ENTRIES_COUNT! Please, use cnt, instead of
>>> COUNT [IgniteAbbrevationsRule]
>>> [ERROR]
>>> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsRebalancingOnNotStableTopologyTest.java:49:
>>> Abbrevation should be used for CHECKPOINT_FREQUENCY! Please, use freq,
>>> instead of FREQUENCY [IgniteAbbrevationsRule]
>>> [ERROR]
>>> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsTransactionsHangTest.java:75:
>>> Abbrevation should be used for MAX_KEY_COUNT! Please, use cnt, instead of
>>> COUNT [IgniteAbbrevationsRule]
>>> [ERROR]
>>> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsTransactionsHangTest.java:78:
>>> Abbrevation should be used for CHECKPOINT_FREQUENCY! Please, use freq,
>>> instead of FREQUENCY [IgniteAbbrevationsRule]
>>> [ERROR]
>>> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/SlowHistoricalRebalanceSmallHistoryTest.java:57:
>>> Abbrevation should be used for SUPPLY_MESSAGE_LATCH! Please, use msg,
>>> instead of MESSAGE [IgniteAbbrevationsRule]
>>> [ERROR]
>>> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/o

Re: [DISCUSSION] Code style. Variable abbrevations

[Nikolay Izhikov]

Hello, Anton, Alexei.

Thanks for the feedback.

Personally, I’m pretty happy current abbreviation rules too.
Let see what we can do to make our codebase even more consistent.

> 7 июня 2021 г., в 13:23, Alexei Scherbakov  
> написал(а):
> 
> -1
> Common abbrevs add quality to the code.
> 
> пн, 7 июн. 2021 г. в 12:38, Anton Vinogradov :
> 
>> -1 here.
>> We can fix the code and set up the rule.
>> 
>> This will help to prevent having a weird abbreviation like "mess" (from
>> "message") or "ign" (from "Ignite").
>> Also, the abbreviations list (hardcoded at IDEA plugin) allows to have same
>> names across the whole code, this should simplify the reading.
>> 
>> On Sat, Jun 5, 2021 at 10:49 PM Valentin Kulichenko <
>> valentin.kuliche...@gmail.com> wrote:
>> 
>>> I also support removing this requirement. It’s not the first time someone
>>> brings this up, and so far we haven’t been able to fix it. Not worth it
>> in
>>> my view.
>>> 
>>> -Val
>>> 
>>> On Sat, Jun 5, 2021 at 11:54 AM Nikolay Izhikov 
>>> wrote:
>>> 
>>>> Hello, guys.
>>>> 
>>>> Thanks for the feedback.
>>>> 
>>>> Dmitry,
>>>> 
>>>>> Manual rule enforcement saves a couple of symbols in code
>>>> 
>>>> I’m talking about automatic check.
>>>> We can implement it easily.
>>>> So, if we decide to keep this rule all we need is to fix current
>>>> violations (several thousand!).
>>>> After it, checkstyle will automatically enforce this rule.
>>>> As you may know, currently, any PR checked according to our checkstyle
>>>> rules.
>>>> Please, take a look at little green check sign after PR name.
>>>> 
>>>> 
>>>>> 5 июня 2021 г., в 00:57, Dmitry Pavlov 
>>> написал(а):
>>>>> 
>>>>> +1 for removal. Cnt and count both seem to be fine.
>>>>> 
>>>>> Manual rule enforcement saves a couple of symbols in code, but
>> requires
>>>> some time from both contributor and reviewer.
>>>>> 
>>>>> Sincerely,
>>>>> Dmitriy Pavlov
>>>>> 
>>>>> On 2021/06/04 19:18:37, Pavel Tupitsyn  wrote:
>>>>>> In my opinion, we should remove this rule.
>>>>>> Looks like a waste of time. freq or frequency, cnt or count, it is
>>> fine
>>>>>> either way.
>>>>>> 
>>>>>> On Fri, Jun 4, 2021 at 7:39 PM Nikolay Izhikov >> 
>>>> wrote:
>>>>>> 
>>>>>>> Hello, Igniters.
>>>>>>> 
>>>>>>> Right now, we have the rule to use some predefined list of
>>> abbrevation
>>>> for
>>>>>>> variable names [1].
>>>>>>> Some of the reviewers ask to follow this rule strictly.
>>>>>>> 
>>>>>>>> It is required to use abbreviated form for code consistency.
>>>>>>> 
>>>>>>> I tried to implement this rule in form of checkstyle check [2] and
>> it
>>>>>>> seems like many of use doesn’t follow this requirement.
>>>>>>> My check found 4124 violation in core module.
>>>>>>> 
>>>>>>> Should we make this rule optional in documentation or should we
>>> remove
>>>> it
>>>>>>> completely?
>>>>>>> Or should someone proceed and fix all the violations?
>>>>>>> 
>>>>>>> WDYT?
>>>>>>> 
>>>>>>> 
>>>>>>> Example of output:
>>>>>>> ```
>>>>>>> [ERROR]
>>>>>>> 
>>>> 
>>> 
>> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsWithTtlTest.java:94:
>>>>>>> Abbrevation should be used for CACHE_NAME_LOCAL_ATOMIC! Please, use
>>>> loc,
>>>>>>> instead of LOCAL [IgniteAbbrevationsRule]
>>>>>>> [ERROR]
>>>>>>> 
>>>> 
>>> 
>> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsWithTtlTest.java:97:
>>>>>>> Abbrevation should be used for CACHE_NAME_LOCAL_TX

Re: [DISCUSSION] Release python-0.5.0

[Nikolay Izhikov]

+1

> 9 июня 2021 г., в 09:48, Ivan Daschinsky  написал(а):
> 
> Hi, folks!
> 
> I suppose, that it's time to release the next version of python thin
> client, because we are ready to ship new cool features.
> 
> 1. Expiry policy (TTL) support (merged to master) [1]
> 2. Transactions (patch available) [2]
> 3. As part of tx api, huge performance improvement of asyncio [3]
> 4. Enable partition awareness by default
> 
> Also, few bugs have been fixed.
> 
> There is a full list of tickets for this release [4]. Most of them are
> resolved, transactions [2] are in PA state, unresolved tickets are
> trivial (documentation and enabling PA by default). I suppose
> that we can freeze scope at the end of next week and prepare RC for voting.
> 
> WDYT?
> 
> [1] -- https://issues.apache.org/jira/browse/IGNITE-14595
> [2] -- https://issues.apache.org/jira/browse/IGNITE-12467
> [3] -- 
> https://issues.apache.org/jira/browse/IGNITE-12467?focusedCommentId=17359293&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17359293
> [4] -- 
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%20python-0.5.0

Re: [DISCUSSION] Code style. Variable abbrevations

[Nikolay Izhikov]

> No validation in the world prevent me from typing manually "mess" instead of 
> "msg"/«message"

Code review will do.

> btw "svc" is more common imo

Agree. I think you can start a discussion and change some abbrevs if you wish.


> 17 июня 2021 г., в 10:23, Ivan Daschinsky  написал(а):
> 
> I'm sorry, but the rule is not strict and, moreover, not clear enough for
> constans. See here [1]
> ```
> Type and method names are usually not abbreviated (except for the
> well-accepted abbreviations such as EOF, Impl, Fifo, etc.).
> 
> Abbreviation applies to local variables, method parameters, class fields
> and in **some cases public static fileds** (constants) of the class.
> ```
> But there is not any list that contains these cases. I suppose, that
> constant naming should not be abbreviated.
> As I see, the most cases of violations, described in initial post, are
> about constant's namings.
> 
> I suppose that we should define strict and clear rules about constants
> naming.
> 
> [1] -- https://cwiki.apache.org/confluence/display/IGNITE/Abbreviation+Rules
> 
> 
> чт, 17 июн. 2021 г. в 10:10, Konstantin Orlov :
> 
>> Enforced validation doesn't guarantee code consistency. No validation in
>> the world prevent me from typing manually "mess" instead of "msg"/"message"
>> or "svc" instead of "srvc"/"service" (btw "svc" is more common imo).
>> 
>> And the fact that someone name a variable "count" instead of "cnt" doesn't
>> make the whole project immature.
>> 
>> --
>> Regards,
>> Konstantin Orlov
>> 
>> 
>> 
>> 
>>> On 17 Jun 2021, at 08:34, Ivan Pavlukhin  wrote:
>>> 
>>> Hi Nikolay,
>>> 
>>> Thanks, it is rather interesting.
>>> 
>>> Do we all agree that using different conventions for different code
>>> packages does not break "consistency"? Or did I get something wrong?
>>> 
>>> 2021-06-17 7:12 GMT+03:00, Николай Ижиков :
>>>> Hello, Ivan.
>>>> 
>>>> We can create checkstyle rule to enforce usage of abbreviations.
>>>> Internal/public code differs by package.
>>>> 
>>>> PoC of rule [1]
>>>> 
>>>> [1] https://github.com/apache/ignite/pull/9153
>>>> 
>>>>> 17 июня 2021 г., в 07:01, Ivan Pavlukhin 
>>>>> написал(а):
>>>>> 
>>>>> Nikita,
>>>>> 
>>>>> Do you have a plan in your mind how to make Ignite codebase consistent?
>>>>> 
>>>>> AFAIR, we had it intentionally inconsistent for a long time at least
>>>>> for one sake: for internal code we used special conventions (e.g.
>>>>> abbreviations, shorten getters) and common Java conventions for public
>>>>> API and examples (e.g. no abbreviations and usual getters).
>>>>> 
>>>>> 2021-06-16 23:37 GMT+03:00, Nikita Ivanov :
>>>>>> Consistency is what makes it easier to contribute to the project and
>>>>>> attract new members. Consistency implies strong, well defined and
>>>>>> universally enforced rules. Just because we have some individuals who
>>>>>> are lazy or inexperienced - it does not mean that the entire project
>>>>>> should relax the basic-level engineering discipline.
>>>>>> 
>>>>>> On a personal node - nothing screams "immaturity" louder that a code
>>>>>> that uses inconsistent naming, commenting, code style & organization,
>>>>>> etc.
>>>>>> --
>>>>>> Nikita Ivanov
>>>>>> 
>>>>>>> On Wed, Jun 16, 2021 at 5:56 AM Andrey Gura 
>> wrote:
>>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I understand that this rule seems too strict or may be weird. But
>>>>>>> removing the rule could lead to review comments like "use future
>>>>>>> instead of fut". So my proposal is to change rule from "required" to
>>>>>>> "recommended".
>>>>>>> 
>>>>>>> On Wed, Jun 16, 2021 at 2:49 PM Valentin Kulichenko
>>>>>>>  wrote:
>>>>>>>> 
>>>>>>>> Konstantin, thanks for chiming in.
>>>>>>>> 
>>>>>>>&

Re: [VOTE] Release pyignite 0.5.0-rc1

[Nikolay Izhikov]

+1 (binding)

> 18 июня 2021 г., в 14:41, Shishkov Ilya  написал(а):
> 
>> in-memory cluster 3x servers
> 
> I forgot to write about the Ignite version: checked with Ignite-2.10 binary
> release.
> 
> пт, 18 июн. 2021 г. в 14:37, Shishkov Ilya :
> 
>> +1 from me.
>> 
>> Checked on Win10:
>> - Python 3.8.8
>> - pip install pyignite-0.5.0-cp38-cp38-win_amd64.whl
>> - in-memory cluster 3x servers
>> 
>> All examples from 'pyignite-0.5.0.zip' work without errors. Errors in
>> server logs not found.
>> Also checked for correct reconnection in the failover example.
>> 
>> пт, 18 июн. 2021 г. в 13:58, Stephen Darlington <
>> stephen.darling...@gridgain.com>:
>> 
>>> You’re right, it’s documented and works as described. My bad.
>>> 
 On 18 Jun 2021, at 11:24, Igor Sapego  wrote:
 
 Well,
 
 This behaviour maybe is not obvious, but it seems to be safe and
 should not cause user issues. On the other hand, if we change this
 behavior now, it may lead to implicit disable of SSL for users that
 updated their client which seems more dangerous to me.
 
 So I'd keep the current behaviour.
 
 Best Regards,
 Igor
 
 
 On Fri, Jun 18, 2021 at 1:17 PM Ivan Daschinsky 
>>> wrote:
 
> I suppose that we should not cancel this release, despite the fact that
> this is not obvious behaviour. This is not a regression, this
>>> behaviour is
> documented and this behaviour lasts for few years. Lets remove it, if
>>> the
> majority are against it, but in the next release.
> 
> пт, 18 июн. 2021 г. в 13:08, Ivan Daschinsky :
> 
 we explicitly set use_ssl=True.
>> Sorry, typo -- implicitly
>> 
>> пт, 18 июн. 2021 г. в 12:59, Ivan Daschinsky :
>> 
>>> AHA! I see, this is not a bug -- this is a feature. If you pass
>>> username
>>> and password, we explicitly set use_ssl=True. So if your cluster is
>>> configured without ssl but with authentication,
>>> you should explicitly pass use_ssl=False.
>>> 
>>> This behaviour is from old version and I suppose it is correct. Who
> wants
>>> authentication that sent without encryption?
>>> 
>>> пт, 18 июн. 2021 г. в 12:54, Ivan Daschinsky :
>>> 
 Just rechecked test on release branch, add extra check with cluster
 activation and putting some data -- everything works ok.
>>> Authentication
 enabled, persistence enabled,
 with and without ssl. Could you please provide you ignite config and
 your code.
 
 пт, 18 июн. 2021 г. в 12:46, Ivan Daschinsky :
 
> There is a test for it.
> 
> пт, 18 июн. 2021 г. в 12:30, Stephen Darlington <
> stephen.darling...@gridgain.com>:
> 
>> Oh… can someone else check this: it appears that authenticated
>> connections fail.
>> 
>> With Ignite 2.10 the connection times-out:
>> 
>> 
> 
>>> [10:28:58,015][WARNING][grid-timeout-worker-#22][ClientListenerNioListener]
>> Unable to perform handshake within timeout [timeout=1,
> remoteAddr=/
>> 127.0.0.1:54044]
>> 
>> Didn’t try this with 0.4.0 so not sure if it’s a regression, but
>>> it’s
>> not great.
>> 
>>> On 18 Jun 2021, at 09:36, Stephen Darlington <
>> stephen.darling...@gridgain.com> wrote:
>>> 
>>> +1
>>> 
>>> Checked on macOS, played with the new expiry APIs and a bunch of
>> thefundamentals.
>>> 
 On 17 Jun 2021, at 12:46, Pavel Tupitsyn 
>> wrote:
 
 +1
 
 Checked pip install from tar.gz on Python 3.8 on Ubuntu 20.04,
>>> ran
>> some of
 the examples.
 
 On Thu, Jun 17, 2021 at 2:32 PM Igor Sapego >>> 
>> wrote:
 
> +1 from me
> 
> Best Regards,
> Igor
> 
> 
> On Thu, Jun 17, 2021 at 12:10 PM Ivan Daschinsky <
>> ivanda...@gmail.com>
> wrote:
> 
>> +1 From me
>> Checked on Ubuntu 20.04 and windows 10
>> 1. Installation from wheels for pythons 3.6 3.7 3.8 3.9
>> 2. Native module work
>> 3. Examples
>> 
>> Checked on Ubuntu 20.04 building from source package and
>>> correct
>> work of
>> result package.
>> 
>> Checked all sha256 checksums and gpg signatures.
>> 
>> Let's extend voting period till June 18, 15:00 UTC
>> 
>> 
>> 
>> ср, 16 июн. 2021 г. в 17:34, Ivan Daschinsky <
>> ivanda...@apache.org>:
>> 
>>> The vote will end at June, 17 15:00 UTC.
>>> 
>>> ср, 16 июн. 2021 г. в 17:33, Ivan

Re: [DISCUSSION] Code style. Variable abbrevations

[Nikolay Izhikov]

ocumented code. Ignite has a big
>> lack
>>> of that.
>>> For example, look at the transactions or PME codebase. It's darkness and
>> no
>>> abbreviations for "consistency" will help to understand that code.
>>> 
>>>>> and attract new members
>>> Who can tell me at least 5 contributors with 10+ commits who are not
>>> working/worked in GridGain or SberBank?
>>> It's sad to hide behind an open source community that really does not
>>> exist.
>>> 
>>> 
>>> пт, 18 июн. 2021 г. в 14:21, Anton Vinogradov :
>>> 
>>>>> Agree. I think you can start a discussion and change some abbrevs if
>>>>> you
>>>> wish.
>>>> 
>>>> How about keeping the abbreviation map file inside the Ignite repo?
>>>> In this case, you may change the abbreviation by issue/pr covered by a
>>>> green TC check.
>>>> 
>>>> On Thu, Jun 17, 2021 at 11:36 AM Nikolay Izhikov 
>>>> wrote:
>>>> 
>>>>>> No validation in the world prevent me from typing manually "mess"
>>>>> instead of "msg"/«message"
>>>>> 
>>>>> Code review will do.
>>>>> 
>>>>>> btw "svc" is more common imo
>>>>> 
>>>>> Agree. I think you can start a discussion and change some abbrevs if
>>>>> you
>>>>> wish.
>>>>> 
>>>>> 
>>>>>> 17 июня 2021 г., в 10:23, Ivan Daschinsky 
>>>>> написал(а):
>>>>>> 
>>>>>> I'm sorry, but the rule is not strict and, moreover, not clear
>> enough
>>>> for
>>>>>> constans. See here [1]
>>>>>> ```
>>>>>> Type and method names are usually not abbreviated (except for the
>>>>>> well-accepted abbreviations such as EOF, Impl, Fifo, etc.).
>>>>>> 
>>>>>> Abbreviation applies to local variables, method parameters, class
>>>> fields
>>>>>> and in **some cases public static fileds** (constants) of the class.
>>>>>> ```
>>>>>> But there is not any list that contains these cases. I suppose, that
>>>>>> constant naming should not be abbreviated.
>>>>>> As I see, the most cases of violations, described in initial post,
>>>>>> are
>>>>>> about constant's namings.
>>>>>> 
>>>>>> I suppose that we should define strict and clear rules about
>>>>>> constants
>>>>>> naming.
>>>>>> 
>>>>>> [1] --
>>>>> https://cwiki.apache.org/confluence/display/IGNITE/Abbreviation+Rules
>>>>>> 
>>>>>> 
>>>>>> чт, 17 июн. 2021 г. в 10:10, Konstantin Orlov >> :
>>>>>> 
>>>>>>> Enforced validation doesn't guarantee code consistency. No
>>>>>>> validation
>>>> in
>>>>>>> the world prevent me from typing manually "mess" instead of
>>>>> "msg"/"message"
>>>>>>> or "svc" instead of "srvc"/"service" (btw "svc" is more common
>> imo).
>>>>>>> 
>>>>>>> And the fact that someone name a variable "count" instead of "cnt"
>>>>> doesn't
>>>>>>> make the whole project immature.
>>>>>>> 
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Konstantin Orlov
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> On 17 Jun 2021, at 08:34, Ivan Pavlukhin 
>>>> wrote:
>>>>>>>> 
>>>>>>>> Hi Nikolay,
>>>>>>>> 
>>>>>>>> Thanks, it is rather interesting.
>>>>>>>> 
>>>>>>>> Do we all agree that using different conventions for different
>> code
>>>>>>>> packages does not break "consistency"? Or did I get something
>>>>>>>> wrong?
>>>>>>>> 
>>>>>>>> 2021-06-17 7:12 GMT+03:00, Николай Ижиков :
>>>>>>>>> Hello, Ivan.
>>>>>>&

IGNITE-14812: Statistics

[Nikolay Izhikov]

Hello, Igniters.

Recently huge commit was merged [1].

Taras, Alexander, can you, please, explain what is purpose of the commit?
What feature it implemented?

Looked inside the ticket and found no explanation.
Description is "Add statistics collection and usage.»

Do we have plans to document this new feature?

Also, I see very strange license in added files [2]

```
 * Copyright 2013 Aggregate Knowledge, Inc.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
```

Is it OK to have those copyright inside ASF codebase?
Is it some kind of external dependency we adopted as part of the code?
Why do we need it?

[1] 
https://github.com/apache/ignite/commit/503a98495433e1d0cf84f8be8c1e2adc57034fbb
[2] 
https://github.com/apache/ignite/blob/master/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/stat/hll/serialization/IHLLMetadata.java

Re: IGNITE-14812: Statistics

[Nikolay Izhikov]

Hello, Taras.

Thanks for feedback.

> AFAIK and as long as I can remember Ignite project try to minimize external 
> dependencies usage and adds new external dependency only when there is no 
> other way out.

Does it mean we have to incapsulate every external library we want to use?

Taras, can you, please, describe the features that was implemented in this 
merge?
How users supposed to use them?
Do we have plans to document?


> 23 июня 2021 г., в 09:21, Taras Ledkov  написал(а):
> 
> Hi,
> 
> We have discussed BCrypt include/add dependency here [1].
> AFAIK and as long as I can remember Ignite project try to minimize external 
> dependencies usage
> and adds new external dependency only when there is no other way out.
> 
> [1]. 
> http://apache-ignite-developers.2346864.n4.nabble.com/Username-password-authentication-for-thin-clients-tp26058p26954.html
> 
> On 23.06.2021 3:08, Valentin Kulichenko wrote:
>> Dmitry,
>> 
>> As the PMC chair, would you mind contacting legal regarding the matter?
>> This is not the only example of such code (e.g. [1]), so we should look
>> into this asap.
>> 
>> [1]
>> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/mindrot/BCrypt.java
>> 
>> As for this particular commit, can HLL be added as a dependency instead? Is
>> there any particular reason to include the source code? @Sasha Belyak
>>  , can you please chime in?
>> 
>> -Val
>> 
>> On Tue, Jun 22, 2021 at 8:10 AM Dmitry Pavlov  wrote:
>> 
>>> Hi Nikolai,
>>> 
>>> thank you for noticing. I guess it's not about license, but about
>>> Intellectual Property (IP) ownership.
>>> 
>>> AFAIK, Apache License 2.0 is here and AL 2.0 is definetely allowed to be
>>> used in the codebase for an Apache project (
>>> https://www.apache.org/legal/resolved.html)
>>> 
>>> But licenese and IP owner are 2 sligthly different things. E.g at the end
>>> of any website you can find:
>>> Copyright © 2021 The Apache Software Foundation, Licensed under the Apache
>>> License, Version 2.0.
>>> 
>>> Incubated projects are mandated to transfer IP to the ASF. And I'm not
>>> aware of any exceptions.
>>> 
>>> In this PR there are 5 classes which licenses with AL 2.0, but IP owner is
>>> 3rd party company.
>>> 
>>> I'm a bit concerned about having such code in the project. I'd rather
>>> reverted it until we have approval from experts at mailing list:
>>> legal-disc...@apache.org
>>> 
>>> Sincerely,
>>> Dmitriy Pavlov
>>> 
>>> On 2021/06/22 14:56:49, Nikolay Izhikov  wrote:
>>>> Hello, Igniters.
>>>> 
>>>> Recently huge commit was merged [1].
>>>> 
>>>> Taras, Alexander, can you, please, explain what is purpose of the commit?
>>>> What feature it implemented?
>>>> 
>>>> Looked inside the ticket and found no explanation.
>>>> Description is "Add statistics collection and usage.»
>>>> 
>>>> Do we have plans to document this new feature?
>>>> 
>>>> Also, I see very strange license in added files [2]
>>>> 
>>>> ```
>>>>  * Copyright 2013 Aggregate Knowledge, Inc.
>>>>  *
>>>>  * Licensed under the Apache License, Version 2.0 (the "License");
>>>> ```
>>>> 
>>>> Is it OK to have those copyright inside ASF codebase?
>>>> Is it some kind of external dependency we adopted as part of the code?
>>>> Why do we need it?
>>>> 
>>>> [1]
>>> https://github.com/apache/ignite/commit/503a98495433e1d0cf84f8be8c1e2adc57034fbb
>>>> [2]
>>> https://github.com/apache/ignite/blob/master/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/stat/hll/serialization/IHLLMetadata.java
>>> 
> -- 
> Taras Ledkov
> Mail-To: tled...@gridgain.com
> 

Re: [DISCUSSION] Code style. Variable abbrevations

[Nikolay Izhikov]

Val.

> But then you said that common sense doesn't work. Code review IS common sense 
> for the most part though.

Code review must catch errors like  "No validation in the world prevent me from 
typing manually "mess" instead of «msg»"
But, when one reviewer prefers `qry` and another requires `query` it confusing 
and stressful.

> I'm not against enforced rules per se, but in this case, we are clearly 
> struggling to come up with a rule that could work

Agree. Discussion shows that different opinions exist in community.
I’m relying on rules written in wiki.
If someone wants to change it - please go ahead with vote.

But, silently ignore rules created by ourselves is simply ridiculous.

> So many abbreviations simply make it look cryptic and weird -- this might 
> easily confuse a newcomer. 

Not agreed, but this seems to be a matter of taste.

> It's also clear why we're struggling. Just one example: we have the rule to 
> abbreviate "exception" as "e».

+1 I’m also don’t like these changes :)
Seems we have to do it only for catch block variables.
Will fix.

> 23 июня 2021 г., в 07:24, Ivan Pavlukhina  написал(а):
> 
> Folks,
> 
> I suppose we should rollout voting regarding abbreviations in Ignite 3.
> 
> BTW, are you actually aware of any other project defining something similar 
> to our abbreviations?
> 
> Sent from my iPhone
> 
>> On 23 Jun 2021, at 14:53, Nikita Ivanov  wrote:
>> 
>> What's worked for countless companies is the combination of the following:
>> 1. Well defined rules (abbreviations, code styles & documentation, code
>> organization, etc., etc.) for common/frequent use cases.
>> 2. Some basic tooling (wherever possible) plus a strong culture of code
>> reviews.
>> 3. And... common sense! None of the rules in (1) are axiomatic but rather a
>> guidance that requires common sense.
>> 
>> Code review culture takes time and organizational discipline to become
>> effective & productive. Rules alone or some fiat enforcement is unlikely to
>> work.
>> --
>> Nikita Ivanov
>> 
>> 
>> 
>> On Tue, Jun 22, 2021 at 4:44 PM Valentin Kulichenko <
>> valentin.kuliche...@gmail.com> wrote:
>> 
>>> Nikolay,
>>> 
>>> I'm confused. Konstantin gave you an example of a bad abbreviation that
>>> cannot be caught by automated validation, and you referred to the code
>>> review as a solution. But then you said that common sense doesn't work.
>>> Code review IS common sense for the most part though.
>>> 
>>> Either way, I'm not against enforced rules per se, but in this case, we are
>>> clearly struggling to come up with a rule that could work. This and several
>>> previous discussions show that.
>>> 
>>> It's also clear why we're struggling. Just one example: we have the rule to
>>> abbreviate "exception" as "e". This is perfectly fine for the following
>>> line:
>>> 
>>>   catch (Exception e) {}
>>> 
>>> But then I see the following in your PR:
>>> 
>>>   boolean isE = false;
>>> 
>>> I'm ready to argue that this is not an example of good code :) How would
>>> you propose to fix that? It's really hard to set up generic rules like that
>>> -- common sense is required.
>>> 
>>> Also, in general, the impression I get from your PR is that the code
>>> becomes less readable. So many abbreviations simply make it look cryptic
>>> and weird -- this might easily confuse a newcomer. And honestly, I have no
>>> idea how to formalize this.
>>> 
>>> What is the exact problem that you're trying to solve? Is it that we have a
>>> rule that is not followed because it's too complicated? In that case, I
>>> propose to cancel the rule and then try to come up with more general
>>> guidelines for contributors and reviewers. Again -- common sense is our
>>> friend here.
>>> 
>>> WDYT?
>>> 
>>> -Val
>>> 
>>> On Tue, Jun 22, 2021 at 12:04 AM Nikolay Izhikov 
>>> wrote:
>>> 
>>>> Hello, Pavel
>>>> 
>>>>> But it doesn't mean that now we need to use only full names.
>>>>> Abbreviations may be used e.g. for local variables with short scope.
>>>>> Here we must follow common sense and existing practices of effective
>>>> naming.
>>>>> It shouldn't be a cargo cult as it is now.
>>>> 
>>>> But «common sense» has

Re: [DISCUSSION] Brand new distributed environment testing framework ready to be reviewed/merged.

[Nikolay Izhikov]

Hello.

As one of the authors of PR fully support merge of this module.
Have plans to use it during development of other features.
Guys, please, share your feedback.

Do we need to improve something prior merge?


> 24 июня 2021 г., в 14:41, Denis Magda  написал(а):
> 
> Anyway, it's much more community-friendly. Thanks, Anton!
> 
> --
> Denis
> 
> -
> Denis
> 
> On Thu, Jun 24, 2021 at 3:43 AM Anton Vinogradov  wrote:
> 
>> Denis,
>> 
>> Unfortunately, we had some issues with this recording (low slides recording
>> resolution, missed slides and random slides switch delay), which makes it
>> slightly unwatchable :(
>> Anyway, if you brave enough, you may open the slides [1] and the recording
>> [2] simultaneously to solve this :)
>> 
>> [1]
>> 
>> https://go.gridgain.com/rs/491-TWR-806/images/11_IgniteSummit2021-Deep-dive-into-testing.pdf
>> [2] https://www.youtube.com/watch?v=uRRlGrSA3NY
>> 
>> On Wed, Jun 23, 2021 at 5:40 PM Denis Magda  wrote:
>> 
>>> Congrats, Anton, that's a valuable contribution! I attended your session
>> at
>>> the Ignite Summit and wonder if you should share that recording with an
>>> English-speaking part of the community?
>>> 
>>> -
>>> Denis
>>> 
>>> On Wed, Jun 23, 2021 at 7:37 AM Anton Vinogradov  wrote:
>>> 
 Folks,
 
 Here's the video [1] that explains the proposal in detail.
 Feel free to ask questions here.
 
 [1] https://www.youtube.com/watch?v=f-i9COU5uAQ (in Russian)
 
 On Tue, Jun 8, 2021 at 2:51 PM Anton Vinogradov  wrote:
 
> Igniters,
> Let me present a framework, we developed, that allows automating
>> Apache
> Ignite testing on a real cluster.
> The framework was initially presented at Ignite Summit.
> 
> In brief,
> The framework allows automating operations with any applications on a
 real
> cluster using ssh in a form of a python test.
> 
> Features:
> - Ignite nodes can be started/stopped on a Docker or a real cluster
>>> with
> any custom configuration
> - Any AI version is supported (released or compiled from sources)
> - Ignite forks are also supported «out of the box»
> - Any other application execution is also possible, eg. we
>> implemented
> starters for Spark and Zookeeper
> - Cluster can be managed using the control.sh, we made this a part of
>>> the
> test API
> - Custom Java applications can be executed remotely with/without a
> built-in Ignite node or a Thin client
> - Any ssh command can be executed remotely, and the result will be
> available locally (at the python test)
> - Network can be broken by iptables change to check communication
>>> issues
> - Tests can be executed in parallel when cluster size bigger than
>> tests
> requirements
> 
> As a result, seems, we may automate any testing scenario we can even
> imagine.
> 
> Framework based on Ducktape [1] library from Kafka team, that's why
>> we
> called it Ducktests.
> 
> The Ducktests were developed during work on IEP-56 [2] and already
>> were
> used during the work on IEP-45 [3].
> IEP-45 measurement results examples [4.1] [4.2] were demonstrated at
>>> the
> HighLoad conference last month.
> 
> Code available at PR-9117 [5] and ready to be reviewed/merged.
> Feel free to ask questions or make proposals.
> 
> [1] https://ducktape-docs.readthedocs.io/en/latest/index.html
> [2]
> 
 
>>> 
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-56%3A+Distributed+environment+tests
> [3]
> 
 
>>> 
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up
> [4.1] https://youtu.be/UZsvCNjbkww?t=1065 (in Russian)
> [4.2] https://youtu.be/UZsvCNjbkww?t=1609 (in Russian)
> [5] https://github.com/apache/ignite/pull/9117
> 
 
>>> 
>> 

Re: [VOTE][EXTENSION] Release Apache Ignite performance-statistics-ext extension 1.0.0 RC5

[Nikolay Izhikov]

+1 (binding)

> 1 июля 2021 г., в 09:29, Petrov Mikhail  написал(а):
> 
> +1
> 
> Checked on Ubuntu 20.04. Checked built-report and print-statistics scripts 
> for the custom Ignite application. Checked archive signs. Built from the 
> sources.
> 
> On 01.07.2021 00:24, Sergei Ryzhov wrote:
>> +1
>> Checked a report with an ignite-performance-statistics-example.
>> 

Re: [VOTE][EXTENSION] Release Apache Ignite spring-data-all-ext extensions 1.0.0 RC5

[Nikolay Izhikov]

+1 (binding)

> 1 июля 2021 г., в 08:35, Petrov Mikhail  написал(а):
> 
> +1
> 
> Checked on Ubuntu 20.04. Created separate Spring Applications for each 
> version of Spring Data Integrations using the Maven RC repository and 
> following the documentation. Tested common use cases. Checked archive signs. 
> Built from the sources.
> 
> On 01.07.2021 00:31, Sergei Ryzhov wrote:
>> +1
>> Building modules from source and running tests successfully on Java 8
>> 
>> ср, 30 июн. 2021 г. в 16:02, Alex Plehanov :
>> 
>>> +1
>>> 
>>> Check sha512, sign of archives and versions of released artifacts.
>>> Check licenses with "mvn validate -DskipTests -P check-licenses"
>>> Build modules from the source with Java 8
>>> Run examples:
>>> from modules/spring-data-2.0-ext/examples with artifacts: Ignite
>>> v2.10.0/ignite-spring-data-2.0-ext v1.0.0/spring-data v2.0
>>> from modules/spring-data-2.2-ext/examples with artifacts: Ignite
>>> v2.10.0/ignite-spring-data-2.2-ext v1.0.0/spring-data v2.2
>>> 
>>> 
>>> ср, 30 июн. 2021 г. в 15:13, Nikita Amelchev :
>>> 
 Dear Ignite Community,
 
 I have uploaded a release candidate of the following extension modules:
 
 spring-data-commons
 spring-data-ext
 spring-data-2.0-ext
 spring-data-2.2-ext
 
 The release candidate of the extensions:
 
 
>>> https://dist.apache.org/repos/dist/dev/ignite/ignite-extensions/ignite-spring-data-all-ext/1.0.0-rc5/
 The instruction for building from the source package is placed in the
 DEVNOTES file.
 
 The following staging can be used for testing:
 https://repository.apache.org/content/repositories/orgapacheignite-1524
 
 Tags were created:
 
 ignite-spring-data-commons-1.0.0-rc5
 ignite-spring-data-ext-1.0.0-rc5
 ignite-spring-data-2.2-ext-1.0.0-rc5
 ignite-spring-data-2.0-ext-1.0.0-rc5
 ignite-spring-data-all-ext-1.0.0-rc5
 
 
>>> https://gitbox.apache.org/repos/asf?p=ignite-extensions.git;a=commit;h=4a3ba7b542c46501d425dfa378ac8670e253f49a
 RELEASE NOTES:
 
 
>>> https://gitbox.apache.org/repos/asf?p=ignite-extensions.git;a=blob;f=RELEASE_NOTES.txt;h=22f8d665c194ba2dfa6167354d97ad53d5ef206f;hb=c8de80ee14d1fb76d6cbb0b18513bb70b499c3cb
 DOCUMENTATION:
 Documentation of listed extensions was updated in the following issues:
 
 
>>> https://issues.apache.org/jira/issues/?filter=-1&jql=key%20in%20(IGNITE-14662%2CIGNITE-14398%2CIGNITE-14493)
 The vote is formal, see voting guidelines
 https://www.apache.org/foundation/voting.html
 
 +1 - to accept all the Apache Ignite spring-data-all-ext extensions
 1.0.0-rc5 listed in the thread
 0 - don't care either way
 -1 - DO NOT accept either of the Apache Ignite spring-data-all-ext
 extensions 1.0.0-rc5 (explain why)
 
 The vote will hold for 3 days and will end on July 3 2021 13:00 UTC
 
 
>>> https://www.timeanddate.com/countdown/generic?iso=20210703T13&p0=0&font=cursive
 --
 Best wishes,
 Amelchev Nikita
 
>> 

[DOCUMENTATION] Create of Ignite-style images

[Nikolay Izhikov]

Hello, Igniters.

We have many beautiful images [1] in documentation.
I want to create images that have similar style with other to keep docs 
consistent.
Do we have some guides or styles I can use?

[1] https://ignite.apache.org/docs/2.10.0/images/ignite_clustering.png

Re: [VOTE] Release pyignite 0.5.1-rc0

[Nikolay Izhikov]

+1 (binding)

> 26 июля 2021 г., в 21:11, Pavel Tupitsyn  написал(а):
> 
> +1
> 
> On Mon, Jul 26, 2021 at 11:36 AM Igor Sapego  wrote:
> 
>> +1 from me
>> 
>> Best Regards,
>> Igor
>> 
>> 
>> On Fri, Jul 23, 2021 at 3:32 PM Ivan Daschinsky 
>> wrote:
>> 
>>> +1 From me
>>> 1. Checked binary packages, c module and examples on windows 10 amd64 for
>>> pythons 3.6, 3.7, 3.8, 3.9
>>> 2. Checked binary packages, c module and examples on ubuntu 20.04 amd64
>> for
>>> pythons 3.6, 3.7, 3.8, 3.9
>>> 3. Checked source installation and building binary packages on ubuntu
>> 20.04
>>> amd 64 for pythons 3.6, 3.7, 3.8, 3.9
>>> 4. Checked documentation on
>>> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.1.rc0
>>> 5. Checked sha512 checksums and gpg signatures (signed by Igor Sapego
>> (CODE
>>> SIGNING KEY)  5C10 A072 2D94 7727 923C  98B5 AF35
>> DBD9
>>> 58FE 8DC5)
>>> key is inside https://downloads.apache.org/ignite/KEYS)
>>> 
>>> пт, 23 июл. 2021 г. в 13:52, Ivan Daschinsky :
>>> 
 The voting finishes at 07/27/2021 12:00 UTC
 
 пт, 23 июл. 2021 г. в 13:49, Ivan Daschinsky :
 
> Dear Igniters!
> 
> Release candidate binaries for subj are uploaded and ready for vote
> You can find them here:
> https://dist.apache.org/repos/dist/dev/ignite/pyignite/0.5.1-rc0
> 
> If you follow the link above, you will find source packages (*.tar.gz
>>> and
> *.zip)
> and binary packages (wheels) for windows (amd64) and linux (x86_64)
> for pythons 36, 37, 38, 39. Also, there are sha512 and gpg signatures.
> Code signing keys can be found here --
> https://downloads.apache.org/ignite/KEYS
> Here you can find instructions how to verify packages
> https://www.apache.org/info/verification.html
> 
> You can install binary package for specific version of python using
>> pip
> For example do this on linux for python 3.8
>>> pip install pyignite-0.5.1-cp38-cp38-manylinux1_x86_64.whl
> 
> You can build and install package from source using this command:
>>> pip install pyignite-0.5.1.tar.gz
> You can build wheel on your platform using this command:
>>> pip wheel --no-deps pyignite-0.5.1.tar.gz
> 
> For building C module, you should have python headers and C compiler
> installed.
> (i.e. for ubuntu sudo apt install build-essential python3-dev)
> In Mac OS X xcode-tools and python from homebrew are the best option.
> 
> In order to check whether C module works, use following:
>>> from pyignite import _cutils
>>> print(_cutils.hashcode('test'))
>>> 3556498
> 
> You can find documentation here:
> 
>>> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.1.rc0
> 
> You can find examples here (to check them, you should start ignite
> locally):
> 
> 
>>> 
>> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.1.rc0/examples.html
> Also, examples can be found in source archive in examples subfolder.
> docker-compose.yml is supplied in order to start ignite quickly. (Use
> `docker-compose up -d` to start 3 nodes cluster and `docker-compose
> down` to shut down it)
> 
> Release notes:
> 
> 
>>> 
>> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=blob;f=RELEASE_NOTES.txt;h=c6cbd419684cd4a97485707471bac84957b42891;hb=b48dd5dec37064b458031358c394789d15a756fc
> 
> Git release tag was created:
> 
> 
>>> 
>> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=tag;h=refs/tags/0.5.1.rc0
> 
> The vote is formal, see voting guidelines
> https://www.apache.org/foundation/voting.html
> 
> +1 - to accept pyignite-0.5.1-rc0
> 0 - don't care either way
> -1 - DO NOT accept pyignite-0.5.1-rc0
> 
> 
 
 --
 Sincerely yours, Ivan Daschinskiy
 
>>> 
>> 

Re: [Announcement] Apache Ignite 2.11 Code Freeze started

[Nikolay Izhikov]

Atri.

Right now code can’t be tested because it doesn’t compile.

> 3 сент. 2021 г., в 17:42, Atri Sharma  написал(а):
> 
> I am not sure why is it can be tested by few members?
> 
> Anyone with azure account should be able to test it
> 
> On Fri, 3 Sep 2021, 19:29 Maxim Muzafarov,  wrote:
> 
>> Folks,
>> 
>> The GCP and AWS should also be moved to extensions, but this is a
>> discussion for another topic.
>> 
>> I trust you all :-)
>> But who can validate the patch [1]? This is a bad practice that only a
>> few members be able to test the code.
>> 
>> 
>> [1] https://issues.apache.org/jira/browse/IGNITE-15388
>> 
>> On Fri, 3 Sept 2021 at 16:43, Atri Sharma  wrote:
>>> 
>>> It was extensively tested by myself (running the tests on an actual Azure
>>> account and running an Ignite cluster using an Azure account) and Ilya
>>> (thanks Ilya!) prior to merging it in the core
>>> 
>>> On Fri, 3 Sep 2021, 18:50 Denis Magda,  wrote:
>>> 
 Disagree to exclude this contribution from the 2.11 release. As Atri
 explained, its implementation and testing approach is identical to the
>> AWS,
 GCP and Kubernetes IP finders.
 If we want to move the modules to extensions and improve the testing
 approach, then it needs to be done for all similar components.
 
 Atri, have you tested the module in the cloud? If it works, then I
>> don't
 see any reason why we need to have someone else double-checking this
>> once
 again. This is the community, we need to trust each other.
 
 -
 Denis
 
 On Fri, Sep 3, 2021 at 9:15 AM Atri Sharma  wrote:
 
> I would argue that the module Co exists with the other IP discovery
 modules
> (such as GCP and AWS), which are part of core.
> 
> In terms of tests, the azure module has exactly the same type of
>> tests as
> the other two modules mentioned above.
> 
> On Fri, 3 Sep 2021, 17:54 Maxim Muzafarov, 
>> wrote:
> 
>> Folks,
>> 
>> The patch [1] for the azure module seems to be ready for review,
>> however, there are still some questions that exist:
>> - Do we have a technical account in azure to test the moude on TC?
>> Manual testing for such changes is really annoying.
>> - Is there any reason to add these changes to the main Apache
>> Ignite
>> project? Why not extensions?
>> - It seems there is a lack of tests for this module.
>> 
>> According to the mentions above, what do you think about moving
>> this
>> functionality to the next 2.12 release to solve all the questions
>> without a rush? I propose to revert the changes related to the
>> azure
>> functionality from the 2.11 branch.
>> 
>> 
>> [1] https://issues.apache.org/jira/browse/IGNITE-15388
>> 
>> On Fri, 27 Aug 2021 at 20:53, Dmitriy Pavlov 
 wrote:
>>> 
>>> We had a separate discussion with Max and Alex, so I tend to
>> agree
 that
>>> both issues are blockers.
>>> 
>>> Sincerely,
>>> Dmitriy Pavlov
>>> 
>>> пт, 27 авг. 2021 г. в 16:41, Alexey Gidaspov <
>> olive.c...@gmail.com>:
>>> 
 Hi, Maxim!
 
 These issues really block release. So we should wait fixes.
 
 On 2021/08/27 12:46:24, Maxim Muzafarov 
>> wrote:
> Folks,
> 
> 
> I've been faced these issues, which seems to be a blocker
>> for the
> release. Please, take a look and share your thoughts.
> 
> The server node fails when the client node with the higher
>> java
> version connects to it [1].
> The Apache Ignite build fails with missing dependency [2].
> 
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-14725
> [2] https://issues.apache.org/jira/browse/IGNITE-15388
> 
> On Mon, 9 Aug 2021 at 16:56, Maxim Muzafarov <
>> mmu...@apache.org>
>> wrote:
>> 
>> +1
>> 
>> On Mon, 9 Aug 2021 at 16:27, Ivan Daschinsky <
> ivanda...@gmail.com>
 wrote:
>>> 
>>> I suppose, that
>> https://issues.apache.org/jira/browse/IGNITE-15274
 is a
>>> blocker.
>>> Fix is quite easy and straightforward
>>> 
>>> пн, 2 авг. 2021 г. в 03:11, Igor Sapego <
>> isap...@apache.org
> :
>>> 
 Cherry-picked to release branch.
 
 Best Regards,
 Igor
 
 
 On Fri, Jul 30, 2021 at 3:11 PM Alexey Gidaspov <
 olive.c...@gmail.com>
 wrote:
 
> Ok, I think we should add [1] to 2.11 release. Can
>> you
 cherry-pick it to
> release branch?
> [1]
>> https://issues.apache.org/jira/browse/IGNITE-14815
> 
> On 2021/07/30 02:25:25, Igor Sapego <
>> isap...@apache.org>
>> wrote:
>> I'm fine with any of these two solutions.
>>

Re: Ban Java Streams usage in Ignite 3 code

[Nikolay Izhikov]

+1 to ban Streams usage.

 

> 9 сент. 2021 г., в 02:59, Valentin Kulichenko  
> написал(а):
> 
> Pavel,
> 
> Quite frankly, I think we used to lean into performance too much. We
> generally preferred it over data consistency, project modularity and code
> readability. Performance, of course, plays a very important rule in Ignite,
> but it's possible to overdo anything.
> 
> There are certainly parts of the project that can benefit from features
> like Stream API, without significant concern over performance. CLI is an
> obvious example, but I'm sure it's not the only one.
> 
> That said, I don't think that banning something is productive. At the same
> time, we should make sure we pay more attention to performance during
> reviews. Maybe we should have a checklist of major things to look for? Not
> as a set of strict rules, but more as a guideline for contributors and
> committers.
> 
> We might also want to start benchmarking the code and tracking the progress.
> 
> -Val
> 
> On Wed, Sep 8, 2021 at 12:09 PM Pavel Tupitsyn  wrote:
> 
>> Alexander, Ivan,
>> 
>>> not very productive to assume that 100% of your code is
>>> on the hot path
>> 
>> That is exactly what we should be doing.
>> When I joined Ignite community 6 years ago, this was the prevalent mindset.
>> 
>> I'm not sure which part of Ignite can be considered "not on a hot path".
>> Create/alter table (mentioned above) should perform well too.
>> 
>>> measured first and only then optimized
>>> https://wiki.c2.com/?OptimizeLater
>> 
>> Extra allocations are a "death by a thousand cuts".
>> They add up here and there, and then there are GC pauses.
>> This would be hard to "optimize later".
>> 
>> It is common for perf-oriented projects to avoid some techniques.
>> For example, LINQ (streams analog from C# with similar perf issues) is
>> avoided in libraries and compilers [1].
>> 
>> [1] https://github.com/dotnet/roslyn/blob/main/CONTRIBUTING.md
>> 
>> On Wed, Sep 8, 2021 at 9:49 PM Valentin Kulichenko <
>> valentin.kuliche...@gmail.com> wrote:
>> 
>>> I don't think we should ban anything. Streams API is just a tool in the
>>> toolbox - it should be used appropriately. It's up to the contributor and
>>> reviewer(s) to identify whether a particular usage might cause
>> performance
>>> issues.
>>> 
>>> -Val
>>> 
>>> On Wed, Sep 8, 2021 at 8:01 AM Alexander Polovtcev <
>>> alexpolovt...@gmail.com>
>>> wrote:
>>> 
 -1
 I think that it is not very productive to assume that 100% of your code
>>> is
 on the hot path, it would be impossible to write and maintain. Humans
>> are
 not very good at guessing where the performance bottlenecks are, so the
 performance of the possible hot paths should be measured first and only
 then optimized and documented.
 
 On Wed, Sep 8, 2021 at 5:53 PM Ivan Pavlukhin 
>>> wrote:
 
> Does not this trivial strategy work for us?
> https://wiki.c2.com/?OptimizeLater
> 
> 2021-09-08 13:52 GMT+03:00, Andrey Gura :
>> Agree that any additional object creation on a hot path could be a
>> problem. So hot path should not contain stream API and any other
>> potentially problem code (e.g. iterator instead of for by index).
>> 
>> On Wed, Sep 8, 2021 at 1:45 PM Pavel Tupitsyn <
>> ptupit...@apache.org>
> wrote:
>>> 
>>> Ok, maybe a total ban is overkill, but right now streams are used
>>> even
> on
>>> some hot paths like getAllAsync [1].
>>> 
>>> Another issue is that Collectors.toList() and other variants don't
> accept
>>> capacity, and we end up with unnecessary reallocations of
>> underlying
>>> arrays.
>>> 
>>> [1]
>>> 
> 
 
>>> 
>> https://github.com/apache/ignite-3/blob/1d7d703ff2b18234b15a9a7b03104fbb73388edf/modules/table/src/main/java/org/apache/ignite/internal/table/KVBinaryViewImpl.java#L83
>>> 
>>> On Wed, Sep 8, 2021 at 1:06 PM Konstantin Orlov <
>>> kor...@gridgain.com>
>>> wrote:
>>> 
 Hi!
 
 Agree with Ivan that it’s an overkill. Code readability and
 maintainability should have
 the same priority as the performance (with some exceptions of
 course).
 
 BTW the result of your benchmark looks quite strange. The
 performance
 penalty on
 my laptop (Core i7 9750H, 6 cores up to 4.50 GHz) is 25%, not 8
 times:
 
 Benchmark Mode  Cnt  Score Error
 Units
 JmhIncrementBenchmark.loopSumthrpt   10  32347.819 ± 676.548
 ops/ms
 JmhIncrementBenchmark.streamSum  thrpt   10  24459.196 ± 610.152
 ops/ms
 
 
 --
 Regards,
 Konstantin Orlov
 
 
> On 8 Sep 2021, at 12:23, Ivan Bessonov >> 
> wrote:
> 
> Hello Igniters,
> 
> I object, banning streams is an overkill. I would argue that
>>> most
 of
> the
>>

Re: Release of pyignite 0.5.2 proposal

[Nikolay Izhikov]

+1 to release ASAP.

> 9 сент. 2021 г., в 13:43, Ivan Daschinsky  написал(а):
> 
> TC build of release branch --
> https://tc.sbt-ignite-dev.ru/buildConfiguration/IgniteThinClients_Tests_ThinClientPython/6130289
> 
> чт, 9 сент. 2021 г. в 13:40, Ivan Daschinsky :
> 
>> Hi, folks.
>> 
>> Unfortunately, a quite severe bug found in pyignite since 0.4.0 was found
>> [1]. Fortunately, it is already fixed. Also there are also a few bugs
>> already fixed.
>> 
>> I propose to prepare the next minor release. Branch was already cut and
>> commits were cherry-picked [3].
>> 
>> If there are no objections, I will prepare a release and will start voting
>> thread in a day or two.
>> 
>> [1] -- https://issues.apache.org/jira/browse/IGNITE-15479
>> [2] --
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%20python-0.5.2
>> [3] --
>> https://github.com/apache/ignite-python-thin-client/commits/pyignite-0.5.2
>> 
> 
> 
> -- 
> Sincerely yours, Ivan Daschinskiy

Re: Deprecating LOCAL cache

[Nikolay Izhikov]

I proposed the following plan:

1. 2.12 - deprecation of LOCAL caches.
2. 2.13 - complete removal LOCAL caches from codebase. 

> 13 сент. 2021 г., в 13:30, Ivan Daschinsky  написал(а):
> 
> I personally support deprecation, but we should at least have a plan.
> I suppose that putting annotations and removing documentation are not
> enough.
> 
> 
> пн, 13 сент. 2021 г. в 13:22, Maxim Muzafarov :
> 
>> Ivan,
>> 
>> I don't think we can remove LOCAL caches at the nearest time, so there
>> is no plan for that. I can only imagine a single release that will
>> contain all the breaking changes we want to apply in 2.x version.
>> 
>> My point here is only about deprecation:
>> - there are a lot of motivation points to remove written in this thread;
>> - I always hear from the support team that they do not recommend using
>> local caches;
>> - I haven't seen any bugs fixed for a long time for local caches
>> (suppose that we are not maintaining them);
>> 
>> I just want to make sure that all these points are reflected in the
>> code base, so propose to mark them as deprecated.
>> 
>> On Mon, 13 Sept 2021 at 11:29, Ivan Daschinsky 
>> wrote:
>>> 
>>> Hi, Maxim. And what is the plan of removing this functionality? And I
>> also
>>> have some questions regarding deprecation in binary protocol
>>> 
>>> Currently thin client binary protocol
>>> 1. Does support LOCAL caches
>>> 2. Does not support node filters.
>>> 
>>> I can hardly imagine the usefulness of this feature on thin clients,
>>> especially with partition awareness, but nevertheless.
>>> What is expected behaviour if this feature is removed from newest version
>>> of Apache Ignite server and and and old client is requesting
>>> creation of LOCAL cache?
>>> 
>>> вс, 12 сент. 2021 г. в 15:10, Maxim Muzafarov :
>>> 
 Folks,
 
 Let's get back to the discussion of obsolete LOCAL caches since a lot
 of time has passed since the last discussion.
 I've created an issue [1] for deprecation. Let's deprecate them at
 least at the next 2.12 release.
 
 WDYT?
 
 
 [1] https://issues.apache.org/jira/browse/IGNITE-15499
 
 On Fri, 27 Jul 2018 at 20:59, Valentin Kulichenko
  wrote:
> 
> Guys,
> 
> Use cases for local caches are rare, but they definitely exist. I
>> don't
> think it's a very good idea to deprecate this functionality at this
 point.
> 
> At the same point, it's obviously not the most critical part of the
> product, so maintaining the whole separate implementation for it is
> probably an overkill. We had exact same story with replicated caches
>> btw
 -
> they were implemented separately which caused maintainability
>> issues, and
> we ended up removing this separate implementation. If we have the
>> same
> situation here, let's use the same solution.
> 
> -Val
> 
> On Fri, Jul 27, 2018 at 3:05 AM Dmitry Pavlov >> 
 wrote:
> 
>> Hi Dmitriy,
>> 
>> I would like to stress this: I'm not saying local cache it
>> useless. I'm
>> supposing it is not used widely. I want to figure out if I'm
>> mistaking.
>> 
>> All folks involved into user list says it is not used, so why not
>> to
>> deprecate? If we make a mistake, somebody will come to user list
>> and
 say,
>> 'Hey, why did you deprecate this, it is used for... in my project'
>> 
>> Being very experienced Igniter you probably know real life usage
 examples.
>> And I appreciate if you or somebody else in community could share
>> it.
>> 
>> Sincerely,
>> Dmitriy Pavlov
>> 
>> пт, 27 июл. 2018 г. в 1:04, Dmitriy Setrakyan <
>> dsetrak...@apache.org>:
>> 
>>> Guys,
>>> 
>>> I just want to make sure we are all on the same page. The main
>> use
 case
>> for
>>> LOCAL caches is to have a local hash map querable with SQL and
>>> automatically persisted to a 3rd party DB.
>>> 
>>> I want to discourage people from saying "nobody needs some
>> feature".
 None
>>> of the people in this discussion are users of any features - we
>> are
 all
>>> developers of the features. Instead of guessing whether to
>> deprecate
>>> something or not, I would actually see if it is even worth a
 discussion.
>>> How much effort is required to fix the bug found in the LOCAL
>> cache?
>>> 
>>> D.
>>> 
>>> On Thu, Jul 26, 2018 at 12:19 PM, Dmitry Pavlov <
 dpavlov@gmail.com>
>>> wrote:
>>> 
 Hi Alexey,
 
 There is nothing to be sorry about :) Сommunity appreciates an
>>> alternative
 vision, this allows us to make as informed decisions as it
 possible.
 
 Thank you for finding this fact, it is very interesting.
 
 I'm not sure all these examples were prepared by experienced
>> Ignite
>>> users.
 So idea of deprecation may have one more argument. Deprecation
>> w

Re: [DISCUSSION] Remove VS project from C++

[Nikolay Izhikov]

+1

> 15 сент. 2021 г., в 11:57, Pavel Tupitsyn  написал(а):
> 
> -1
> 
> This may become an obstacle for some of the users and I'm not sure how it
> improves anything.
> 
>> 3. Sometimes even maintainers forget to add test sources to VS projects
> [1]
> We can add an automatic check for this (in form of a test).
> 
> On Wed, Sep 15, 2021 at 10:28 AM Zhenya Stanilovsky
>  wrote:
> 
>> 
>> 
>> completely support !
>> 
>>> Igniters!
>>> 
>>> Currently we have CMake build system, that works on Windows, Linux and
>>> MacOs flawlessly
>>> 
>>> 1. CMake is supported natively in VS 2019
>>> 2. CMake can generate VS projects for about 20 years flawlessly.
>>> 3. Sometimes even maintainers forget to add test sources to VS projects
>> [1]
>>> 4. Currently on TC we build Ignite C++ on windows and linux flawlessly
>>> using CMake
>>> 5. VS projects are not backward compatible. We have to add manually (or by
>>> sed or patch) some dependencies in order to build current VS projects on
>>> newer versions of VS.
>>> 
>>> So I suggest simpy to remove VS projects because of reasons I've written
>>> above.
>>> 
>>> WDYT?
>>> 
>>> 
>>> 
>>> [1] --  https://issues.apache.org/jira/browse/IGNITE-15511
>> 
>> 
>> 
>> 

Re: [VOTE] Release Apache Ignite 2.11.0 RC2

[Nikolay Izhikov]

+1 (binding)

> 15 сент. 2021 г., в 12:24, Alex Plehanov  написал(а):
> 
> +1
> 
> Checked sha512 and sign of archives, built from sources, started a cluster
> with a few nodes.
> 
> вт, 14 сент. 2021 г. в 21:23, Pavel Tupitsyn :
> 
>> Ilya,
>> 
>>> I'm still confused by this publish/ directory. Why are we shipping
>>> something which is not for publishing in our binary package? I've also
>> not
>>> heard of it before.
>> 
>> I think this was introduced in 2.8.0 as part of [1].
>> It is how the build process works by default and we just copy the entire
>> bin directory to the package,
>> while only publish subdir is actually required.
>> 
>> We can probably drop support for legacy (unsupported) .NET versions in the
>> next release and simplify our build process, I'll look into this.
>> 
>> [1] https://issues.apache.org/jira/browse/IGNITE-11805
>> 
>> On Tue, Sep 14, 2021 at 6:12 PM Ilya Kasnacheev >> 
>> wrote:
>> 
>>> Hello!
>>> 
>>> + 0.5
>>> 
>>> I have got it to run with the following command:
>>> % COMPlus_EnableAlternateStackCheck=1 dotnet --fx-version 2.1.30
>>> Apache.Ignite.dll
>>> 
>>> I'm still confused by this publish/ directory. Why are we shipping
>>> something which is not for publishing in our binary package? I've also
>> not
>>> heard of it before.
>>> 
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>> 
>>> 
>>> вт, 14 сент. 2021 г. в 12:37, Pavel Tupitsyn :
>>> 
 Ilya,
 
> When I have also installed 3.1 and 2.0
 Please try .NET SDK 2.1 to run tests.
 Yes, it went out of support last month, I've created a ticket [1].
 
 
> dotnet Apache.Ignite.dll
 That's correct, but should be done
 in platforms/dotnet/bin/netcoreapp2.0/publish, please try again.
 We should deal with duplicate binaries, I've created another ticket [2]
 
 
> Why do we still ship Apache.Ignite.exe if it does not run?
 It runs on Windows. For Linux we ship Apache.Ignite.dll.
 
 
 I don't think any of this is a blocker for the current release.
 - Out-of-support SDK only affects those who build from source, and it
>>> went
 out of support only recently
 - Duplicate binaries are there for a long time and no one was
>> complaining
 
 
 [1] https://issues.apache.org/jira/browse/IGNITE-15504
 [2] https://issues.apache.org/jira/browse/IGNITE-15505
 
 On Tue, Sep 14, 2021 at 11:58 AM Ilya Kasnacheev <
 ilya.kasnach...@gmail.com>
 wrote:
 
> Hello!
> 
> - 0.5
> 
> Installed DEB package, built binary from source, ran sqlline, built
>>> .NET.
> 
> I have failed to run .NET tests though.
> 
> If I only install Current .NET, I get the following:
> It was not possible to find any compatible framework version
> The framework 'Microsoft.NETCore.App', version '2.0.0' was not found.
> 
> 
> - The following frameworks were found:
> 
> 
> 5.0.9 at [/home/ilyak/.dotnet/shared/Microsoft.NETCore.App]
> 
> 
> You can resolve the problem by installing the specified framework
>>> and/or
> SDK.
> 
> 
> The specified framework can be found at:
> 
> 
> -
> 
> 
 
>>> 
>> https://aka.ms/dotnet-core-applaunch?framework=Microsoft.NETCore.App&framework_version=2.0.0&arch=x64&rid=ubuntu.21.04-x64
> 
> When I have also installed 3.1 and 2.0, now I see:
> 
> The active test run was aborted. Reason: Test host process crashed :
>> No
> usable version of the libssl was found
> 
> It looks like we have to bump the permitted .NET version.
> 
> From the slim binary package I also can't run .NET:
> 
> ~/D/apache-ignite-slim-2.11.0-bin/platforms/dotnet/bin/netcoreapp2.0%
> dotnet Apache.Ignite.dll
> Error:
> An assembly specified in the application dependencies manifest
> (Apache.Ignite.deps.json) was not found:
>   package: 'System.Configuration.ConfigurationManager', version:
>>> '4.4.0'
>   path:
 'lib/netstandard2.0/System.Configuration.ConfigurationManager.dll'
> 
> Do we have any instructions on how to run it under Linux? Why do we
>>> still
> ship Apache.Ignite.exe if it does not run?
> 
> Regards,
> --
> Ilya Kasnacheev
> 
> 
> вт, 14 сент. 2021 г. в 11:28, 18624049226 <18624049...@163.com>:
> 
>> check again, 15503 not cherry-pick, right?
>> 
>> 在 2021/9/14 15:55, Shishkov Ilya 写道:
 Should the following two issue be merged into the ignite-2.11
 branch?
>>> Hi, as I see, both tickets are fixed in 2.10 and 2.11.
>>> 
>>> вт, 14 сент. 2021 г. в 04:13, 18624049226<18624049...@163.com>:
>>> 
 Should the following two issue be merged into the ignite-2.11
 branch?
 
 https://issues.apache.org/jira/browse/IGNITE-14404
 
 https://issues.apache.org/jira/browse/IGNITE-15503
 
 在 2021/9/13 17:12, Pavel Tupitsyn 写道:
> +

Re: [DISCUSSION] Separate Jira project and Confluence space for Ignite 3

[Nikolay Izhikov]

>>>>>>>> Denis,
>>>>>>>>>>>> 
>>>>>>>>>>>> From a purely technical perspective, these are indeed two
>>>> separate
>>>>>>>>>>>> projects, because they are based on different codebases. The
>>>> split
>>>>>>>>>> you're
>>>>>>>>>>>> talking about happened a year ago, when we created the repo for
>>>>>>>>>>>> Ignite
>>>>>>>>>> 3.
>>>>>>>>>>>> This significantly differs from the 1.x->2.x transition, as
>> these
>>>>>>>>>>>> two
>>>>>>>>>>>> shared the codebase.
>>>>>>>>>>>> 
>>>>>>>>>>>> For the same reason, a bug filed for 2.x can't be just
>>>> transitioned
>>>>>>>>>>>> to
>>>>>>>>>>>> 3.x. It will either not exist in 3.x in the first place, or
>> will
>>>>>>>>>> require
>>>>>>>>>>> a
>>>>>>>>>>>> completely different fix, which will mean two different
>> tickets.
>>>>>>>>>>>> 
>>>>>>>>>>>> That said, I still believe that Ignite 2 and Ignite 3 are just
>>>>>>>>>> different
>>>>>>>>>>>> versions of the same product, because, as you correctly
>>>> mentioned,
>>>>>>>>>>>> they
>>>>>>>>>>>> target "same users, community, use cases". At the same time,
>> they
>>>>>>>>>>>> are
>>>>>>>>>>>> developed as different projects on the technical level. Let's
>> not
>>>>>>>>>> confuse
>>>>>>>>>>>> these two aspects with each other - they are largely
>> orthogonal.
>>>>>>>>>>>> 
>>>>>>>>>>>> At this point, creating a Jira project doesn't change anything
>>>>>>>>>>>> fundamentally. It's only about ease of use of our tooling and
>>>>>>>>>>>> efficient
>>>>>>>>>>>> ticket management.
>>>>>>>>>>>> 
>>>>>>>>>>>> -Val
>>>>>>>>>>>> 
>>>>>>>>>>>> On Thu, Sep 23, 2021 at 10:15 PM Denis Magda <
>> dma...@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Folks, you confuse me. I've never treated Ignite 3 as a
>>>> different
>>>>>>>>>>>>> project. It's the same Ignite (distributed database for
>>>>>>>>>> high-performance
>>>>>>>>>>>>> computing...) but on a modernized architecture and APIs -
>> thus,
>>>> a
>>>>>>>>>> major
>>>>>>>>>>>>> version. Same users, community, use cases.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> But, I'm against separate JIRA or Confluence projects. This is
>>>> how
>>>>>>>>>>> you're
>>>>>>>>>>>>> truly stepping on a project-split path. When we used to work
>> on
>>>>>>>>>>>>> Ignite
>>>>>>>>>>> 2 we
>>>>>>>>>>>>> could live within the same JIRA space with Ignite 1. Moreover,
>>>> many
>>>>>>>>>>> tickets
>>>>>>>>>>>>> that are filed against Ignite 2 can be fixed in Ignite 3 only
>> -
>>>>>>>>>>>>> which
>>>>>>>>>>> is a
>>>>>>>>>>>>> version change in our JIRA.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> So, -1 from me for the separate JIRA proposal.
>>>>>>>>>>>>> 
>>>>>>&g

Re: [DISCUSSION] Separate Jira project and Confluence space for Ignite 3

[Nikolay Izhikov]

or that purpose we can ask INFRA to create a
>>>>>>>>>>>>> special mandatory field such as "Architecture" with two
>>>> predefined
>>>>>>>>>>>> values -
>>>>>>>>>>>>> "Ignite 2.x" and "Ignite 3.x". Come up with a better name, it
>>>>>> needs
>>>>>>>>>>>>> to
>>>>>>>>>>>>> be
>>>>>>>>>>>>> intuitive enough even for users who submit issues. What disturbs
>>>>>> me
>>>>>>>>>>>>> is
>>>>>>>>>>>> that
>>>>>>>>>>>>> neither Kafka nor Spark have a different project for the
>>>> recently
>>>>>>>>>>>> released
>>>>>>>>>>>>> versions 3. A different GitHub project is not that disturbing.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -
>>>>>>>>>>>>> Denis
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Fri, Sep 24, 2021 at 4:09 AM Valentin Kulichenko <
>>>>>>>>>>>>> valentin.kuliche...@gmail.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Denis,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> From a purely technical perspective, these are indeed two
>>>>>> separate
>>>>>>>>>>>>>> projects, because they are based on different codebases. The
>>>>>> split
>>>>>>>>>>>> you're
>>>>>>>>>>>>>> talking about happened a year ago, when we created the repo for
>>>>>>>>>>>>>> Ignite
>>>>>>>>>>>> 3.
>>>>>>>>>>>>>> This significantly differs from the 1.x->2.x transition, as
>>>> these
>>>>>>>>>>>>>> two
>>>>>>>>>>>>>> shared the codebase.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> For the same reason, a bug filed for 2.x can't be just
>>>>>> transitioned
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> 3.x. It will either not exist in 3.x in the first place, or
>>>> will
>>>>>>>>>>>> require
>>>>>>>>>>>>> a
>>>>>>>>>>>>>> completely different fix, which will mean two different
>>>> tickets.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> That said, I still believe that Ignite 2 and Ignite 3 are just
>>>>>>>>>>>> different
>>>>>>>>>>>>>> versions of the same product, because, as you correctly
>>>>>> mentioned,
>>>>>>>>>>>>>> they
>>>>>>>>>>>>>> target "same users, community, use cases". At the same time,
>>>> they
>>>>>>>>>>>>>> are
>>>>>>>>>>>>>> developed as different projects on the technical level. Let's
>>>> not
>>>>>>>>>>>> confuse
>>>>>>>>>>>>>> these two aspects with each other - they are largely
>>>> orthogonal.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> At this point, creating a Jira project doesn't change anything
>>>>>>>>>>>>>> fundamentally. It's only about ease of use of our tooling and
>>>>>>>>>>>>>> efficient
>>>>>>>>>>>>>> ticket management.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> -Val
>>>>>>>>>>>>>> 
>>>>>>>>>>>>

Re: [DISCUSSION] Separate Jira project and Confluence space for Ignite 3

[Nikolay Izhikov]

> — issues related to Maven build? possible Gradle upgrade?

I’m not aware of the issues.
Can you, please, send a tickets or description of existing issues?
Anyway, it seems change of build tool can be done at any time we want

> — issues related to run scripts?
> — issues related to release and delivery processes and scripts?

I’m not aware of those too.
Can you point to then, please?

> Are they going to be addressed during Apache Ignite evolution too?

Yes, from my point of view.

> 29 сент. 2021 г., в 14:03, Petr Ivanov  написал(а):
> 
> And what about:
> — issues related to Maven build? possible Gradle upgrade?
> — issues related to run scripts?
> — issues related to release and delivery processes and scripts?
> 
> Are they going to be addressed during Apache Ignite evolution too?
> 
>> On 29 Sep 2021, at 13:47, Nikolay Izhikov  wrote:
>> 
>>> Does you vision of evolutionary improvement involve technical debt 
>>> addressing
>> 
>> Yes, of course.
>> 
>> My vision was the following (from the bird eye):
>> 
>> - 2.20 - removals of LOCAL caches, MVCC and other abandoned features. (User 
>> API doesn’t change).
>> - 2.30 - replace static XML configuration with the new dynamic approach.
>> - 2.40 - replace H2 SQL engine with the Calcite
>> 
>> etc. 
>> 
>> Versions depends on feature readiness.
>> 
>> Anyway, I step back with the initial Ignite3 development, because, don’t 
>> want to interfere the progress.
>> 
>> Respect to the developers who have courage to develop such complex things 
>> from scratch.
>> 
>>> 29 сент. 2021 г., в 12:55, Petr Ivanov  написал(а):
>>> 
>>> 
>>>> I believe that we should improve Ignite evolutionary and not revolutionary.
>>>> First of all, change user API with the slow improvements step by step.
>>> 
>>> Nikolay,
>>> 
>>> Does you vision of evolutionary improvement involve technical debt 
>>> addressing?
>>> 
>>> 
>>>> 
>>>> 
>>>>> 29 сент. 2021 г., в 11:43, Ilya Kasnacheev  
>>>>> написал(а):
>>>>> 
>>>>> Hello!
>>>>> 
>>>>> If we go the second route, we can call the field "Generation".
>>>>> 
>>>>> Generation: Ignite 2.x
>>>>> Generation: Ignite 3
>>>>> 
>>>>> (no new tickets should ever be filed for Ignite 1.x but if they are, they
>>>>> should go to the first Generation)
>>>>> 
>>>>> Regards.
>>>>> -- 
>>>>> Ilya Kasnacheev
>>>>> 
>>>>> 
>>>>> ср, 29 сент. 2021 г. в 00:33, Valentin Kulichenko <
>>>>> valentin.kuliche...@gmail.com>:
>>>>> 
>>>>>> As for the original topic, we need to come to a solution. Let me 
>>>>>> summarize
>>>>>> what we've discussed so far.
>>>>>> 
>>>>>> -PROBLEM-
>>>>>> 
>>>>>> Ignite 3 is the next major version of Apache Ignite. It targets the same
>>>>>> use cases and provides a similar set of features as Ignite 2. At the same
>>>>>> time, Ignite 2 and Ignite 3 are *technically* separate projects. They are
>>>>>> developed in different repositories (and therefore are based on different
>>>>>> codebases) and implement different internal architectures. To achieve a
>>>>>> more efficient development process, we need to create a clear separation
>>>>>> between 2.x and 3.x within *development resources* (Jira and Confluence).
>>>>>> 
>>>>>> -POSSIBLE SOLUTIONS-
>>>>>> 
>>>>>> 1. Create a separate Jira project and Confluence space for Ignite 3
>>>>>> (initial suggestion).
>>>>>> 2. Add a *mandatory* field in Jira to identify whether a ticket belongs 
>>>>>> to
>>>>>> 2.x or 3.x.
>>>>>> 
>>>>>> If we go with #2, there are still several things to figure out:
>>>>>> 
>>>>>> - What is the name of this field? It needs to be intuitive to anyone who
>>>>>> joins the community.
>>>>>> - We need to make sure that Ignite 3 tickets are not mapped to 2.x
>>>>>> versions, and vice versa. Can we restrict this in Jira? Or we will have
>>>>>> to
>>>>>> monitor this manually?
>&g

Re: [DISCUSS] Release spring-tx-ext and spring-cache-ext Ignite extensions

[Nikolay Izhikov]

+1

> 5 окт. 2021 г., в 11:48, Nikita Amelchev  написал(а):
> 
> Folks, we should release the spring-data-commons module with the 1.1.0
> version to release spring-tx, spring-cache modules.
> It contains some proxies required for the new modules.
> 
> I suggest:
> 
> 1. Release the spring-data-commons 1.1.0, spring-tx 1.0.0,
> spring-cache 1.0.0 modules with separate source packages:
> 
> /ignite/ignite-extensions/ignite-spring-data-commons/1.1.0/[src]
> /ignite/ignite-extensions/ignite-spring-tx-ext/1.0.0/[src]
> /ignite/ignite-extensions/ignite-spring-cache-ext/1.0.0/[src]
> 
> (The spring-data-commons 1.0.0 was included in the
> ignite-spring-data-all-ext/1.0.0 source package)
> 
> 2. Use one voting thread for this activity.
> 
> Any objections?
> 
> пт, 24 сент. 2021 г. в 15:14, Nikita Amelchev :
>> 
>> Igniters,
>> 
>> Since Ignite 2.11 has been released the following extension modules
>> can be released too:
>> 
>> spring-cache-ext
>> spring-tx-ext
>> 
>> I want to be a release manager for these if nobody minds.
>> 
>> The release process can be started after checking documentation.
>> 
>> --
>> Best wishes,
>> Amelchev Nikita
> 
> 
> 
> -- 
> Best wishes,
> Amelchev Nikita

Re: [DISCUSS] Custom service proxy context

[Nikolay Izhikov]

+1 to have service proxy context.

> 11 окт. 2021 г., в 09:43, Ivan Daschinsky  написал(а):
> 
> Val, Pavel both of you are right, but on the other hand there are some
> other tasks
> 
> 1. Distributed tracing.
> 2. Custom metrics/measurements
> 3. Auth and some related tasks (i.e. ingests full User info by calling some
> auth service in middleware).
> 
> Do you both think that this is a good idea in business code?
> 
> Without this functionality, our service grid cannot compete with grpc and
> others as microservice framework, unfortunately.
> 
> But if we introduce limited support for this "request headers", it can
> drastically improves this aspects of our service grid framework.
> 
> 
> пн, 11 окт. 2021 г. в 00:48, Valentin Kulichenko <
> valentin.kuliche...@gmail.com>:
> 
>> I agree with Pavel. The suggested approach is indeed utilized quite
>> frequently, but it's inherently error-prone.
>> 
>> The main issue is that it creates implicit assumptions about the behavior
>> of both the service and the user's code. For example, if the user's code
>> must provide a username, what if it doesn't? I assume it will get an error,
>> which is very counterintuitive. Even more importantly, how should one learn
>> about this requirement in the first place? It is not reflected in the API
>> in any way - and that's a big problem.
>> 
>> The fact that the service implementor needs to update the API methods when
>> such requirements are introduced is actually a good thing, in my opinion.
>> This forces the developer to stop and think about how the updated API
>> should look like and how to make sure it's backward-compatible (or not, in
>> case the new requirements are mandatory). Doing this through an external
>> context is basically the equivalent of saying "let the end user deal with
>> this". Not a good practice, in my view.
>> 
>> Conversely, passing everything exclusively via method arguments guarantees
>> that:
>> 
>>   - The user's code is always compliant with the service contract. You
>>   can't "forget" to pass something to the service.
>>   - Any changes in the service contract (backward-compatible or otherwise)
>>   are explicitly reflected in the API.
>> 
>> 
>> -Val
>> 
>> 
>> On Sun, Oct 10, 2021 at 6:21 AM Pavel Tupitsyn 
>> wrote:
>> 
>>> Ivan,
>>> 
>>> Yes, this approach is used by some other systems, and still, I don't like
>>> it very much.
>>> Let's hear more opinions.
>>> 
>>> On Sat, Oct 9, 2021 at 9:00 PM Ivan Daschinsky 
>>> wrote:
>>> 
 Hi.
 Pavel T., Ok, http rest dosn't have the clean design, in your opinion.
 
 But what about grpc? The same?
 
 As for me, it is ok to pass additional parameters as list of key-value
 pairs with keys as strings and values as bytearrays or strings. It is
>> ok
>>> to
 allow user to set up middlewares for services and allow to enrich
>> request
 context in this middlewares. It is very common approach everywhere and
>> is
 very useful in distributed systems. The use cases are so obvious,
>> aren't
 they?
 
 
 
 сб, 9 окт. 2021 г., 20:14 Pavel Tupitsyn :
 
> Pavel,
> 
> Thanks for the explanation, I understand the use cases.
> 
>> in REST service, he can set such parameters in request headers
> 
> I don't consider HTTP-based services as a good example of a
> clean architecture.
> Data can be passed in URL parameters, in headers, and in body, and
>> each
 of
> those ways has its own limitations.
> There is no obvious correct way to do things.
> 
>>> Ambient state is not obvious and the API looks confusing even
>>> though I
> understand our services stack quite well both in Java and .NET
>> Can you clarify please?
> 
> The proposed API adds a "side channel" for the data.
> Some is passed as arguments, which is obvious, and some becomes
>>> magically
> available on the server side through some external context.
> - You have to know about the context
> - You have to understand that the context is only available during
>> the
> method call (can't use it in some background logic)
> 
> In my opinion, this is a bit too clever. I'm a fan of the functional
> programming approach where everything you need is passed as
>> arguments.
> 
> 
> 
> On Fri, Oct 8, 2021 at 4:29 PM Pavel Pereslegin 
 wrote:
> 
>> Igor, Pavel.
>> 
>>> Why can not a user implement such context on application level? I
>> believe Ignite provides all necessary tools for that.
>> The user wants to trace the source of the service call. For
>> example,
>>> a
>> service must log the name of the user who made the calls of the
>> service. For now, there's no possibility to do that without
>> modifying
>> the service interface and implementation. Moreover, the user must
>> modify all methods of service to pass this parameter. For example,
>> in
>> REST service, he can set such paramet

Re: Move Azure, AWS, GCE to the ignite-extensions

[Nikolay Izhikov]

Maxim.

> Currently, they are copied from the optional
> directory of the ignite binary package but would be copied from an
> appropriate ignite extension binary package.

But how, the user will download this binary package?
Right now, all the user need is Ignite distributive.


> 13 окт. 2021 г., в 14:32, Maxim Muzafarov  написал(а):
> 
> Stephen,
> 
> I guess the required classes of IP-finders should be in the classpath
> (libs directory). Currently, they are copied from the optional
> directory of the ignite binary package but would be copied from an
> appropriate ignite extension binary package. Probably I'm missing
> something but almost nothing changes in that process from my point of
> view. The documentation pages will be updated prior to the release.
> 
> On Wed, 13 Oct 2021 at 13:44, Stephen Darlington
>  wrote:
>> 
>> I understand the motivation from a development point of view, but how will 
>> this work for end users? Currently, the documentation talks about extensions 
>> only in terms of importing maven dependencies (download.cgi 
>> ). If I’m trying to start 
>> a cluster on Azure, how does that work? Do I need to build my own server?
>> 
>> Regards,
>> Stephen
>> 
>>> On 13 Oct 2021, at 11:35, Nikita Amelchev  wrote:
>>> 
>>> +1 to migrate and include to the Ignite 2.12 scope
>>> 
>>> пн, 20 сент. 2021 г. в 17:09, Denis Magda :
 
 Perfect, thanks, Maxim!
 
 -
 Denis
 
 
 On Mon, Sep 20, 2021 at 8:29 AM Maxim Muzafarov  wrote:
 
> Folks,
> 
> 
> I've created an issue [1] to move all cloud-based IP-finders to the
> ignite-extensions. The motivation is the same as with migration of
> Spring Data integration - to remove integration dependency of the
> release cycle on Ignite releases.
> 
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-15541
> 
>>> 
>>> 
>>> 
>>> --
>>> Best wishes,
>>> Amelchev Nikita
>> 
>> 

[DISCUSSION][IEP-80] Breaking changes in Ignite-2.x

[Nikolay Izhikov]

Hello, Igniters.

I’ve prepared IEP-80 [1] to track breaking changes that should be done in 
Ignite 2.x.

We agreed on the following algorithm to deal with breaking changes:

- Ignite 2.[x] - deprecate the API + notification of the users.
- Ignite 2.[x+1] - API removal.


I propose to start these improvements with the d&r (depuration & removal) of 
the following features:

- LOCAL caches
- MVCC caches
- legacy service grid implementation
- legacy JMX metrics beans.

Deprecation in 2.12
Removal in 2.13

Please, share your opinion.

[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191334475

Re: [DISCUSSION][IEP-80] Breaking changes in Ignite-2.x

[Nikolay Izhikov]

THanks, Maksim.

Tickets included in IEP scope and marked for d&r in 2.12-2.13

> 15 окт. 2021 г., в 16:03, Maxim Muzafarov  написал(а):
> 
> Let's deprecate RebalanceDelay and RebalanceMode=NONE also.
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-12662
> [2] https://issues.apache.org/jira/browse/IGNITE-14613
> 
> On Fri, 15 Oct 2021 at 15:46, Anton Vinogradov  wrote:
>> 
>> +1
>> 
>> On Fri, Oct 15, 2021 at 3:41 PM Nikita Amelchev 
>> wrote:
>> 
>>> +1 for deprecation in the 2.12 release
>>> 
>>> пт, 15 окт. 2021 г. в 15:35, Nikolay Izhikov :
>>>> 
>>>> Hello, Igniters.
>>>> 
>>>> I’ve prepared IEP-80 [1] to track breaking changes that should be done
>>> in Ignite 2.x.
>>>> 
>>>> We agreed on the following algorithm to deal with breaking changes:
>>>> 
>>>>- Ignite 2.[x] - deprecate the API + notification of the users.
>>>>- Ignite 2.[x+1] - API removal.
>>>> 
>>>> 
>>>> I propose to start these improvements with the d&r (depuration &
>>> removal) of the following features:
>>>> 
>>>>- LOCAL caches
>>>>- MVCC caches
>>>>- legacy service grid implementation
>>>>- legacy JMX metrics beans.
>>>> 
>>>> Deprecation in 2.12
>>>> Removal in 2.13
>>>> 
>>>> Please, share your opinion.
>>>> 
>>>> [1]
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191334475
>>> 
>>> 
>>> 
>>> --
>>> Best wishes,
>>> Amelchev Nikita
>>> 

Re: [DISCUSS] Custom service proxy context

[Nikolay Izhikov]

lic API to create the context. In other words, it will
>> be
>>>> up to the end user to create this context properly, which affects the
>>>> business code - and that's exactly where I see an issue.
>>>> 
>>>> The examples you mentioned are more related to internal activities
>> (e.g.,
>>>> if authentication is handled by an Ignite server node, it can create
>> its
>>>> internal context for a connection - this is certainly reasonable). I'm
>> only
>>>> worried about exposing this to the end user.
>>>> 
>>>> Maybe you can pick one of the use cases that you think would benefit
>> from
>>>> this feature the most, and provide a little more detail? How would you
>> like
>>>> to see the use case to be addressed and what is currently missing?
>>>> 
>>>> Also, just to be clear: I'm not necessarily against the suggestion, and
>>>> it's highly unlikely that I will want to veto it if you or someone else
>>>> will decide to implement it. Just expressing my concerns.
>>>> 
>>>> -Val
>>>> 
>>>> On Sun, Oct 10, 2021 at 11:52 PM Nikolay Izhikov 
>>>> wrote:
>>>> 
>>>>> +1 to have service proxy context.
>>>>> 
>>>>>> 11 окт. 2021 г., в 09:43, Ivan Daschinsky 
>>>>> написал(а):
>>>>>> 
>>>>>> Val, Pavel both of you are right, but on the other hand there are
>> some
>>>>>> other tasks
>>>>>> 
>>>>>> 1. Distributed tracing.
>>>>>> 2. Custom metrics/measurements
>>>>>> 3. Auth and some related tasks (i.e. ingests full User info by
>> calling
>>>>> some
>>>>>> auth service in middleware).
>>>>>> 
>>>>>> Do you both think that this is a good idea in business code?
>>>>>> 
>>>>>> Without this functionality, our service grid cannot compete with
>> grpc
>>>> and
>>>>>> others as microservice framework, unfortunately.
>>>>>> 
>>>>>> But if we introduce limited support for this "request headers", it
>> can
>>>>>> drastically improves this aspects of our service grid framework.
>>>>>> 
>>>>>> 
>>>>>> пн, 11 окт. 2021 г. в 00:48, Valentin Kulichenko <
>>>>>> valentin.kuliche...@gmail.com>:
>>>>>> 
>>>>>>> I agree with Pavel. The suggested approach is indeed utilized
>> quite
>>>>>>> frequently, but it's inherently error-prone.
>>>>>>> 
>>>>>>> The main issue is that it creates implicit assumptions about the
>>>>> behavior
>>>>>>> of both the service and the user's code. For example, if the
>> user's
>>>> code
>>>>>>> must provide a username, what if it doesn't? I assume it will get
>> an
>>>>> error,
>>>>>>> which is very counterintuitive. Even more importantly, how should
>> one
>>>>> learn
>>>>>>> about this requirement in the first place? It is not reflected in
>> the
>>>>> API
>>>>>>> in any way - and that's a big problem.
>>>>>>> 
>>>>>>> The fact that the service implementor needs to update the API
>> methods
>>>>> when
>>>>>>> such requirements are introduced is actually a good thing, in my
>>>>> opinion.
>>>>>>> This forces the developer to stop and think about how the updated
>> API
>>>>>>> should look like and how to make sure it's backward-compatible (or
>>>> not,
>>>>> in
>>>>>>> case the new requirements are mandatory). Doing this through an
>>>> external
>>>>>>> context is basically the equivalent of saying "let the end user
>> deal
>>>>> with
>>>>>>> this". Not a good practice, in my view.
>>>>>>> 
>>>>>>> Conversely, passing everything exclusively via method arguments
>>>>> guarantees
>>>>>>> that:
>>>>>>> 
>>>>>>>  - The user's code is always compliant with the service
>> contract. You
>>>>>

Re: [DISCUSS] Custom service proxy context

[Nikolay Izhikov]

> and it fully switches to annotation-based injection.

+1 to do it.

> 19 окт. 2021 г., в 22:14, Valentin Kulichenko  
> написал(а):
> 
> That's actually a good point. In Java, we can do the following:
> 1. Add init/execute/cancel methods without parameters.
> 2. Add default no-op implementations for the new methods (this is required
> to preserve compatibility).
> 3. For old methods that take ServiceContext as a parameter, add default
> implementations that delegate to new methods.
> 4. Deprecate the old methods on the API.
> 5. On the implementation level, still use the old methods (again - for
> compatibility).
> 6. Finally, add a @ServiceContextResource annotation to inject
> ServiceContext.
> 
> If I haven't missed anything, this is not a breaking change, and it fully
> switches to annotation-based injection.
> 
> I'm not sure this is possible in .NET though.
> 
> -Val
> 
> On Tue, Oct 19, 2021 at 11:47 AM Pavel Pereslegin  wrote:
> 
>>> Removing parameters from a public interface method is a breaking change,
>>> or do you mean something else?
>> 
>> Sorry, I meant that we can inject the service context, but leave it
>> available in the init/execute/cancel methods and add a default "no-op"
>> implementation in the interface for them. Can we do this?
>> 
>>> Regarding .NET - let's have a separate ticket for that?
>> If we decide to "inject" a service context - this should be done in a
>> separate ticket.
>> If you are talking about "proxy service context" - I can split it into
>> 3 parts (java, Net, and thin clients)
>> 
>> вт, 19 окт. 2021 г. в 21:23, Pavel Tupitsyn :
>>> 
>>> Pavel,
>>> 
>>>> From my point of view, this should not break anything
>>> Removing parameters from a public interface method is a breaking change,
>>> or do you mean something else?
>>> 
>>> Regarding .NET - let's have a separate ticket for that?
>>> We can discuss and implement Java changes first.
>>> 
>>> On Tue, Oct 19, 2021 at 8:27 PM Pavel Pereslegin 
>> wrote:
>>> 
>>>> Thanks a lot for your suggestions.
>>>> 
>>>>> We might consider injecting the ServiceContext instead of passing it
>> to
>>>>> IgniteService methods, but I believe this will be a breaking change?
>>>> 
>>>> From my point of view, this should not break anything. We can inject a
>>>> service context when initializing a service and keep it accessible in
>>>> state transition methods (init/execute/cancel).
>>>> Currently, in .Net ServiceContext doesn't share the same instance, but
>>>> this can be reworked - for example, we can store the service context
>>>> (with a reference to the service) in the resource registry instead of
>>>> the service itself.
>>>> 
>>>> But I don't see much usability improvement with such a feature if the
>>>> user still needs to implement state transition methods. I think it
>>>> would be nice to add default "no-op" implementations for them.
>>>> Unfortunately, we are currently unable to do the same in .Net because
>>>> such a feature is not supported in C# 4.0 (it's available in C# 8.0).
>>>> 
>>>> Can we add default "no-op" implementations for init/execute/cancel
>>>> methods in Java and leave them unchanged in .Net?
>>>> 
>>>> вт, 19 окт. 2021 г. в 18:51, Valentin Kulichenko
>>>> :
>>>>> 
>>>>> I support #2, because we already have the ServiceContext. Having
>>>>> both ServiceContext and @ServiceRequestContextResource that injects
>> some
>>>>> function (or any other mechanism for that matter) will be VERY
>> confusing.
>>>>> Let's keep it simple.
>>>>> 
>>>>> At the same time, I do agree with Nikolay that injection is the
>> approach
>>>>> taken across the platform, so I'm not sure why we are not using it
>> here.
>>>> We
>>>>> might consider injecting the ServiceContext instead of passing it to
>>>>> IgniteService methods, but I believe this will be a breaking change?
>>>>> 
>>>>> -Val
>>>>> 
>>>>> On Tue, Oct 19, 2021 at 4:49 AM Ivan Daschinsky >> 
>>>> wrote:
>>>>> 
>>>>>> I am for limiting types of attributes values only to UTF-8 strings
>> an

Ci and ci2 settings synchronization

[Nikolay Izhikov]

Hello, Igniters.

As you may know, recently, Sberbank sponsored servers capacity to setup new 
instance of Team City [1]  to run Ignite tests.

Right now, ci2 is fully ready for daily usage.
We have exported and prepared Kotlin scripts [2] to keep TC settings as a code 
[3].

I think we should adjust both TC instances to use the same settings and run 
tests in the same way.

ci.ignite.apache.org is sponsored and run by Grid Gain so 

Denis, Valentin, Petr, let's have a call to discuss how and when we can 
implement these changes?

[1] https://ci2.ignite.apache.org
[2] https://github.com/willy983/ci_tc_configuration
[3] https://www.jetbrains.com/help/teamcity/kotlin-dsl.html


> 3 авг. 2021 г., в 18:33, Dmitry Pavlov  написал(а):
> 
> Hi Igniters,
> 
> I'm glad to announce that SberTech made an internal aggreement to sponsor a 
> computing power to provide backup TeamCity instance. This instance is 
> intended to be a geo-independent, secondary instance with availablility for 
> community members. 
> 
> Thanks to JetBrains for providing license keys for TeamCity as their part of 
> opensource sponsoring program.
> 
> Most likely, we'll setup some domain name as tc.i.a.o or ci2.i.a.o. Please 
> share your vision what would be most obvious.
> 
> After a private discussions with Vitaliy Osipov and Petr Ivanov, I do believe 
> that it would be possible to preserve current registered users. Build 
> configurations are to be the same for the secondary instance. Technical 
> details on how is could be achieved will be available later (most likely 
> we'll start a sepearate thread to dicuss that).
> 
> You are more than welcome to be an early adopters.
> 
> Sincerely,
> Dmitriy Pavlov
> 

Re: Apache Ignite 2.12 RELEASE [Time, Scope, Manager]

[Nikolay Izhikov]

Hello.

In the scope of IEP-80 Breaking changes in Ignite-2.x PR for the following API 
are prepared:

1. MVCC 
- https://issues.apache.org/jira/browse/IGNITE-15757 
- https://github.com/apache/ignite/pull/9516

2. LOCAL caches 
- https://issues.apache.org/jira/browse/IGNITE-15756 
- https://github.com/apache/ignite/pull/9515

3. CacheConfiguration#rebalanceDelay 
- https://issues.apache.org/jira/browse/IGNITE-15764 
- https://github.com/apache/ignite/pull/9515

Please, review.


> 21 окт. 2021 г., в 20:33, Nikita Amelchev  написал(а):
> 
> Petr, thank you!
> 
> чт, 21 окт. 2021 г. в 18:19, Petr Ivanov :
>> 
>> Try now, please.
>> 
>>> On 21 Oct 2021, at 17:47, Nikita Amelchev  wrote:
>>> 
>>> Hi,
>>> 
>>> Petr, could you grant me TC permissions to run release builds please?
>>> 
>>> ср, 20 окт. 2021 г. в 10:25, Pavel Tupitsyn :
>>> 
 
 Igniters,
 
 I've filed a blocker - release configurations on TeamCity should be fixed
 after recent .NET project structure refactoring:
 https://issues.apache.org/jira/browse/IGNITE-15779
 
 I'm on vacation right now with limited internet access, I'll work on this
 ASAP next week.
 
 On Wed, Oct 13, 2021 at 2:33 PM Nikita Amelchev 
 wrote:
 
> Igniters,
> 
> Scope freeze is planned for Friday, October 15.
> I suggest creating the release branch on that date.
> 
> The wiki page with the release info. [1] Please, check the fix version
> of your issues.
> 
> There are no unresolved blockers.
> There are 3 important issues in the review state. [2]
> 
> Also, please, pay attention to unresolved documentation issues. [3]
> 
> [1] https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.12
> [2]
> https://issues.apache.org/jira/issues/?jql=(resolution%20%3D%20Unresolved%20AND%20project%20%3D%20%27Ignite%27%20AND%20type%20in%20(%22New%20Feature%22%2C%20Task%2C%20Sub-task%2C%20Improvement)%20AND%20fixVersion%20in%20(%272.12%27)%20AND%20(component%20is%20EMPTY%20OR%20component%20not%20in%20(documentation))%20AND%20(labels%20in%20(%27important%27)%20OR%20Flags%20%3D%20Important))%20order%20by%20summary
> [3]
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.12%27)%20AND%20status%20NOT%20IN%20(Resolved%2C%20Closed)%20AND%20component%20in%20(documentation)%20ORDER%20BY%20priority%20%20%20%20%20%20%20%20%20
> 
> пт, 1 окт. 2021 г. в 19:06, Pavel Tupitsyn :
>> 
>> I'm working on IGNITE-15504 and plan to finish it early next week.
>> 
>> On Fri, Oct 1, 2021 at 6:08 PM Nikita Amelchev 
> wrote:
>> 
>>> Igniters,
>>> 
>>> I have created the wiki page for the 2.12 release. [1]
>>> 
>>> The release scope contains 1 issue that is marked as a blocker:
>>> IGNITE-15504 .NET: SDK 2.1 is out of support, make sure project can be
>>> developed with newer SDKs [2]
>>> 
>>> Folks, do not forget to document features (65 issues still not
> resolved).
>>> [3]
>>> 
>>> Maxim, Ivan,
>>> +1 to add these issues to the 2.12 scope.
>>> 
>>> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.12
>>> [2] https://issues.apache.org/jira/browse/IGNITE-15504
>>> [3]
>>> 
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.12%27)%20AND%20status%20NOT%20IN%20(Resolved%2C%20Closed)%20AND%20component%20in%20(documentation)%20ORDER%20BY%20priority%20%20%20%20%20%20%20%20%20
>>> 
>>> ср, 29 сент. 2021 г. в 11:28, Ivan Daschinsky :
 
 Created issues related building ODBC module, running C++ suites and
 removing VS project files
 
 1. https://issues.apache.org/jira/browse/IGNITE-15637
 2.https://issues.apache.org/jira/browse/IGNITE-15636
 
 вс, 26 сент. 2021 г. в 19:50, Ivan Daschinsky :
 
> +1
> 
> Let's consider building odbc driver using Visual C++ 2017?
> 
> 2015, 2017 and 2019 share the same redistributable package and it
> is
>>> the
> default one in Windows 10.
> 
> Also, let's remove vs project files, since the most of you all
> agree
>>> that
> it is redundant.
> 
> If both suggestions are ok, I will create ticketa soon.
> 
> вс, 26 сент. 2021 г., 17:10 Pavel Tupitsyn :
> 
>> +1
>> 
>> On Fri, Sep 24, 2021 at 8:04 PM Maxim Muzafarov <
> mmu...@apache.org>
>> wrote:
>> 
>>> Hello Nikita,
>>> 
>>> +1 if you will lead this release.
>>> 
>>> 
>>> I'd suggest including into the release scope this one issue too:
>>> 
>>> Snapshot restore m

Re: [DISCUSSION][IEP-80] Breaking changes in Ignite-2.x

[Nikolay Izhikov]

Hello.

In the scope of IEP-80 Breaking changes in Ignite-2.x PR for the following API 
are prepared:

1. MVCC 
- https://issues.apache.org/jira/browse/IGNITE-15757 
- https://github.com/apache/ignite/pull/9516

2. LOCAL caches 
- https://issues.apache.org/jira/browse/IGNITE-15756 
- https://github.com/apache/ignite/pull/9515

3. CacheConfiguration#rebalanceDelay 
- https://issues.apache.org/jira/browse/IGNITE-15764 
- https://github.com/apache/ignite/pull/9515

Please, review.

> 15 окт. 2021 г., в 16:23, Nikolay Izhikov  написал(а):
> 
> THanks, Maksim.
> 
> Tickets included in IEP scope and marked for d&r in 2.12-2.13
> 
>> 15 окт. 2021 г., в 16:03, Maxim Muzafarov  написал(а):
>> 
>> Let's deprecate RebalanceDelay and RebalanceMode=NONE also.
>> 
>> [1] https://issues.apache.org/jira/browse/IGNITE-12662
>> [2] https://issues.apache.org/jira/browse/IGNITE-14613
>> 
>> On Fri, 15 Oct 2021 at 15:46, Anton Vinogradov  wrote:
>>> 
>>> +1
>>> 
>>> On Fri, Oct 15, 2021 at 3:41 PM Nikita Amelchev 
>>> wrote:
>>> 
>>>> +1 for deprecation in the 2.12 release
>>>> 
>>>> пт, 15 окт. 2021 г. в 15:35, Nikolay Izhikov :
>>>>> 
>>>>> Hello, Igniters.
>>>>> 
>>>>> I’ve prepared IEP-80 [1] to track breaking changes that should be done
>>>> in Ignite 2.x.
>>>>> 
>>>>> We agreed on the following algorithm to deal with breaking changes:
>>>>> 
>>>>>   - Ignite 2.[x] - deprecate the API + notification of the users.
>>>>>   - Ignite 2.[x+1] - API removal.
>>>>> 
>>>>> 
>>>>> I propose to start these improvements with the d&r (depuration &
>>>> removal) of the following features:
>>>>> 
>>>>>   - LOCAL caches
>>>>>   - MVCC caches
>>>>>   - legacy service grid implementation
>>>>>   - legacy JMX metrics beans.
>>>>> 
>>>>> Deprecation in 2.12
>>>>> Removal in 2.13
>>>>> 
>>>>> Please, share your opinion.
>>>>> 
>>>>> [1]
>>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191334475
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Best wishes,
>>>> Amelchev Nikita
>>>> 
> 

Re: [VOTE][EXTENSION] Release Apache Ignite spring-tx-ext, spring-cache-ext extensions 1.0.0 RC1

[Nikolay Izhikov]

+1 (binding)

> 1 нояб. 2021 г., в 17:41, Denis Magda  написал(а):
> 
> +1 (binding)
> 
> Support to release the integration and then bring in essential automation
> for testing.
> 
> -
> Denis
> 
> 
> On Wed, Oct 27, 2021 at 4:48 AM Nikita Amelchev 
> wrote:
> 
>> Dear Ignite Community,
>> 
>> I have uploaded a release candidate of extensions modules:
>> - spring-tx-ext 1.0.0
>> - spring-cache-ext 1.0.0
>> - spring-data-commons 1.1.0
>> 
>> The following staging can be used for testing:
>> https://repository.apache.org/content/repositories/orgapacheignite-1533
>> 
>> The release candidate of the spring-tx-ext 1.0.0 extension:
>> 
>> https://dist.apache.org/repos/dist/dev/ignite/ignite-extensions/ignite-spring-tx-ext/1.0.0-rc1/
>> 
>> The release candidate of the spring-cache-ext 1.0.0 extension:
>> 
>> https://dist.apache.org/repos/dist/dev/ignite/ignite-extensions/ignite-spring-cache-ext/1.0.0-rc1/
>> 
>> The release candidate of the spring-data-common 1.1.0 dependent module
>> is in the packages mentioned above.
>> 
>> Tags were created:
>> ignite-spring-tx-ext-1.0.0-rc1
>> ignite-spring-cache-ext-1.0.0-rc1
>> ignite-spring-data-commons-1.1.0-rc1
>> 
>> 
>> https://gitbox.apache.org/repos/asf?p=ignite-extensions.git;a=commit;h=refs/tags/ignite-spring-tx-ext-1.0.0-rc1
>> 
>> RELEASE NOTES:
>> 
>> https://gitbox.apache.org/repos/asf?p=ignite-extensions.git;a=blob;f=RELEASE_NOTES.txt;h=c6b2a393e042ce956bfefd5d7d92d5ab4ff04cda;hb=HEAD
>> 
>> DOCUMENTATION:
>> Documentation of the spring-tx-ext extension:
>> 
>> https://github.com/apache/ignite/blob/master/docs/_docs/extensions-and-integrations/spring/spring-tx.adoc
>> 
>> Documentation of the spring-cache-ext extension:
>> 
>> https://github.com/apache/ignite/blob/master/docs/_docs/extensions-and-integrations/spring/spring-caching.adoc
>> 
>> The vote is formal, see voting guidelines
>> https://www.apache.org/foundation/voting.html
>> 
>> +1 - to accept the extensions modules listed in the thread
>> 0 - don't care either way
>> -1 - DO NOT accept either of the extensions modules listed in the
>> thread (explain why)
>> 
>> The vote will hold for 5 days and will end on November 1, 2021, 10:00 UTC
>> https://www.timeanddate.com/countdown/generic?iso=20211101T10&p0=0
>> 
>> --
>> Best wishes,
>> Amelchev Nikita
>> 

Re: [DISCUSSION] Array to BinaryObject serialization

[Nikolay Izhikov]

Hello, Igniters.

I've prepared implementation of support of BinaryArray.
Please, join the review.

Ticket - https://issues.apache.org/jira/browse/IGNITE-14742
PR - https://github.com/apache/ignite/pull/9490



ср, 19 мая 2021 г. в 14:44, Ilya Kasnacheev :

> Hello!
>
> We don't have to do it in toBinary() - we may preserve its behavior if you
> like, and only do that transition for Cache API, where additional support
> may be added to
> e.g.
> org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 19 мая 2021 г. в 14:23, Nikolay Izhikov :
>
> > Ilya.
> >
> > Actually, current behaviour described even in documentation [1]
> >
> >
> > > Note that not all objects are converted to the binary object format.
> The
> > following classes are never converted (e.g., the toBinary(Object) method
> > returns the original object, and instances of these classes are stored
> > without changes):
> > ...
> > > ... arrays of objects (but the objects inside them are reconverted if
> > they are binary)
> >
> > [1]
> >
> https://ignite.apache.org/docs/latest/key-value-api/binary-objects#enabling-binary-mode-for-caches
> >
> > > 19 мая 2021 г., в 14:12, Ilya Kasnacheev 
> > написал(а):
> > >
> > > Hello!
> > >
> > > Why do you need to take compatibility into account here at all?
> > > You can start writing entries to cache in new format while reading both
> > old
> > > and new format.
> > >
> > > I consider returning Object[] instead of ConcreteType[] a bug so it's
> > > totally OK to start returning the "better" ConcreteType[] instead.
> > >
> > > I think you can just fix the bug while preserving accessibility of old
> > > (pre-bugfix) data as it was pre-2.11.
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > ср, 19 мая 2021 г. в 14:08, Nikolay Izhikov :
> > >
> > >> Ilya,
> > >>
> > >>> Maybe we should just automate that, e.g., whenever user stores
> Type[],
> > >> always store one-field ArrayWrapper object, and automatically unwrap
> it
> > on
> > >> get()
> > >>
> > >> Yes, I’m talking about this opportunity from the beginning of the
> > thread.
> > >>
> > >>> 1. Implement binary serialization that correctly Ser and Deser array
> > >> using some kind of the wrapper (BinaryArrayWrapper).
> > >>
> > >>
> > >>> 19 мая 2021 г., в 14:05, Ilya Kasnacheev 
> > >> написал(а):
> > >>>
> > >>> Hello!
> > >>>
> > >>> Yes, it does not look pretty, I agree. But I'm saying that I've not
> > >>> encountered this issue on the user list before, and it can probably
> be
> > >>> easily countered by storing some one-field ArrayWrapper object in the
> > >> cache
> > >>> as value.
> > >>>
> > >>> Maybe we should just automate that, e.g., whenever user stores
> Type[],
> > >>> always store one-field ArrayWrapper object, and automatically unwrap
> it
> > >> on
> > >>> get()
> > >>> This way we may not even need any changes to storage format, if
> binary
> > >>> marshaller does not mind.
> > >>>
> > >>> Regards,
> > >>>
> > >>> Regards,
> > >>> --
> > >>> Ilya Kasnacheev
> > >>>
> > >>>
> > >>> ср, 19 мая 2021 г. в 13:33, Nikolay Izhikov :
> > >>>
> > >>>> Igniters.
> > >>>>
> > >>>> Just to clarify the issue:
> > >>>>
> > >>>> ```
> > >>>> public class BinaryObjectTest extends GridCommonAbstractTest {
> > >>>>   /** */
> > >>>>   @Test
> > >>>>   public void testArray() throws Exception {
> > >>>>   Ignite ign = startGrid();
> > >>>>
> > >>>>   IgniteCache cache =
> > >>>> ign.createCache("my-cache");
> > >>>>
> > >>>>   cache.put(1, new TestClass1[] {new TestClass1(), new
> > >>>> TestClass1()});
> > >>>>   TestClass1[] obj = cache.get(1); //This will fail with

Re: Issue - IGNITE-15348

[Nikolay Izhikov]

Hello, Andrei.

Welcome to the Apache Ignite community.

I’ve added you to the Contributors list.
You can assign JIRA to yourself.

> 15 нояб. 2021 г., в 15:26, Andrei Demidov  
> написал(а):
> 
> Hello Ignite Community!
> 
> My name is Andrei. I want to contribute to Apache Ignite and want to start 
> with this issue - 
> IGNITE-15348, my JIRA 
> username andydem. Please, assign it to me.
> 
> Any help on this will be appreciated.
> Thanks!
> 
> 
> CONFIDENTIALITY CAUTION AND DISCLAIMER
> 
> This message is intended only for the use of the individual(s) or entity(ies) 
> to which it is addressed and contains information that is legally privileged 
> and confidential. If you are not the intended recipient, or the person 
> responsible for delivering the message to the intended recipient, you are 
> hereby notified that any dissemination, distribution or copying of this 
> communication is strictly prohibited. All unintended recipients are obliged 
> to delete this message and destroy any printed copies.
> 

Re: Apache Ignite 2.12 RELEASE [Time, Scope, Manager]

[Nikolay Izhikov]

Hello.

> [2] https://issues.apache.org/jira/browse/IGNITE-15781

Cherry-picked to 2.12.

> 19 нояб. 2021 г., в 10:12, Pavel Tupitsyn  написал(а):
> 
> I'd like to add this simple fix as well [1]
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-15954
> 
> On Fri, Nov 19, 2021 at 9:50 AM Maksim Timonin 
> wrote:
> 
>> Hi, I want to add to 2.12 a fix [1] of a bug [2]. It affects users of
>> metrics in case there are too many partitions (> 2), we observed
>> extensive heap usage.
>> 
>> [1] https://github.com/apache/ignite/pull/9518/files
>> [2] https://issues.apache.org/jira/browse/IGNITE-15781
>> 
>> On Thu, Nov 18, 2021 at 12:59 PM Maksim Timonin 
>> wrote:
>> 
>>> Hi, guys, we have a blocker [1]. NPE for some SQL queries. I will fix it
>>> in 1-2 days.
>>> 
>>> https://issues.apache.org/jira/browse/IGNITE-15943
>>> 
>>> On Mon, Nov 15, 2021 at 2:57 PM Maxim Muzafarov 
>> wrote:
>>> 
 Folks,
 
 I've fixed almost all the review comments [1]. Hopefully, the changes
 will be merged by the end of the week.
 
 [1] https://issues.apache.org/jira/browse/IGNITE-14744
 
 On Mon, 8 Nov 2021 at 15:42, Nikita Amelchev 
 wrote:
> 
> Hello, Pavel.
> 
> The code freeze is blocked by two issues discussed above [1] [2]. The
> issues are in the final review state and should be merged soon.
> 
> The bugfix is useful and can be cherry-picked, thank you.
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-15530
> [2] https://issues.apache.org/jira/browse/IGNITE-14744
> 
> 
> пн, 8 нояб. 2021 г. в 14:46, Pavel Tupitsyn :
> 
>> 
>> Igniters,
>> 
>> What's the code freeze status?
>> Can I cherry-pick a bugfix [1] ?
>> 
>> [1] https://issues.apache.org/jira/browse/IGNITE-15807
>> 
>> On Tue, Nov 2, 2021 at 4:39 PM Ivan Daschinsky >> 
 wrote:
>> 
>>> Thanks, cherry-picked to master.
>>> 
>>> вт, 2 нояб. 2021 г. в 16:00, Nikita Amelchev <
>> namelc...@apache.org
> :
>>> 
 Ivan,
 
 Yes, the patch can be cherry-picked.
 
 вт, 2 нояб. 2021 г. в 15:55, Ivan Daschinsky <
>> ivanda...@gmail.com
> :
> 
> Since code freeze is postponed, may be I can cherry-pick my
 patch
>>> already
> merged to master?
> https://issues.apache.org/jira/browse/IGNITE-15806
> 
> пт, 29 окт. 2021 г. в 20:24, Nikita Amelchev <
 namelc...@apache.org>:
> 
>> Maksim T., Maxim M., Ok.
>> 
>> I think we can move code freeze to +1 week to await these
 issues.
>> 
>> пт, 29 окт. 2021 г. в 15:53, Maxim Muzafarov <
 mmu...@apache.org>:
>>> 
>>> Folks,
>>> 
>>> I've previously mentioned that I'd like to include the [1]
 issue in
>>> the release. Currently, the status is "Patch Available"
>> and
 it's
>>> actively reviewing now. I'd be happy if we wait a couple
>> of
 days
>>> and
>>> add this improvement to the release.
>>> 
>>> I've also created an issue [2] to remove the ambiguity
 output
>>> message
>>> for the snapshot check and idle verify procedures. This is
 a minor
>>> output message change, however, it was directly requested
 by users.
>>> Would you don't mind if we include this issue in the
 release also?
>>> 
>>> [1] https://issues.apache.org/jira/browse/IGNITE-14744
>>> [2] https://issues.apache.org/jira/browse/IGNITE-15849
>>> 
>>> On Fri, 29 Oct 2021 at 14:19, Maksim Timonin <
 timonin.ma...@gmail.com>
>> wrote:
 
 Status update on my issue:
 
 IGNITE-15530 [1] is in "Patch Available" state, Anton
 Vinogradov
>> agreed to
 review this ticket.
 
 Also, I'm preparing IndexQuery docs in ticket
 IGNITE-15745 [2].
>>> I'm
>> going
 to submit a patch with docs today/tomorrow, and want to
 include
 them in
 2.12 too.
 
 [1] https://issues.apache.org/jira/browse/IGNITE-15530
 [2] https://issues.apache.org/jira/browse/IGNITE-15745
 
 On Thu, Oct 28, 2021 at 11:06 AM Ilya Kasnacheev <
>> ilya.kasnach...@gmail.com>
 wrote:
 
> Hello!
> 
> I have prepared a fix for sqlline.sh -e:
> https://github.com/apache/ignite/pull/9536
> 
> If smbd reviews it, we can put it on 2.12.
> 
> Regards,
> --
> Ilya Kasnacheev
> 
> 
> чт, 28 окт. 2021 г. в 10:39, Nikita Amelchev <
 namelc...@apache.org>:
> 
>> Maksim, ok.
>> 
>> Ivan,
>>

Re: Apache Ignite 2.12 RELEASE [Time, Scope, Manager]

[Nikolay Izhikov]

Hello.

One more tiny fix that will improve java thin client performance in certain 
cases [1]
I think it worth to cherry-pick it to 2.12 release.

Any objections?

[1] https://issues.apache.org/jira/browse/IGNITE-15924

> 19 нояб. 2021 г., в 14:56, Nikita Amelchev  написал(а):
> 
> Vladimir, +1.
> 
> I have cherry-picked the issue.
> 
> пт, 19 нояб. 2021 г. в 14:53, Steshin Vladimir :
>> 
>> Hi all.
>> 
>> 
>> Let's add simple and useful thin-client-pool monitoring in 2.12:
>> 
>> https://issues.apache.org/jira/browse/IGNITE-15183
>> 
>> https://github.com/apache/ignite/pull/9525
>> 
>> 
>> 19.11.2021 09:50, Maksim Timonin пишет:
>>> Hi, I want to add to 2.12 a fix [1] of a bug [2]. It affects users of
>>> metrics in case there are too many partitions (> 2), we observed
>>> extensive heap usage.
>>> 
>>> [1] https://github.com/apache/ignite/pull/9518/files
>>> [2] https://issues.apache.org/jira/browse/IGNITE-15781
>>> 
>>> On Thu, Nov 18, 2021 at 12:59 PM Maksim Timonin 
>>> wrote:
>>> 
 Hi, guys, we have a blocker [1]. NPE for some SQL queries. I will fix it
 in 1-2 days.
 
 https://issues.apache.org/jira/browse/IGNITE-15943
 
 On Mon, Nov 15, 2021 at 2:57 PM Maxim Muzafarov  wrote:
 
> Folks,
> 
> I've fixed almost all the review comments [1]. Hopefully, the changes
> will be merged by the end of the week.
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-14744
> 
> On Mon, 8 Nov 2021 at 15:42, Nikita Amelchev 
> wrote:
>> Hello, Pavel.
>> 
>> The code freeze is blocked by two issues discussed above [1] [2]. The
>> issues are in the final review state and should be merged soon.
>> 
>> The bugfix is useful and can be cherry-picked, thank you.
>> 
>> [1] https://issues.apache.org/jira/browse/IGNITE-15530
>> [2] https://issues.apache.org/jira/browse/IGNITE-14744
>> 
>> 
>> пн, 8 нояб. 2021 г. в 14:46, Pavel Tupitsyn :
>> 
>>> Igniters,
>>> 
>>> What's the code freeze status?
>>> Can I cherry-pick a bugfix [1] ?
>>> 
>>> [1] https://issues.apache.org/jira/browse/IGNITE-15807
>>> 
>>> On Tue, Nov 2, 2021 at 4:39 PM Ivan Daschinsky 
> wrote:
 Thanks, cherry-picked to master.
 
 вт, 2 нояб. 2021 г. в 16:00, Nikita Amelchev > :
> Ivan,
> 
> Yes, the patch can be cherry-picked.
> 
> вт, 2 нояб. 2021 г. в 15:55, Ivan Daschinsky > :
>> Since code freeze is postponed, may be I can cherry-pick my
> patch
 already
>> merged to master?
>> https://issues.apache.org/jira/browse/IGNITE-15806
>> 
>> пт, 29 окт. 2021 г. в 20:24, Nikita Amelchev <
> namelc...@apache.org>:
>>> Maksim T., Maxim M., Ok.
>>> 
>>> I think we can move code freeze to +1 week to await these
> issues.
>>> пт, 29 окт. 2021 г. в 15:53, Maxim Muzafarov <
> mmu...@apache.org>:
 Folks,
 
 I've previously mentioned that I'd like to include the [1]
> issue in
 the release. Currently, the status is "Patch Available" and
> it's
 actively reviewing now. I'd be happy if we wait a couple of
> days
 and
 add this improvement to the release.
 
 I've also created an issue [2] to remove the ambiguity
> output
 message
 for the snapshot check and idle verify procedures. This is
> a minor
 output message change, however, it was directly requested
> by users.
 Would you don't mind if we include this issue in the
> release also?
 [1] https://issues.apache.org/jira/browse/IGNITE-14744
 [2] https://issues.apache.org/jira/browse/IGNITE-15849
 
 On Fri, 29 Oct 2021 at 14:19, Maksim Timonin <
> timonin.ma...@gmail.com>
>>> wrote:
> Status update on my issue:
> 
> IGNITE-15530 [1] is in "Patch Available" state, Anton
> Vinogradov
>>> agreed to
> review this ticket.
> 
> Also, I'm preparing IndexQuery docs in ticket
> IGNITE-15745 [2].
 I'm
>>> going
> to submit a patch with docs today/tomorrow, and want to
> include
> them in
> 2.12 too.
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-15530
> [2] https://issues.apache.org/jira/browse/IGNITE-15745
> 
> On Thu, Oct 28, 2021 at 11:06 AM Ilya Kasnacheev <
>>> ilya.kasnach...@gmail.com>
> wrote:
> 
>> Hello!
>> 
>> I have prepared a fix for sqlline.sh -e:
>> https://github.com/apache/ignite/pull/9536
>> 
>> If smbd reviews it, we can put it on 2.12.
>>>

Re: Apache Ignite 2.12 RELEASE [Time, Scope, Manager]

[Nikolay Izhikov]

> [1] https://issues.apache.org/jira/browse/IGNITE-15951

Cherry-picked to 2.12

> 23 нояб. 2021 г., в 16:38, Nikita Amelchev  написал(а):
> 
> Hi, +1 to cherry-pick.
> 
> вт, 23 нояб. 2021 г. в 15:25, Ivan Daschinsky :
>> 
>> Hi! Lets add one simple bug fix, but very useful
>> 
>> https://issues.apache.org/jira/browse/IGNITE-15767
>> 
>> пн, 22 нояб. 2021 г. в 15:22, Pavel Tupitsyn :
>> 
>>> Yes, let's fix those auth issues, there are so many complaints on the user
>>> list.
>>> 
>>> On Mon, Nov 22, 2021 at 1:03 PM Mikhail Petrov 
>>> wrote:
>>> 
>>>> Igniters,
>>>> it seems that issues [1] and [2] that were discovered recently are
>>>> blockers for the 2.12 release.
>>>> 
>>>> [1] https://issues.apache.org/jira/browse/IGNITE-15951
>>>> [2] https://issues.apache.org/jira/browse/IGNITE-15966
>>>> 
>>>> On 22.11.2021 11:04, Nikolay Izhikov wrote:
>>>>> Hello.
>>>>> 
>>>>> One more tiny fix that will improve java thin client performance in
>>>> certain cases [1]
>>>>> I think it worth to cherry-pick it to 2.12 release.
>>>>> 
>>>>> Any objections?
>>>>> 
>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-15924
>>>>> 
>>>>>> 19 нояб. 2021 г., в 14:56, Nikita Amelchev 
>>>> написал(а):
>>>>>> 
>>>>>> Vladimir, +1.
>>>>>> 
>>>>>> I have cherry-picked the issue.
>>>>>> 
>>>>>> пт, 19 нояб. 2021 г. в 14:53, Steshin Vladimir :
>>>>>>> Hi all.
>>>>>>> 
>>>>>>> 
>>>>>>> Let's add simple and useful thin-client-pool monitoring in 2.12:
>>>>>>> 
>>>>>>> https://issues.apache.org/jira/browse/IGNITE-15183
>>>>>>> 
>>>>>>> https://github.com/apache/ignite/pull/9525
>>>>>>> 
>>>>>>> 
>>>>>>> 19.11.2021 09:50, Maksim Timonin пишет:
>>>>>>>> Hi, I want to add to 2.12 a fix [1] of a bug [2]. It affects users
>>> of
>>>>>>>> metrics in case there are too many partitions (> 2), we observed
>>>>>>>> extensive heap usage.
>>>>>>>> 
>>>>>>>> [1] https://github.com/apache/ignite/pull/9518/files
>>>>>>>> [2] https://issues.apache.org/jira/browse/IGNITE-15781
>>>>>>>> 
>>>>>>>> On Thu, Nov 18, 2021 at 12:59 PM Maksim Timonin <
>>>> timonin.ma...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi, guys, we have a blocker [1]. NPE for some SQL queries. I will
>>>> fix it
>>>>>>>>> in 1-2 days.
>>>>>>>>> 
>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-15943
>>>>>>>>> 
>>>>>>>>> On Mon, Nov 15, 2021 at 2:57 PM Maxim Muzafarov >>> 
>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Folks,
>>>>>>>>>> 
>>>>>>>>>> I've fixed almost all the review comments [1]. Hopefully, the
>>>> changes
>>>>>>>>>> will be merged by the end of the week.
>>>>>>>>>> 
>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-14744
>>>>>>>>>> 
>>>>>>>>>> On Mon, 8 Nov 2021 at 15:42, Nikita Amelchev <
>>> namelc...@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>> Hello, Pavel.
>>>>>>>>>>> 
>>>>>>>>>>> The code freeze is blocked by two issues discussed above [1] [2].
>>>> The
>>>>>>>>>>> issues are in the final review state and should be merged soon.
>>>>>>>>>>> 
>>>>>>>>>>> The bugfix is useful and can be cherry-picked, thank you.
>>>>>>>>>>> 
>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-15530
>>>>>>>>>>> [2] https://issues.apac

Re: Issue - IGNITE-15348

[Nikolay Izhikov]

Hello, Andrei.

I will review your changes shortly.

> 25 нояб. 2021 г., в 14:31, Andrei Demidov  
> написал(а):
> 
> Hi!
> 
> Could somebody please explain me.
> Should I do something for my changes to be merged?
> In contributing guide it's said that changes will be merged by any committer 
> after review.
> 
> Thanks!

Re: [DISCUSSION] [IEP-81] New Cluster Management API (Ignite 2.x)

[Nikolay Izhikov]

Hello, Maxim.

Thanks for the IEP.
I support the intentions and design of the IEP in general but have some 
comments:

> AnnotationCommandScanner, PackageCommandScanner, URICommandScanner

Don’t think we should rely on auto scan class path capabilities.
1. How this automatic scanner will distinguish ignite class from user 
class if they both in node class path and same package like 
«org.apache.ignite.internal»?
2. It seems hard to debug any errors with auto class path scanner. How 
we ensure correct order of scanning in case different jar order in class path?

I think we should go with some kind of  hardcoded «on startup» command 
registration.


> execute(ProxyManagementTask.class, Map atts)

1. Do we have some well-defined place to validate `atts`? 

2. Do we really need to rely on `java.lang.Class` parameter? 
This means executor (thin client) should have all class definitions 
which can be hard for plugins provided class.
Can we rely on some kind of «path array» like `String[] { «baseline», 
«set» }` or `String[] { «user», «add» }`
So the API usage will looks like:

```
String execute(String[] path, Map params)

Map params = new HashMap();

params.put(«—login», «admin»)
params.put(«—password», «mypassw0rd»)

String res = execute(new String[] {«user», «add»}, params);
```


> Design issues …   it is not possible to add and run new management tasks 
> at the cluster runtime (e.g. REPL mode for CLI tools can't be used);

It seems to me as a good thing?
We shouldn’t have ability to add management tasks in runtime from the user side 
because of security reasons.

But, we should be able to run any existing task dynamically in some scripting 
environment - REPL, CLI as you mentioned.


> Features

I think we should focus on:

1. Subsystem then stores all available management tasks
2. Unified execution flow that guarantees all required things like 
authorization, authentication, logging and event generation.
3. Creation of the specific protocol implementation that will expose all 
existing commands **in the same way** through all supported protocols
**without any additional development**.
4. Simplification of new command development and improvement of existing.


> Compatibility

IEP doesn’t clearly define compatibility.
Do we keep all existing commands as is?
Is new subsystem a replacement for current API?

Is new subsystem will co-exist with current API? For how long?

> 26 нояб. 2021 г., в 11:44, Maxim Muzafarov  написал(а):
> 
> Igniters,
> 
> 
> I'd like to propose a new internal cluster management API that will
> simplify new cluster management commands creation and expose new
> commands to CLI, REST, JMX Ignite interfaces without any additional
> actions from the engineer adding a new command. This main goal of the
> IEP is supposed to be available after the implementation of the 1-st
> phase. From my point of view, the implementation will also provide
> some additional features like auto ASCII-docs generation which can be
> achieved with minor code changes.
> 
> Please, take a look at the IEP-81 [1] with a detailed explanation of
> my proposal. Below you can find some crucial points from it.
> 
> 
> = Current Limitations =
> 
> - The same commands through CLI, REST, JMX APIs don't have the same
> input arguments and use different subsystems for command execution;
> - A new command that is added must be manually exposed to all the
> Ignite APIs (new implementation required for each new command being
> added);
> - New commands can't be added via Ignite Plugins and exposed to API;
> - The own binary protocol is used (GridClient) for command executions
> instead of the ignite thin client (IgniteClient);
> - Security and role model: a user have to add compute tasks
> permissions the same time as adding permissions for the process he
> intended to use.
> - New commands can't be added or executed at runtime
> 
> 
> = Crutial Impelemntation Notes =
> 
> 1. Create a one for all proxy compute task gate that will accept a map
> of parameters for preparing management command based on input
> parameters and executing it on ignite nodes.
> For instance:
> 
> IgniteClient.compute().execute(ProxyManagementTask.class.getName(), attrs);
> 
> Map:
> baseline.add=execute
> baseline.add.projection=SINGLE
> baseline.add.nodes=[consistendtId3, consistendeId4]
> 
> 2. Create a CommandRegisty that will contain all available commanded
> on the local ignite node. Commands will be added by command scanners
> e.g. AnnotationCommandScanner, PackageCommandScanner,
> URICommandScanner as well as registered manually at runtime by calling
> `add` method on command registry. The CommandRegistry will also be
> available for the thin clients in a standalone mode.
> 
> 3. Prepare a command parsers for REST, JMX, CLI interfaces that will
> use command registry and calling the proxy management task right away
> with correct task input parameters.
> 
> 4. Che

Re: [DISCUSSION] Remove outdated ignite-sys-cache

[Nikolay Izhikov]

Vyacheslav, Val, can you please clarify - What is the issue if third-party 
plugins will create «ignite-sys-cache» from the code?

Like just replacing `Ignite#cache` with the `Ignite#getOrCreateCache`.

> 2 дек. 2021 г., в 16:13, Вячеслав Коптилин  
> написал(а):
> 
> Hello Maxim,
> 
>> I don't understand why you are using *backwards compatibility* for
> completely internal things.
>> Why you are thinking in terms of users usage when are talking about
> ignite-sys-cache but not thinking when refactoring
> First of all, we are talking about all plugin developers. It does not
> matter it is open-source or proprietary plugins.
> Honestly, I don't have a list of all possible plugins that have already
> been developed and available under the ASF license, for instance.
> Do you have such a list? Can you be sure that this change will not affect
> anyone?
> 
>> I don't understand why you are using *backwards compatibility* for
> completely internal things.
>> Why you are thinking in terms of users usage when are talking about
> ignite-sys-cache but not thinking when refactoring
> The system cache was the crucial thing in the architecture of Apache Ignite
> 1.x and 2.x (at least 2.1 - 2.11?)
> 
>> All the internals have been reworked and nobody even mentioned that it
> may affect some of the plugin developers.
> Yep, perhaps, some internal parts of Apache Ignite were reworked in order
> to avoid using the system cache.
> However, it is still a part of Ignite and it works and can be used in
> plugins.
> Honestly, the mentioned alternative, I mean the distributed metastorage, is
> the INTERNAL thing as well.
> It means that plugin developer depends on INTERNAL entities. (it does not
> matter system-cache/metastorage)
> IMHO, such questions should be CAREFULLY discussed with no rush.
> 
>> I do not propose to rush with the ignite-sys-cache removal. I'd like to
> collect all the technical stuff that we depend on, fix all of them and
> after everything will be ready do a final removal.
> Good! We are on the same page!
> 
>> 1. Add deprecation for the 2.12 release. I hope it is still possible.
> If I am not mistaken, the code freeze was October 29. I think it is too
> late to introduce this change. We can add deprecation for the 2.13 release.
> 
>> Apache Ignite core still have some dependencies on ignite-sys-cache that
> should fix for 2.13
> Ok, I agree. We need to try to find out all edge cases and add new
> abilities to the metastorage in order to cover all known
> issues/requirements.
> 
>> Remove the system cache in 2.14.
> It depends on our progress with improving the distributed metastorage. In
> general, I hope it will be possible.
> 
> Thanks,
> S.
> 
> чт, 2 дек. 2021 г. в 13:36, Maxim Muzafarov :
> 
>> Pavel,
>> 
>> 
>> I don't understand why you are using *backwards compatibility* for
>> completely internal things. Why you are thinking in terms of users
>> usage when are talking about ignite-sys-cache but not thinking when
>> refactoring, for instance, all the checkpoint classes? Take a look at
>> the [1] issue. All the internals have been reworked and nobody even
>> mentioned that it may affect some of the plugin developers. This is
>> really strange, so I can't agree with you.
>> 
>> 
>> I do not propose to rush with the ignite-sys-cache removal. I'd like
>> to collect all the technical stuff that we depend on, fix all of them
>> and after everything will be ready do a final removal. Slava already
>> mentioned some crucial cases, so I hope you and Val also help with the
>> rest of them. Let's be more precise in terms *what we need to fix*.
>> 
>> 
>> I've made some investigations related to the ignite-sys-cache and here
>> is my proposal:
>> 
>> 1. Add deprecation for the 2.12 release. I hope it is still possible.
>> 
>> 2. Apache Ignite core still have some dependencies on ignite-sys-cache
>> that should fix for 2.13:
>> 
>> 2.1. CLUSTER_NAME property: when it's not set the `deploymentId` of
>> system cache is used. Let's move it to metastorage.
>> 2.2. When the Security is enabled each compute task name (hash to be
>> more precise) is written to the ignite-sys-cache [2]. From my point of
>> view, we can remove it in 2.13. Can anyone check?
>> 2.3. Slava mentioned some issues with the distributed metastorage that
>> might be fixed. I think this is doable.
>> 
>> 3. Remove the system cache in 2.14.
>> 
>> 
>> 
>> [1] https://issues.apache.org/jira/browse/IGNITE-13143
>> [2]
>> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/task/GridTaskProcessor.java#L201
>> 
>> On Thu, 2 Dec 2021 at 12:56, Pavel Tupitsyn  wrote:
>>> 
>>> Maxim,
>>> 
 I don't think that we should wait for 3-rd party plugins to be updated
 this is an awful practice when the open-source product releases depend
 on some of the proprietary plugins
>>> 
>>> This makes no sense, sorry.
>>> 
>>> It is not that we depend on 3rd party plugins.
>>> It is that *our users depen

Re: [DISCUSSION] Remove outdated ignite-sys-cache

[Nikolay Izhikov]

I don’t understand how it possible.

Are you talking about plugin that uses some kind of internal API?

> 3 дек. 2021 г., в 18:50, Вячеслав Коптилин  
> написал(а):
> 
> Hello Nikolay,
> 
> If I am not mistaken, the method you mentioned, allows to create a "user"
> cache that is available through public api for the user. This does not
> cover the case when the plugin developer wants to hide such cache and
> protect it form the end user (at least, via public api).
> 
> Thanks,
> S.
> 
> пт, 3 дек. 2021 г., 14:15 Nikolay Izhikov :
> 
>> Vyacheslav, Val, can you please clarify - What is the issue if third-party
>> plugins will create «ignite-sys-cache» from the code?
>> 
>> Like just replacing `Ignite#cache` with the `Ignite#getOrCreateCache`.
>> 
>>> 2 дек. 2021 г., в 16:13, Вячеслав Коптилин 
>> написал(а):
>>> 
>>> Hello Maxim,
>>> 
>>>> I don't understand why you are using *backwards compatibility* for
>>> completely internal things.
>>>> Why you are thinking in terms of users usage when are talking about
>>> ignite-sys-cache but not thinking when refactoring
>>> First of all, we are talking about all plugin developers. It does not
>>> matter it is open-source or proprietary plugins.
>>> Honestly, I don't have a list of all possible plugins that have already
>>> been developed and available under the ASF license, for instance.
>>> Do you have such a list? Can you be sure that this change will not affect
>>> anyone?
>>> 
>>>> I don't understand why you are using *backwards compatibility* for
>>> completely internal things.
>>>> Why you are thinking in terms of users usage when are talking about
>>> ignite-sys-cache but not thinking when refactoring
>>> The system cache was the crucial thing in the architecture of Apache
>> Ignite
>>> 1.x and 2.x (at least 2.1 - 2.11?)
>>> 
>>>> All the internals have been reworked and nobody even mentioned that it
>>> may affect some of the plugin developers.
>>> Yep, perhaps, some internal parts of Apache Ignite were reworked in order
>>> to avoid using the system cache.
>>> However, it is still a part of Ignite and it works and can be used in
>>> plugins.
>>> Honestly, the mentioned alternative, I mean the distributed metastorage,
>> is
>>> the INTERNAL thing as well.
>>> It means that plugin developer depends on INTERNAL entities. (it does not
>>> matter system-cache/metastorage)
>>> IMHO, such questions should be CAREFULLY discussed with no rush.
>>> 
>>>> I do not propose to rush with the ignite-sys-cache removal. I'd like to
>>> collect all the technical stuff that we depend on, fix all of them and
>>> after everything will be ready do a final removal.
>>> Good! We are on the same page!
>>> 
>>>> 1. Add deprecation for the 2.12 release. I hope it is still possible.
>>> If I am not mistaken, the code freeze was October 29. I think it is too
>>> late to introduce this change. We can add deprecation for the 2.13
>> release.
>>> 
>>>> Apache Ignite core still have some dependencies on ignite-sys-cache that
>>> should fix for 2.13
>>> Ok, I agree. We need to try to find out all edge cases and add new
>>> abilities to the metastorage in order to cover all known
>>> issues/requirements.
>>> 
>>>> Remove the system cache in 2.14.
>>> It depends on our progress with improving the distributed metastorage. In
>>> general, I hope it will be possible.
>>> 
>>> Thanks,
>>> S.
>>> 
>>> чт, 2 дек. 2021 г. в 13:36, Maxim Muzafarov :
>>> 
>>>> Pavel,
>>>> 
>>>> 
>>>> I don't understand why you are using *backwards compatibility* for
>>>> completely internal things. Why you are thinking in terms of users
>>>> usage when are talking about ignite-sys-cache but not thinking when
>>>> refactoring, for instance, all the checkpoint classes? Take a look at
>>>> the [1] issue. All the internals have been reworked and nobody even
>>>> mentioned that it may affect some of the plugin developers. This is
>>>> really strange, so I can't agree with you.
>>>> 
>>>> 
>>>> I do not propose to rush with the ignite-sys-cache removal. I'd like
>>>> to collect all the technical stuff that we depend on, fix all of them
>>>> and after everything will 

Re: [DISCUSSION] Remove outdated ignite-sys-cache

[Nikolay Izhikov]

Valentin

>  plugin framework allows to implement internal components and use the 
> internal API.

Can you please, point out to documentation or place in Ignite code that 
describe this behaviour?
AFAIK plugin can only use public API, internal API can be changed any time we 
want.

> Folks, can anyone explain the rush? 

No rush from my side.

Just want to understand your vision of integration points between Ignite and 
plugins.

> Is there any specific reason for it?

Intention of IEP-80 is to improve Ignite codebase by removing abandoned 
features.

> But the fact is that there are users that can depend on the system cache via 
> the plugin framework.

Why this dependency can’t be changed to any regular cache?
Just replace `Ignite#cache` to `Ignite#getOrCreateCache` and that’s it.

Do we have some special guarantees or logic besides system cache?


> 4 дек. 2021 г., в 02:14, Valentin Kulichenko  
> написал(а):
> 
> Nikolay, that's right, plugin framework allows to implement internal
> components and use the internal API. There is a difference between a plugin
> and an extension that uses only public API
> 
> Folks, can anyone explain the rush? Is there any specific reason for it?
> 
> I think we all agree that this is a good change - system cache has to go. I
> guess technically it's not even a breaking change, because system cache is
> not a public API feature. But the fact is that there are users that can
> depend on the system cache via the plugin framework. Let's be respectful to
> those users, provide a reasonable documented alternative and give time.
> 
> -Val
> 
> On Fri, Dec 3, 2021 at 8:18 AM Nikolay Izhikov  wrote:
> 
>> I don’t understand how it possible.
>> 
>> Are you talking about plugin that uses some kind of internal API?
>> 
>>> 3 дек. 2021 г., в 18:50, Вячеслав Коптилин 
>> написал(а):
>>> 
>>> Hello Nikolay,
>>> 
>>> If I am not mistaken, the method you mentioned, allows to create a "user"
>>> cache that is available through public api for the user. This does not
>>> cover the case when the plugin developer wants to hide such cache and
>>> protect it form the end user (at least, via public api).
>>> 
>>> Thanks,
>>> S.
>>> 
>>> пт, 3 дек. 2021 г., 14:15 Nikolay Izhikov :
>>> 
>>>> Vyacheslav, Val, can you please clarify - What is the issue if
>> third-party
>>>> plugins will create «ignite-sys-cache» from the code?
>>>> 
>>>> Like just replacing `Ignite#cache` with the `Ignite#getOrCreateCache`.
>>>> 
>>>>> 2 дек. 2021 г., в 16:13, Вячеслав Коптилин 
>>>> написал(а):
>>>>> 
>>>>> Hello Maxim,
>>>>> 
>>>>>> I don't understand why you are using *backwards compatibility* for
>>>>> completely internal things.
>>>>>> Why you are thinking in terms of users usage when are talking about
>>>>> ignite-sys-cache but not thinking when refactoring
>>>>> First of all, we are talking about all plugin developers. It does not
>>>>> matter it is open-source or proprietary plugins.
>>>>> Honestly, I don't have a list of all possible plugins that have already
>>>>> been developed and available under the ASF license, for instance.
>>>>> Do you have such a list? Can you be sure that this change will not
>> affect
>>>>> anyone?
>>>>> 
>>>>>> I don't understand why you are using *backwards compatibility* for
>>>>> completely internal things.
>>>>>> Why you are thinking in terms of users usage when are talking about
>>>>> ignite-sys-cache but not thinking when refactoring
>>>>> The system cache was the crucial thing in the architecture of Apache
>>>> Ignite
>>>>> 1.x and 2.x (at least 2.1 - 2.11?)
>>>>> 
>>>>>> All the internals have been reworked and nobody even mentioned that it
>>>>> may affect some of the plugin developers.
>>>>> Yep, perhaps, some internal parts of Apache Ignite were reworked in
>> order
>>>>> to avoid using the system cache.
>>>>> However, it is still a part of Ignite and it works and can be used in
>>>>> plugins.
>>>>> Honestly, the mentioned alternative, I mean the distributed
>> metastorage,
>>>> is
>>>>> the INTERNAL thing as well.
>>>>> It means that plugin developer depends on INTERNAL entities. (it does
>> not
>&g

Re: [DISCUSSION] Remove outdated ignite-sys-cache

[Nikolay Izhikov]

Slava, I don’t get it.

> Plugins have access to different internal Ignite components, such as security 
> processor and others, and can extend the programmatic API of Ignite.

Where docs say that we, as a community, take responsibility to keep internals 
in the way plugin expect?

> 6 дек. 2021 г., в 14:48, Вячеслав Коптилин  
> написал(а):
> 
> Hello Nikolay,
> 
>>> plugin framework allows to implement internal components and use the
> internal API.
>> Can you please, point out to documentation or place in Ignite code that
> describe this behaviour?
> 
> Looks like it states here https://ignite.apache.org/docs/latest/plugins
> 
>> The Ignite plugin system allows you to extend the core functionality of
>> Ignite. Plugins have access to different internal Ignite components, such
>> as security processor and others, and can extend the programmatic API of
>> Ignite.
> 
> 
> Thanks,
> S.
> 
> 
> сб, 4 дек. 2021 г. в 14:54, Nikolay Izhikov :
> 
>> Valentin
>> 
>>> plugin framework allows to implement internal components and use the
>> internal API.
>> 
>> Can you please, point out to documentation or place in Ignite code that
>> describe this behaviour?
>> AFAIK plugin can only use public API, internal API can be changed any time
>> we want.
>> 
>>> Folks, can anyone explain the rush?
>> 
>> No rush from my side.
>> 
>> Just want to understand your vision of integration points between Ignite
>> and plugins.
>> 
>>> Is there any specific reason for it?
>> 
>> Intention of IEP-80 is to improve Ignite codebase by removing abandoned
>> features.
>> 
>>> But the fact is that there are users that can depend on the system cache
>> via the plugin framework.
>> 
>> Why this dependency can’t be changed to any regular cache?
>> Just replace `Ignite#cache` to `Ignite#getOrCreateCache` and that’s it.
>> 
>> Do we have some special guarantees or logic besides system cache?
>> 
>> 
>>> 4 дек. 2021 г., в 02:14, Valentin Kulichenko <
>> valentin.kuliche...@gmail.com> написал(а):
>>> 
>>> Nikolay, that's right, plugin framework allows to implement internal
>>> components and use the internal API. There is a difference between a
>> plugin
>>> and an extension that uses only public API
>>> 
>>> Folks, can anyone explain the rush? Is there any specific reason for it?
>>> 
>>> I think we all agree that this is a good change - system cache has to
>> go. I
>>> guess technically it's not even a breaking change, because system cache
>> is
>>> not a public API feature. But the fact is that there are users that can
>>> depend on the system cache via the plugin framework. Let's be respectful
>> to
>>> those users, provide a reasonable documented alternative and give time.
>>> 
>>> -Val
>>> 
>>> On Fri, Dec 3, 2021 at 8:18 AM Nikolay Izhikov 
>> wrote:
>>> 
>>>> I don’t understand how it possible.
>>>> 
>>>> Are you talking about plugin that uses some kind of internal API?
>>>> 
>>>>> 3 дек. 2021 г., в 18:50, Вячеслав Коптилин 
>>>> написал(а):
>>>>> 
>>>>> Hello Nikolay,
>>>>> 
>>>>> If I am not mistaken, the method you mentioned, allows to create a
>> "user"
>>>>> cache that is available through public api for the user. This does not
>>>>> cover the case when the plugin developer wants to hide such cache and
>>>>> protect it form the end user (at least, via public api).
>>>>> 
>>>>> Thanks,
>>>>> S.
>>>>> 
>>>>> пт, 3 дек. 2021 г., 14:15 Nikolay Izhikov :
>>>>> 
>>>>>> Vyacheslav, Val, can you please clarify - What is the issue if
>>>> third-party
>>>>>> plugins will create «ignite-sys-cache» from the code?
>>>>>> 
>>>>>> Like just replacing `Ignite#cache` with the `Ignite#getOrCreateCache`.
>>>>>> 
>>>>>>> 2 дек. 2021 г., в 16:13, Вячеслав Коптилин >> 
>>>>>> написал(а):
>>>>>>> 
>>>>>>> Hello Maxim,
>>>>>>> 
>>>>>>>> I don't understand why you are using *backwards compatibility* for
>>>>>>> completely internal things.
>>>>>>>> Why you are thinking in terms of users us

Re: [DISCUSSION] Remove outdated ignite-sys-cache

[Nikolay Izhikov]

Valentin,

> However, changes like system cache removal are much more critical, because a 
> plugin might rely on it. 

I’m still don’t understand - what is the difference between system cache and 
any other Ignite cache except the name?
Do we have some special guarantees for system cache?

> Any objections to the plan I've suggested earlier?
> 1. Write down the differences between the system cache and the metastorage, 
> and provide a transition guide for plugin developers.

Don’t think we need any transition guide.

> I don't think it's reasonable to do this earlier than mid next year

This date are questionable, also.

Other points of your plan makes sense.
+1 to follow it.

> 6 дек. 2021 г., в 20:16, Valentin Kulichenko  
> написал(а):
> 
> That's correct.
> 
> Folks, can we agree on how we want to approach the removal of the system
> cache? Any objections to the plan I've suggested earlier? As a reminder,
> here it is:
> 1. Write down the differences between the system cache and the metastorage,
> and provide a transition guide for plugin developers.
> 2. Deprecate the system cache in 2.13.
> 3. Remove the system cache in one of the further releases. I don't think
> it's reasonable to do this earlier than mid next year (even that is
> potentially too early).
> 
> As for general compatibility requirements related to plugins, I think they
> obviously should not be as strict as for the public API. It's
> understandable that a method signature can be updated, or another similar
> change can be made in internal APIs. However, changes like system cache
> removal are much more critical, because a plugin might rely on it. It's our
> responsibility as a good friendly community to provide a reasonable
> alternative and a reasonable timeline for such a case.
> 
> -Val
> 
> On Mon, Dec 6, 2021 at 8:56 AM Вячеслав Коптилин 
> wrote:
> 
>> Hi,
>> 
>>> Plugins have access to different internal Ignite components, such as
>> security processor and others, and can extend the programmatic API of
>> Ignite.
>> 
>>> Where docs say that we, as a community, take responsibility to keep
>> internals in the way plugin expect?
>> Nikolay, it seems to me, that quoted text clearly states about that.
>> Plugin has access to internal APIs and so it depends on it. If we want to
>> be a friendly community then we should discuss such changes
>> (removing/changing internal APIs) and provide a reasonable time in order to
>> update such dependencies.
>> 
>> I think no one in this topic is against removing sys-cache. The question
>> is: is it suitable for the community to deprecate using system cache in
>> 2.13 and removing it in 2.14 || 2.15?
>> Am I missing something? Am I correct?
>> 
>> Thanks,
>> Slava.
>> 
>> 
>> пн, 6 дек. 2021 г. в 15:00, Nikolay Izhikov :
>> 
>>> Slava, I don’t get it.
>>> 
>>>> Plugins have access to different internal Ignite components, such as
>>> security processor and others, and can extend the programmatic API of
>>> Ignite.
>>> 
>>> Where docs say that we, as a community, take responsibility to keep
>>> internals in the way plugin expect?
>>> 
>>>> 6 дек. 2021 г., в 14:48, Вячеслав Коптилин 
>>> написал(а):
>>>> 
>>>> Hello Nikolay,
>>>> 
>>>>>> plugin framework allows to implement internal components and use the
>>>> internal API.
>>>>> Can you please, point out to documentation or place in Ignite code
>> that
>>>> describe this behaviour?
>>>> 
>>>> Looks like it states here
>> https://ignite.apache.org/docs/latest/plugins
>>>> 
>>>>> The Ignite plugin system allows you to extend the core functionality
>> of
>>>>> Ignite. Plugins have access to different internal Ignite components,
>>> such
>>>>> as security processor and others, and can extend the programmatic API
>> of
>>>>> Ignite.
>>>> 
>>>> 
>>>> Thanks,
>>>> S.
>>>> 
>>>> 
>>>> сб, 4 дек. 2021 г. в 14:54, Nikolay Izhikov :
>>>> 
>>>>> Valentin
>>>>> 
>>>>>> plugin framework allows to implement internal components and use the
>>>>> internal API.
>>>>> 
>>>>> Can you please, point out to documentation or place in Ignite code
>> that
>>>>> describe this behaviour?
>>>>> AFAIK plugin can only use public

Re: Apache Ignite 2.12 RELEASE [Time, Scope, Manager]

[Nikolay Izhikov]

Hello.

Let’s include [1] in 2.12 scope.

After migrations authentication processor to security API we have an issue.
Persistent data region should exists on client node if authentication is 
enabled.

It seems very annoying for the users.

[1] https://issues.apache.org/jira/browse/IGNITE-15969

> 2 дек. 2021 г., в 19:50, Nikita Amelchev  написал(а):
> 
> I would like to formally announce code freeze for 2.12.0. Only
> blockers are accepted to be included to the scope. [1]
> 
> We have one blocker issue [2] that will be fixed soon.
> 
> [1] 
> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P3.Stabilization
> [2] https://issues.apache.org/jira/browse/IGNITE-15966
> 
> чт, 2 дек. 2021 г. в 16:01, Nikita Amelchev :
>> 
>> Hi, Igniters.
>> 
>> The release is blocked by the auth issue [1] discussed above. The
>> patch will be prepared at the nearest time.
>> 
>> I want to include the useful issue that adds the ability to configure
>> snapshot thread pool size if nobody minds.
>> 
>> [1] https://issues.apache.org/jira/browse/IGNITE-15966
>> [2] https://issues.apache.org/jira/browse/IGNITE-16014
>> 
>> пт, 26 нояб. 2021 г. в 11:44, Nikita Amelchev :
>>> 
>>> Hello,
>>> 
>>> I want to include the issue [1] to the 2.12 scope. It fixes some
>>> metrics according to the JSR107.
>>> 
>>> Any objections?
>>> 
>>> [1] https://issues.apache.org/jira/browse/IGNITE-16002 The remove
>>> cache method should update statistics if the method returns true
>>> 
>>> чт, 25 нояб. 2021 г. в 17:12, Nikita Amelchev :
>>>> 
>>>> Hello Ivan,
>>>> 
>>>> +1, the issue can be cherry-picked to the 2.12.
>>>> 
>>>> чт, 25 нояб. 2021 г. в 17:07, Ivan Bessonov :
>>>>> 
>>>>> Hello everyone,
>>>>> 
>>>>> is there a chance to add this issue to the release scope? [1]
>>>>> Currently, defragmentation of certain types of indexes can corrupt them.
>>>>> Fix is straightforward and should not cause any problems.
>>>>> Thank you!
>>>>> 
>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-15968
>>>>> 
>>>>> вт, 23 нояб. 2021 г. в 20:44, Nikita Amelchev :
>>>>> 
>>>>>> Ivan,
>>>>>> 
>>>>>> I have cherry-picked the issue:
>>>>>> https://issues.apache.org/jira/browse/IGNITE-15767
>>>>>> 
>>>>>> вт, 23 нояб. 2021 г. в 20:31, Nikolay Izhikov :
>>>>>>> 
>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-15951
>>>>>>> 
>>>>>>> Cherry-picked to 2.12
>>>>>>> 
>>>>>>>> 23 нояб. 2021 г., в 16:38, Nikita Amelchev 
>>>>>> написал(а):
>>>>>>>> 
>>>>>>>> Hi, +1 to cherry-pick.
>>>>>>>> 
>>>>>>>> вт, 23 нояб. 2021 г. в 15:25, Ivan Daschinsky :
>>>>>>>>> 
>>>>>>>>> Hi! Lets add one simple bug fix, but very useful
>>>>>>>>> 
>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-15767
>>>>>>>>> 
>>>>>>>>> пн, 22 нояб. 2021 г. в 15:22, Pavel Tupitsyn :
>>>>>>>>> 
>>>>>>>>>> Yes, let's fix those auth issues, there are so many complaints on
>>>>>> the user
>>>>>>>>>> list.
>>>>>>>>>> 
>>>>>>>>>> On Mon, Nov 22, 2021 at 1:03 PM Mikhail Petrov <
>>>>>> pmgheap@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Igniters,
>>>>>>>>>>> it seems that issues [1] and [2] that were discovered recently are
>>>>>>>>>>> blockers for the 2.12 release.
>>>>>>>>>>> 
>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-15951
>>>>>>>>>>> [2] https://issues.apache.org/jira/browse/IGNITE-15966
>>>>>>>>>>> 
>>>>>>>>>>> On 22.11.2021 11:04, Nikolay Izhikov wrote:
>>>>>>>>>>>> Hello.
>>>>>>>>>&g

Re: [DISCUSSION] Reject join of nodes with different character encodings

[Nikolay Izhikov]

Hello, Ivan.

UTF-8 can’t encode all UNICODE characters.

> 13 дек. 2021 г., в 12:49, Ivan Daschinsky  написал(а):
> 
> Khm, maybe a better variant is  to enforce all strings to be encoded in
> UTF-8?
> AFAIK multi OS cluster is a quite common case.
> 
> 
> пн, 13 дек. 2021 г. в 11:36, Mikhail Petrov :
> 
>> Igniters,
>> 
>> Recently we faced the problem that if the cluster consists of nodes
>> running in the JVM with different encodings, many issues arise.
>> The root cause of the mentioned issues is components that use
>> `String#getBytes()` and `new String()`, which relies on the
>> system default encoding. Thus, if a string is deserialized on a node
>> with a different encoding from the one that serialized it, the
>> deserialized string can be different from the original one.
>> 
>> For example:
>> 
>> Serialization/deserialization of string in communication messages may be
>> broken for some strings on nodes running in a JVM with a different
>> encoding as DirectByteBufferStreamImplV2 uses String#getBytes() to
>> serialize strings - [1]
>> 
>> Or the IgniteAuthenticationProcessor can compute different security IDs
>> for the user on different nodes in this case - [2]
>> 
>> What do you think, if we solve this problem globally, by rejecting to
>> join nodes that run on JVMs with different encodings?
>> 
>> As a result, we will be sure that all cluster nodes have the same
>> encoding and all related problems will be solved.
>> 
>> [1] - https://issues.apache.org/jira/browse/IGNITE-16106
>> [2] - https://issues.apache.org/jira/browse/IGNITE-16068
>> 
>> --
>> Mikhail
>> 
>> 
> 
> -- 
> Sincerely yours, Ivan Daschinskiy

Re: [DISCUSSION] Reject join of nodes with different character encodings

[Nikolay Izhikov]

> Does Java String support all unicode characters and particularly does it 
> support more characters than UTF-8

It’s not about Java, it’s about UTF-8 standard.

Please, take a look at [1] 

> In November 2003, UTF-8 was restricted by RFC 3629 to match the constraints 
> of the UTF-16 character encoding: explicitly prohibiting code points 
> corresponding to the high and low surrogate characters removed more than 3% 
> of the three-byte sequences, and ending at U+10 removed more than 48% of 
> the four-byte sequences and all five- and six-byte sequences.

And [2] 

> The definition of UTF-8 prohibits encoding character numbers between U+D800 
> and U+DFFF, which are reserved for use with the UTF-16 encoding form (as 
> surrogate pairs) and do not directly represent characters.

Actually, we already has some modes to support this restriction of UTF-8.
Please, take a look at BinaryUtils#utf8BytesToStr [3]


[1] https://en.wikipedia.org/wiki/UTF-8
[2] https://datatracker.ietf.org/doc/html/rfc3629
[3] 
https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/binary/BinaryUtils.java#L2387

> 13 дек. 2021 г., в 13:57, Ivan Pavlukhin  написал(а):
> 
>> UTF-8 can’t encode all UNICODE characters.
> 
> Nikolay, could you please elaborate? My understanding is that encoding
> we speak about matters for conversion from byte arrays to strings.
> Does Java String support all unicode characters and particularly does
> it support more characters than UTF-8 (I am not saying here that java
> String uses UTF-8)?
> 
> 2021-12-13 12:56 GMT+03:00, Ivan Daschinsky :
>> UTF-8 is already a default encoding in our BinaryObject format. So.... I am
>> for unification.
>> 
>> пн, 13 дек. 2021 г. в 12:50, Nikolay Izhikov :
>> 
>>> Hello, Ivan.
>>> 
>>> UTF-8 can’t encode all UNICODE characters.
>>> 
>>>> 13 дек. 2021 г., в 12:49, Ivan Daschinsky 
>>> написал(а):
>>>> 
>>>> Khm, maybe a better variant is  to enforce all strings to be encoded in
>>>> UTF-8?
>>>> AFAIK multi OS cluster is a quite common case.
>>>> 
>>>> 
>>>> пн, 13 дек. 2021 г. в 11:36, Mikhail Petrov :
>>>> 
>>>>> Igniters,
>>>>> 
>>>>> Recently we faced the problem that if the cluster consists of nodes
>>>>> running in the JVM with different encodings, many issues arise.
>>>>> The root cause of the mentioned issues is components that use
>>>>> `String#getBytes()` and `new String()`, which relies on
>>>>> the
>>>>> system default encoding. Thus, if a string is deserialized on a node
>>>>> with a different encoding from the one that serialized it, the
>>>>> deserialized string can be different from the original one.
>>>>> 
>>>>> For example:
>>>>> 
>>>>> Serialization/deserialization of string in communication messages may
>>>>> be
>>>>> broken for some strings on nodes running in a JVM with a different
>>>>> encoding as DirectByteBufferStreamImplV2 uses String#getBytes() to
>>>>> serialize strings - [1]
>>>>> 
>>>>> Or the IgniteAuthenticationProcessor can compute different security
>>>>> IDs
>>>>> for the user on different nodes in this case - [2]
>>>>> 
>>>>> What do you think, if we solve this problem globally, by rejecting to
>>>>> join nodes that run on JVMs with different encodings?
>>>>> 
>>>>> As a result, we will be sure that all cluster nodes have the same
>>>>> encoding and all related problems will be solved.
>>>>> 
>>>>> [1] - https://issues.apache.org/jira/browse/IGNITE-16106
>>>>> [2] - https://issues.apache.org/jira/browse/IGNITE-16068
>>>>> 
>>>>> --
>>>>> Mikhail
>>>>> 
>>>>> 
>>>> 
>>>> --
>>>> Sincerely yours, Ivan Daschinskiy
>>> 
>>> 
>> 
>> --
>> Sincerely yours, Ivan Daschinskiy
>> 
> 
> 
> -- 
> 
> Best regards,
> Ivan Pavlukhin

Re: Apache Ignite 2.12 RELEASE [Time, Scope, Manager]

[Nikolay Izhikov]

Nikita.

I don’t think IGNITE-16068 is a release blocker.

1. It only happens when authentication enabled.
2. There is at lease 2 workarounds:
a. Use the same file.encoding for all Ignite nodes.
b. Use only latin characters in user login.

I propose to postpone this ticket and move on with the release.

> 13 дек. 2021 г., в 15:28, Nikita Amelchev  написал(а):
> 
> Hello.
> 
> I have cherry-picked the blocked [1] to the 2.12. It updates the log4j
> version to 2.15.
> 
> The release is blocked by one blocker issue with auth.
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-16101
> [2] https://issues.apache.org/jira/browse/IGNITE-16068
> 
> чт, 9 дек. 2021 г. в 12:44, Nikita Amelchev :
>> 
>> Hello, Nikolay.
>> 
>> I have cherry-picked the issue fix.
>> 
>> Thank you.
>> 
>> ср, 8 дек. 2021 г. в 18:47, Nikolay Izhikov :
>>> 
>>> Hello.
>>> 
>>> Let’s include [1] in 2.12 scope.
>>> 
>>> After migrations authentication processor to security API we have an issue.
>>> Persistent data region should exists on client node if authentication is 
>>> enabled.
>>> 
>>> It seems very annoying for the users.
>>> 
>>> [1] https://issues.apache.org/jira/browse/IGNITE-15969
>>> 
>>>> 2 дек. 2021 г., в 19:50, Nikita Amelchev  написал(а):
>>>> 
>>>> I would like to formally announce code freeze for 2.12.0. Only
>>>> blockers are accepted to be included to the scope. [1]
>>>> 
>>>> We have one blocker issue [2] that will be fixed soon.
>>>> 
>>>> [1] 
>>>> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P3.Stabilization
>>>> [2] https://issues.apache.org/jira/browse/IGNITE-15966
>>>> 
>>>> чт, 2 дек. 2021 г. в 16:01, Nikita Amelchev :
>>>>> 
>>>>> Hi, Igniters.
>>>>> 
>>>>> The release is blocked by the auth issue [1] discussed above. The
>>>>> patch will be prepared at the nearest time.
>>>>> 
>>>>> I want to include the useful issue that adds the ability to configure
>>>>> snapshot thread pool size if nobody minds.
>>>>> 
>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-15966
>>>>> [2] https://issues.apache.org/jira/browse/IGNITE-16014
>>>>> 
>>>>> пт, 26 нояб. 2021 г. в 11:44, Nikita Amelchev :
>>>>>> 
>>>>>> Hello,
>>>>>> 
>>>>>> I want to include the issue [1] to the 2.12 scope. It fixes some
>>>>>> metrics according to the JSR107.
>>>>>> 
>>>>>> Any objections?
>>>>>> 
>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-16002 The remove
>>>>>> cache method should update statistics if the method returns true
>>>>>> 
>>>>>> чт, 25 нояб. 2021 г. в 17:12, Nikita Amelchev :
>>>>>>> 
>>>>>>> Hello Ivan,
>>>>>>> 
>>>>>>> +1, the issue can be cherry-picked to the 2.12.
>>>>>>> 
>>>>>>> чт, 25 нояб. 2021 г. в 17:07, Ivan Bessonov :
>>>>>>>> 
>>>>>>>> Hello everyone,
>>>>>>>> 
>>>>>>>> is there a chance to add this issue to the release scope? [1]
>>>>>>>> Currently, defragmentation of certain types of indexes can corrupt 
>>>>>>>> them.
>>>>>>>> Fix is straightforward and should not cause any problems.
>>>>>>>> Thank you!
>>>>>>>> 
>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-15968
>>>>>>>> 
>>>>>>>> вт, 23 нояб. 2021 г. в 20:44, Nikita Amelchev :
>>>>>>>> 
>>>>>>>>> Ivan,
>>>>>>>>> 
>>>>>>>>> I have cherry-picked the issue:
>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-15767
>>>>>>>>> 
>>>>>>>>> вт, 23 нояб. 2021 г. в 20:31, Nikolay Izhikov :
>>>>>>>>>> 
>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-15951
>>>>>>>>>> 
>>>>>>>>>> Cherry-picked to 2.12
>>>>>>

Re: [VOTE] Release Apache Ignite 2.11.1 RC2

[Nikolay Izhikov]

+1 (binding)

> 20 дек. 2021 г., в 16:20, Ivan Daschinsky  написал(а):
> 
> +1 from me
> Checked ODBC drivers (32-bit and 64-bit) installers on Windows with running
> locally Ignite with
> pyodbc and python 3.9 using this script [1]
> 
> 
> [1] -- https://gist.github.com/ivandasch/6cc0b56e055b826e5840b5c04fa3b2fa
> 
> пн, 20 дек. 2021 г. в 15:45, Pavel Tupitsyn :
> 
>> +1
>> 
>> Checked .NET binaries, nugets, and examples.
>> 
>> On Mon, Dec 20, 2021 at 3:42 PM Ilya Kasnacheev >> 
>> wrote:
>> 
>>> Hello!
>>> 
>>> +1
>>> 
>>> There seems to be a correct version of log4j2 now.
>>> 
>>> Btw, I've noticed that we ship log2j 1.x with ignite-rest-http. This is
>>> quite bad.
>>> 
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>> 
>>> 
>>> пн, 20 дек. 2021 г. в 15:35, Maxim Muzafarov :
>>> 
 The second release candidate for the 2.11.1 version is ready.
 
 Since this is an emergency release the vote will remain open for as
 short an amount as time as required to vet the release. All votes are
 welcome and we encourage everyone to test the release, but only PMC
 votes are “officially” counted. As always, at least 3 +1 votes and
 more positive than negative votes are required. See voting guidelines:
 https://www.apache.org/foundation/voting.html
 
 +1 - to accept Apache Ignite 2.11.1-rc2
 0 - don't care either way
 -1 - DO NOT accept Apache Ignite Ignite 2.11.1-rc2 (explain why)
 
 
 * RELEASE CANDADATE *
 
 Check the release difference between 2.11 and 2.11.1:
 https://github.com/apache/ignite/compare/ignite-2.11...ignite-2.11.1
 
 I have uploaded a release candidate to:
 https://dist.apache.org/repos/dist/dev/ignite/2.11.1-rc2/
 https://dist.apache.org/repos/dist/dev/ignite/packages_2.11.1-rc2/
 
 The following staging can be used for testing:
 
>> https://repository.apache.org/content/repositories/orgapacheignite-1535/
 
 
>>> 
>> https://www.myget.org/feed/apache-ignite-staging/package/nuget/Apache.Ignite
 
 The tag name is 2.11.1-rc2:
 
 
>>> 
>> https://gitbox.apache.org/repos/asf?p=ignite.git;a=commit;h=eae1147dd1cd625cc229098489be8d208f6cabcc
 
 RELEASE_NOTES:
 
 
>>> 
>> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=RELEASE_NOTES.txt;hb=ignite-2.11.1
 
 Complete list of resolved issues:
 
 
 
>>> 
>> https://issues.apache.org/jira/issues/?jql=(project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.11.1%27))
 
 
 * CHECKS *
 
 Additional checks have been performed (available for users included
>> into
 the release group on TeamCity).
 
 TC [Check RC: Licenses, compile, chksum]
 
 
>>> 
>> https://ci.ignite.apache.org/viewLog.html?buildId=6333527&buildTypeId=ApacheIgniteReleaseJava8_PrepareVote3BuildNuGetPackages
 
 See notes on how to verify release here
 https://www.apache.org/info/verification.html
 and
 
 
>>> 
>> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P5.VotingonReleaseandReleaseVerification
 
>>> 
>> 
> 
> 
> -- 
> Sincerely yours, Ivan Daschinskiy

Re: [VOTE] Release Apache Ignite 2.12.0 RC2

[Nikolay Izhikov]

+1

> 11 янв. 2022 г., в 02:38, Vladimir Steshin  написал(а):
> 
> +1
> 
> 10.01.2022 15:52, Nikita Amelchev пишет:
>> Dear Community,
>> 
>> The release candidate (2.12.0-rc2) is ready.
>> 
>> I have uploaded a release candidate to:
>> https://dist.apache.org/repos/dist/dev/ignite/2.12.0-rc2/
>> https://dist.apache.org/repos/dist/dev/ignite/packages_2.12.0-rc2/
>> 
>> The following staging can be used for testing:
>> https://repository.apache.org/content/repositories/orgapacheignite-1539
>> 
>> Tag name is 2.12.0-rc2:
>> https://gitbox.apache.org/repos/asf?p=ignite.git;a=commit;h=refs/tags/2.12.0-rc2
>> 
>> RELEASE_NOTES:
>> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=RELEASE_NOTES.txt;hb=ignite-2.12
>> 
>> Complete list of resolved issues:
>> https://issues.apache.org/jira/issues/?jql=(project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.12%27))%20and%20status%20in%20(%27CLOSED%27%2C%20%27RESOLVED%27)%20ORDER%20BY%20priority
>> 
>> DEVNOTES:
>> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=DEVNOTES.txt;hb=ignite-2.12
>> 
>> Additional checks have been performed (available for users included
>> into the release group on TeamCity).
>> 
>> TC [Check RC: Licenses, compile, chksum]
>> https://ci2.ignite.apache.org/buildConfiguration/ignite2_Release_ApacheIgniteReleaseJava8_PrepareVote4CheckRcLicensesChecksum/6266150?showRootCauses=false&expandBuildChangesSection=true
>> 
>> TC [2] Compare w/ Previous Release
>> https://ci2.ignite.apache.org/buildConfiguration/ignite2_Release_ApacheIgniteReleaseJava8_IgniteRelease72CheckFileConsistency/6266148?showRootCauses=false&expandBuildChangesSection=true
>> 
>> The vote is formal, see voting guidelines
>> https://www.apache.org/foundation/voting.html
>> 
>> +1 - to accept Apache Ignite 2.12.0-rc2
>> 0 - don't care either way
>> -1 - DO NOT accept Apache Ignite Ignite 2.12.0-rc2 (explain why)
>> 
>> See notes on how to verify release here
>> https://www.apache.org/info/verification.html
>> and
>> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P5.VotingonReleaseandReleaseVerification
>> 
>> This vote will be open until Thu Jan 13, 2022, 16:00 UTC. Please,
>> write me down the thread if you need additional time to check the
>> release.
>> https://www.timeanddate.com/countdown/vote?iso=20220113T16&p0=0&msg=VOTE+on+the+Apache+Ignite+Release+2.12.0+RC2&font=sanserif
>> 

Re: Adding a system view of recently completed compute tasks

[Nikolay Izhikov]

+1 to have compute history.

@ткаленко кирилл Lets’add both - compute task and compute job history?

> 12 янв. 2022 г., в 16:32, Zhenya Stanilovsky  
> написал(а):
> 
> 
> Ok, thanks, now it`s clear, seems we need additional documentation here and 
> also property renaming.
> 
> 
>  
>> Judging by the code, this is the task session timeout. 
>  
>  
>  
>  

Re: join the community

[Nikolay Izhikov]

Hello, Sergey.

Welcome to the Ignite community.
I’ve added you to the contributor list.

> 18 янв. 2022 г., в 14:22, Sergey Korotkov  
> написал(а):
> 
> Hello, All
> 
> I’d like to join the community and contribute to the Ignite.
> 
> Would you please grant me the contributors permissions in Jira. My account
> is serge.korotkov
> 
> Thanks,
> --
>  Sergey

Re: java 17 support

[Nikolay Izhikov]

>  will definitely not be moved to JDK more than 8th.

I think is a matter to discuss :)
Why we should stay on JDK8 forever?


> 2 февр. 2022 г., в 15:23, Petr Ivanov  написал(а):
> 
> Hi, Davide!
> 
> 
> Ignite 3 will support at least 11th JDK but that can be changed (for the 
> better) on initial release.
> 
> Ignite 2 is subject to discuss, it currently runs on 11th and there should be 
> a patch that added JDK 17 runtime support, but codebase (compile) will 
> definitely not be moved to JDK more than 8th.
> 
>> On 28 Jan 2022, at 18:28, Davide Imbriaco  wrote:
>> 
>> hi,
>> 
>> I've been using ignite in a project based on java 17 (which is the latest
>> LTS java release). I see that ignite requires java 8 or 11... with java 17
>> it works fine so far (just a couple minor issues easily resolved), but I
>> was wondering if there is a roadmap for official java 17 support, just to
>> be sure. Maybe this is scheduled for Ignite 3?
>> 
>> thank you, bye,
>> D
> 

Re: [ANNOUNCE] Welcome Pavel Pereslegin as a new committer

[Nikolay Izhikov]

Great news! Congratulations!

> 10 февр. 2022 г., в 18:26, Maksim Timonin  
> написал(а):
> 
> Hi Pavel,
> 
> Congratulations!
> 
> On Thu, Feb 10, 2022 at 6:16 PM Maxim Muzafarov  wrote:
> 
>> The Project Management Committee (PMC) for Apache Ignite has invited
>> Pavel Pereslegin to become a committer and we are pleased to announce that
>> he has accepted.
>> 
>> He made a lot of major contributions to the Apache Ignite codebase
>> like a snapshot restore procedure, batch update operation to the
>> PageMemory, TDE cache key rotation procedure, Service context
>> injection and etc.
>> 
>> Being a committer enables easier contribution to the project since there is
>> no need to go via the patch submission process. This should enable better
>> productivity.
>> 
>> Please join me in welcoming Pavel, and congratulating him on the new role
>> in
>> the Apache Ignite Community.
>> 
>> 
>> Best Regards,
>> Maxim Muzafarov
>> 

Re: [DISCUSSION] Apache Ignite Extension Release Process Improvements

[Nikolay Izhikov]

Hello, Maxim.

Thanks for investigation!

> 4. log4j-test.xml, tests.properties these files from the ignite-core module 
> required to run tests and it seems they should be shared between projects 
> also as tests resources.

As you may see most copies of these files are empty.
I think we should modify `GridAbstractTest` and other classes to use defaults 
if `log4j-test.xml` or `tests.properties` unavailable.

> 4. Configure Travis [10] to deploy a new snapshot version to the apache maven 
> snapshots repository [5] each time the Ignite master branch is updated.

Can we attach Ignite sources as maven modules to Ignite extensions?
And use Ignite sources as dependencies?

> 18 февр. 2022 г., в 00:30, Maxim Muzafarov  написал(а):
> 
> Hello Igniters,
> 
> 
> We have a lot of Ignite extensions waiting to be released (e.g. aws,
> gce, topology-validator, ignite-spring-tx-ext etc.). I think it is a
> good point in time to make some kind of automation and simplify our
> release process for Ignite extension modules removing manual steps
> from it.
> 
> 
> During the investigation I've found some issues that we have:
> 
> 1. Ignite extensions have their own parent pom file [2]. The code and
> dependency versions are fully duplicated with the Ignite parent pom
> [1], however, these properties must be shared for Ignite Extension
> project. Take a look at the :
> - master Apache Ignite: 9.4.39.v20210325
> - master Apache Ignite Extensions: 9.4.11.v20180605
> 
> 2. The checkstyle configuration between these two repositories are
> different [3] [4]. Each of Apache Ignite and Apache Ignite Extensions
> projects have their own checkstyle config file, however, it must be
> shared the same as the maven checkstyle profile.
> 
> 3. The Extensions have a dependency on the latest Apache Ignite
> version in the master branch (e.g. 2.13-SNAPSHOT), but without
> deploying it in the local repository the Extension projects can be
> built. It seems the latest version should be published to the maven
> snapshots repository [5].
> 
> 4. log4j-test.xml, tests.properties these files from the ignite-core
> module required to run tests and it seems they should be shared
> between projects also as tests resources.
> 
> 5. The apache-release profile [7] from the apache parent which will
> prepare the sources, checksums, gpg sign is not used in the release
> process at all.
> 
> 
> Here are my suggestions to fix all the issues mentioned above:
> 
> 1. Create the `ignite-parent` module that will be released each time
> with the latest release version. This module will be based on the
> parent/pom.xml [1]. This will allow us to share all common
> configurations, profiles, dependency versions and properties to the
> Ignite Extensions.
> 
> 2. Create the `ignite-plugin-bom` bill of materials [8] module with
> the common Ignite dependencies required for developing a new
> extension. This is a common practice for companies that have their own
> parent pom file and can't inherit from Ignite's one (e.g.
> spring-boot-bom example [9] with all spring dependencies).
> 
> 3. Create the additional `ignite-resources` jar file to share
> checkstyle configuration, resources, tests resources and etc. for the
> Extension project.
> 
> 4. Configure Travis [10] to deploy a new snapshot version to the
> apache maven snapshots repository [5] each time the Ignite master
> branch is updated.
> 
> 5. Clean up the pom.xml Extensions file from profiles and dependencies
> that are not required to release extensions e.g. the release profile
> [11].
> 
> 6. Reuse the `apache-release` profile for preparing sources of the
> extensions during the release.
> 
> 
> WDYT?
> 
> 
> [1] https://github.com/apache/ignite/blob/master/parent/pom.xml#L35
> [2] https://github.com/apache/ignite-extensions/blob/master/parent/pom.xml#L36
> [3] 
> https://github.com/apache/ignite-extensions/blob/master/checkstyle/checkstyle.xml
> [4] https://github.com/apache/ignite/blob/master/checkstyle/checkstyle.xml
> [5] https://repository.apache.org/content/repositories/snapshots
> [6] 
> https://github.com/apache/ignite-extensions/tree/master/modules/aws-ext/modules/core/src/test/config
> [7] https://github.com/apache/maven-apache-parent/blob/apache-24/pom.xml#L375
> [8] 
> https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#bill-of-materials-bom-poms
> [9] https://github.com/snowdrop/spring-boot-bom/blob/sb-2.5.x/pom.xml#L31
> [10] 
> https://blog.travis-ci.com/2017-03-30-deploy-maven-travis-ci-packagecloud/
> [11] https://github.com/apache/ignite-extensions/blob/master/pom.xml#L242

Re: [DISCUSSION] Exclude ignite-log4j, log4j 1.2.17

[Nikolay Izhikov]

Hello, Anton.

+1 to remove outdated logging library.

But, seems, we can’t do it right now, because of existing deployments.
Let’s mark this module as deprecated and remove it in 2.14?


> Not every deployment require to be secured.

Disagree.
We should update or workaround known security issues ASAP.


> Not every deployment requires to use of log4j.



Agree, but we shouldn’t provide or support modules with known security issues.


> 28 февр. 2022 г., в 18:41, Anton Vinogradov  написал(а):
> 
> Your deployment has vulnerabilities only in case you configured log4j as a
> logger.
> Not every deployment require to be secured.
> Not every deployment requires to use of log4j.
> 
> We must change the default logging library if the current is log4j and
> provide the ability to use log4j as before (where it is required) but with
> a warning, I think.
> 
> On Mon, Feb 28, 2022 at 3:55 PM Sergei Ryzhov  wrote:
> 
>> Hello, Igniters.
>> 
>> log4j 1.2.17 is not supported and contains critical vulnerabilities
>> I suggest excluding log4j 1.2.17 and module ignite-log4j from ignite[1].
>> 
>> Direct vulnerabilities:
>> https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-23305
>> https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-23302
>> https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-4104
>> https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17571
>> 
>> WDYT?
>> 
>> [1] https://issues.apache.org/jira/browse/IGNITE-16626
>> 
>> --
>> Best regards,
>> Sergei Ryzhov
>> 

[DISCUSSION] Removal of ignitevisorcmd

[Nikolay Izhikov]

Hello, Igniters.

There are legacy modules - visor-console, visor-plugins, visor-console-2.10
Looks like that modules are not  evolving by community and barely used by
the Ignite users.

I propose to remove it completely.
There are IEP [1] for this

It seems that removal of visor modules allow to us to remove another legacy
code from Ignite codebase:

* support of daemon nodes.
* GridClient and related classes.

What do you think?

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217392419

Re: [DISCUSSION] Removal of ignitevisorcmd

[Nikolay Izhikov]

PR is ready for review - https://github.com/apache/ignite/pull/10411

чт, 1 дек. 2022 г. в 16:49, Taras Ledkov :

> Hi,
>
> +1 for remove Visor related code.
> Unfortunately we have to migrate `control-utility` to use IgniteClient
> (thin client) before drop GridClient. I guess we will do it in the future.
>
> Also, the old thin JDBC based on the GridClient (classes placed at the
> org.apache.ignite.internal.jdbc.*) must be removed.
>

Re: [DISCUSSION] Removal of ignitevisorcmd

[Nikolay Izhikov]

Hello, Slava.

> However, do we have a list of useful features that do not exist in our
control utility?

I don't have the list on hand.
Do you have some feature in mind that must be transferred in control.sh?

I don't find any mentions of visorcmd on user list and in user chats so my
guess is - this utility is not used by the users in any way.
Do you have some other inputs?

If yes, don't hesitate to create a ticket with the IEP-94 label.


чт, 1 дек. 2022 г. в 19:11, Вячеслав Коптилин :

> Hi Igniters,
>
> I fully support the idea to stop supporting Visor and removing its
> implementation.
> However, do we have a list of useful features that do not exist in our
> control utility?
> Perhaps, it makes sense to reimplement such functionality and provide it
> based on control.sh script.
>
> Thanks,
> Slava.
>
> чт, 1 дек. 2022 г. в 15:51, Nikolay Izhikov :
>
> > PR is ready for review - https://github.com/apache/ignite/pull/10411
> >
> > чт, 1 дек. 2022 г. в 16:49, Taras Ledkov :
> >
> > > Hi,
> > >
> > > +1 for remove Visor related code.
> > > Unfortunately we have to migrate `control-utility` to use IgniteClient
> > > (thin client) before drop GridClient. I guess we will do it in the
> > future.
> > >
> > > Also, the old thin JDBC based on the GridClient (classes placed at the
> > > org.apache.ignite.internal.jdbc.*) must be removed.
> > >
> >
>

Re: [DISCUSSION] Removal of ignitevisorcmd

[Nikolay Izhikov]

THanks, Slava, Stephen.

Ticket to reimplement cache scan created -
https://issues.apache.org/jira/browse/IGNITE-18320

пт, 2 дек. 2022 г. в 13:30, Stephen Darlington <
stephen.darling...@gridgain.com>:

> One option that’s in Visor but not control.sh is the ability to view cache
> content (cache -scan). I do hear of people using that.
>
> > On 2 Dec 2022, at 09:17, Andrey Mashenkov 
> wrote:
> >
> > Hi Igniters,
> >
> > I agree with Slava, we have to provide alternative ways for users before
> > dropping a feature that just worked.
> > A user may use some feature for a long time without any issues, so, this
> is
> > why there is no mention in chats/mails.
> > I mean here, the feature is not only the whole visor command, but a
> command
> > option that just makes UX better.
> >
> > I think we can revise visor features, which are not supported by
> > control.sh, and create an umbrella ticket for supporting in control.sh.
> > Set this ticket either as 'required' for a 'dropping visor ticket', or
> as a
> > blocker for the next release if visor dropping will come first.
> >
> > On Fri, Dec 2, 2022 at 11:09 AM Nikolay Izhikov 
> wrote:
> >
> >> Hello, Slava.
> >>
> >>> However, do we have a list of useful features that do not exist in our
> >> control utility?
> >>
> >> I don't have the list on hand.
> >> Do you have some feature in mind that must be transferred in control.sh?
> >>
> >> I don't find any mentions of visorcmd on user list and in user chats so
> my
> >> guess is - this utility is not used by the users in any way.
> >> Do you have some other inputs?
> >>
> >> If yes, don't hesitate to create a ticket with the IEP-94 label.
> >>
> >>
> >> чт, 1 дек. 2022 г. в 19:11, Вячеслав Коптилин  >:
> >>
> >>> Hi Igniters,
> >>>
> >>> I fully support the idea to stop supporting Visor and removing its
> >>> implementation.
> >>> However, do we have a list of useful features that do not exist in our
> >>> control utility?
> >>> Perhaps, it makes sense to reimplement such functionality and provide
> it
> >>> based on control.sh script.
> >>>
> >>> Thanks,
> >>> Slava.
> >>>
> >>> чт, 1 дек. 2022 г. в 15:51, Nikolay Izhikov :
> >>>
> >>>> PR is ready for review - https://github.com/apache/ignite/pull/10411
> >>>>
> >>>> чт, 1 дек. 2022 г. в 16:49, Taras Ledkov :
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> +1 for remove Visor related code.
> >>>>> Unfortunately we have to migrate `control-utility` to use
> >> IgniteClient
> >>>>> (thin client) before drop GridClient. I guess we will do it in the
> >>>> future.
> >>>>>
> >>>>> Also, the old thin JDBC based on the GridClient (classes placed at
> >> the
> >>>>> org.apache.ignite.internal.jdbc.*) must be removed.
> >>>>>
> >>>>
> >>>
> >>
> >
> >
> > --
> > Best regards,
> > Andrey V. Mashenkov
>
>

Re: [DISCUSSION] Removal of ignitevisorcmd

[Nikolay Izhikov]

Thanks.

Ticket created - https://issues.apache.org/jira/browse/IGNITE-18327

пт, 2 дек. 2022 г. в 15:25, 18624049226 <18624049...@163.com>:

> Visor has a 'cache -clear' function, this function is very useful,
> especially in some O&M scenarios.
>
> Because Ignite does not support the TRUNCATE command, it is necessary to
> provide a high-performance clear table data command.
>
> 在 2022/12/2 19:13, Nikolay Izhikov 写道:
> > THanks, Slava, Stephen.
> >
> > Ticket to reimplement cache scan created -
> > https://issues.apache.org/jira/browse/IGNITE-18320
> >
> > пт, 2 дек. 2022 г. в 13:30, Stephen Darlington <
> > stephen.darling...@gridgain.com>:
> >
> >> One option that’s in Visor but not control.sh is the ability to view
> cache
> >> content (cache -scan). I do hear of people using that.
> >>
> >>> On 2 Dec 2022, at 09:17, Andrey Mashenkov
> >> wrote:
> >>> Hi Igniters,
> >>>
> >>> I agree with Slava, we have to provide alternative ways for users
> before
> >>> dropping a feature that just worked.
> >>> A user may use some feature for a long time without any issues, so,
> this
> >> is
> >>> why there is no mention in chats/mails.
> >>> I mean here, the feature is not only the whole visor command, but a
> >> command
> >>> option that just makes UX better.
> >>>
> >>> I think we can revise visor features, which are not supported by
> >>> control.sh, and create an umbrella ticket for supporting in control.sh.
> >>> Set this ticket either as 'required' for a 'dropping visor ticket', or
> >> as a
> >>> blocker for the next release if visor dropping will come first.
> >>>
> >>> On Fri, Dec 2, 2022 at 11:09 AM Nikolay Izhikov
> >> wrote:
> >>>> Hello, Slava.
> >>>>
> >>>>> However, do we have a list of useful features that do not exist in
> our
> >>>> control utility?
> >>>>
> >>>> I don't have the list on hand.
> >>>> Do you have some feature in mind that must be transferred in
> control.sh?
> >>>>
> >>>> I don't find any mentions of visorcmd on user list and in user chats
> so
> >> my
> >>>> guess is - this utility is not used by the users in any way.
> >>>> Do you have some other inputs?
> >>>>
> >>>> If yes, don't hesitate to create a ticket with the IEP-94 label.
> >>>>
> >>>>
> >>>> чт, 1 дек. 2022 г. в 19:11, Вячеслав Коптилин <
> slava.kopti...@gmail.com
> >>> :
> >>>>> Hi Igniters,
> >>>>>
> >>>>> I fully support the idea to stop supporting Visor and removing its
> >>>>> implementation.
> >>>>> However, do we have a list of useful features that do not exist in
> our
> >>>>> control utility?
> >>>>> Perhaps, it makes sense to reimplement such functionality and provide
> >> it
> >>>>> based on control.sh script.
> >>>>>
> >>>>> Thanks,
> >>>>> Slava.
> >>>>>
> >>>>> чт, 1 дек. 2022 г. в 15:51, Nikolay Izhikov:
> >>>>>
> >>>>>> PR is ready for review -https://github.com/apache/ignite/pull/10411
> >>>>>>
> >>>>>> чт, 1 дек. 2022 г. в 16:49, Taras Ledkov:
> >>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> +1 for remove Visor related code.
> >>>>>>> Unfortunately we have to migrate `control-utility` to use
> >>>> IgniteClient
> >>>>>>> (thin client) before drop GridClient. I guess we will do it in the
> >>>>>> future.
> >>>>>>> Also, the old thin JDBC based on the GridClient (classes placed at
> >>>> the
> >>>>>>> org.apache.ignite.internal.jdbc.*) must be removed.
> >>>>>>>
> >>>
> >>> --
> >>> Best regards,
> >>> Andrey V. Mashenkov
> >>

Re: Replacing default work dir from tmp to current dir

[Nikolay Izhikov]

> > > > > > > > > > > > > I agree that user home maybe not the best place
> > 
> > from
> > > > > > Linux
> > > > > > > > > > > > > perspective
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > philosophy, but  "user.home"/ignite/work  is
> > 
> > more or
> > > > > > less
> > > > > > > > > > > portable.
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > For the Linux environment, we can add a
> > 
> > suggestion
> > > > > about
> > > > > > > > > > where to
> > > > > > > > > > > > > place
> > > > > > > > > > > > > > > > persisted data. For very first testing of Apache
> > > > > 
> > > > > Ignite
> > > > > > > user
> > > > > > > > > > home
> > > > > > > > > > > > > still
> > > > > > > > > > > > > > > > looks good for me.
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > Sincerely,
> > > > > > > > > > > > > > > > Dmitriy Pavlov
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > вт, 27 авг. 2019 г. в 11:56, Pavel Pereslegin <
> > > > > > > > > > 
> > > > > > > > > > xxt...@gmail.com
> > > > > > > > > > > > :
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > Or instead of a WARNING, we can add a
> > 
> > suggestion
> > > > > with
> > > > > > a
> > > > > > > > > > > > > > recommendation
> > > > > > > > > > > > > > > > > for the production environment.
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > вт, 27 авг. 2019 г. в 11:41, Petr Ivanov <
> > > > > > > > > > 
> > > > > > > > > > mr.wei...@gmail.com
> > > > > > > > > > > > :
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > /opt is either does not exist on fresh
> > 
> > system, or
> > > > > > has
> > > > > > > > the
> > > > > > > > > > > same
> > > > > > > > > > > > > > > > > restriction: no user access without admin
> > > > > > 
> > > > > > intervention.
> > > > > > > > > > > > > > > > > > /usr/local, /var/lib, etc. — all this is
> > > > > 
> > > > > implemented
> > > > > > > in
> > > > > > > > > > our
> > > > > > > > > > > > DEB /
> > > > > > > > > > > > > > RPM
> > > > > > > > > > > > > > > > > packages already.
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > For ZIP installation %HOME% seems to be the
> > 
> > best
> > > > > > > > approach
> > > > > > > > > > for
> > > > > > > > > > > > > > > "2-click"
> > > > > > > > > > > > > > > > > launch.
> > > > > > > > > > > > > > > > > > Later user can update preferences and set
> > 
> > working
> > > > > > dir
> > > > > > > to
> > > > > > > > > > > > whatever
> > > > > > > > > >

Re: Replacing default work dir from tmp to current dir

[Nikolay Izhikov]

Alexey.

> Ignite is widely used in embedded scenarios; the ability to process data
> in-process locally is very powerful and I see no reason why we should
> remove it

I don't propose to remove something.
I wrote about "same distribution".

Seems, we should have different defaults and even distributions for different 
usage scenarios.

> As an example,
> both SQLite, and rocksdb are distributed as a library, so I see no issues
> in Ignite server side being a library.

As far as I know, both rocksdb and SQLite is local only libraries and don't 
have any distrubted features.


>"current directory" as persistence directory is not consistent.

I agree with you.
How it happens, that after wide discussion we implemented, reviewed and merged 
wrong defaults?

As I know, we have explicit release only to change this default.

This release is broken, isn't it?

В Чт, 03/10/2019 в 14:03 +0300, Alexey Goncharuk пишет:
> Nikolay,
> 
> Ignite is widely used in embedded scenarios; the ability to process data
> in-process locally is very powerful and I see no reason why we should
> remove it. I absolutely agree with Pavel T. on the subject. As an example,
> both SQLite, and rocksdb are distributed as a library, so I see no issues
> in Ignite server side being a library.
> 
> As long as Ignite is available as a maven library, we should provide a
> consistent node behavior; "current directory" as persistence directory is
> not consistent.
> 
> чт, 3 окт. 2019 г. в 13:52, Nikolay Izhikov :
> 
> > Pavel.
> > 
> > > As a user, why would I want to define a system-wide property just to use
> > > some library?
> > 
> > Why do you think Ignite is a library?
> > May be the root of usability issues in using same distribution for a
> > library and server side dbms?
> > 
> > 
> > В Чт, 03/10/2019 в 13:40 +0300, Pavel Tupitsyn пишет:
> > > Ivan, which vendors place files into current work dir, can you please
> > 
> > give
> > > an example?
> > > 
> > > > Generally IGNITE_HOME should be defined
> > > 
> > > This is an inconvenience for the users, bad usability.
> > > As a user, why would I want to define a system-wide property just to use
> > > some library?
> > > 
> > > > As for .NET. Should not we define IGNITE_HOME for it?
> > > 
> > > No, for the reasons stated above.
> > > 
> > > I'd like everyone to pay more attention to Maven/NuGet distribution
> > > scenario. Forget about zip archive for a while.
> > > As a user, I add a dependency to Ignite package and call
> > 
> > Ignition.start().
> > > That's all, it should work right away, no env vars, no additional
> > > configuration.
> > > And current work dir should not matter, because different tools, IDEs and
> > > workflows dictate different work directories.
> > > 
> > > > user.home can be not writable as well
> > > 
> > > In this case users won't even be able to use Maven or NuGet, let's not
> > > consider those rare scenarios.
> > > 
> > > 
> > > To summarize: home directory is the way to go as a default location.
> > > 
> > > On Thu, Oct 3, 2019 at 12:14 PM Ivan Pavlukhin 
> > 
> > wrote:
> > > 
> > > > As for .NET. Should not we define IGNITE_HOME for it?
> > > > 
> > > > чт, 3 окт. 2019 г. в 12:13, Ivan Pavlukhin :
> > > > > 
> > > > > Folks,
> > > > > 
> > > > > I am with Ilya here. I remind that we are talking not about general
> > > > > case for Ignite usage. Generally IGNITE_HOME should be defined.
> > > > > Otherwise we fallback to a default, and user.dir usually points to a
> > > > > directory where java launcher command was called (work dir).
> > > > > 
> > > > > user.home seems to cause more surprises to me:
> > > > > * user.home can be undefined for JVM;
> > > > > * user.home can be not writable as well (e.g. some special service
> > 
> > user).
> > > > > 
> > > > > And as far as know other vendors usually place files required for an
> > > > > application in current work dir.
> > > > > 
> > > > > чт, 3 окт. 2019 г. в 01:45, Denis Magda :
> > > > > > 
> > > > > > I was always expecting this to be a user *home* directory that can
> > 
> > be
> > > > > > resolved in any operating system and will work for any

Re: [DISCUSSION][IEP-35] Replace RunningQueryManager with GridSystemViewManager

[Nikolay Izhikov]

Hello, Ivan.

Thanks for feedback.

Initial IEP [1] naming was changed during code review.
I updated the IEP [1] with the current naming.

Can you take a look and check is all clear now?

[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392

В Ср, 02/10/2019 в 17:21 +0300, Ivan Pavlukhin пишет:
> Hi Nikolay,
> 
> Actually I do not fully understand what is SystemView API. I have not
> found it in IEP [1] (I searched for words "system" and "view").
> 
> RunningQueryManager is a component responsible for tracking running
> queries internally. This info is exposed to users as SQL view via
> SqlSystemViewRunningQueries. In the same package you can find a plenty
> of other SQL views.
> 
> [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> 
> вт, 1 окт. 2019 г. в 06:42, Nikolay Izhikov :
> > 
> > Hello, Igniters.
> > 
> > Since the last release `RunningQueryManager` [1] was added.
> > It used to track a running query.
> > 
> > In IEP-35 [2] SystemView API was added.
> > SystemView API supposed to be used to track all kinds of internal Ignite 
> > objects.
> > 
> > I think this RunningQueryManager should be replaced [3] with the more 
> > unified SystemView API.
> > 
> > Any objections?
> > 
> > [1] https://issues.apache.org/jira/browse/IGNITE-10754
> > [2] 
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> > [3] https://issues.apache.org/jira/browse/IGNITE-12223
> > [4] https://issues.apache.org/jira/browse/IGNITE-12224
> 
> 
> 


signature.asc
Description: This is a digitally signed message part

Re: [DISCUSSION][IEP-35] Replace RunningQueryManager with GridSystemViewManager

[Nikolay Izhikov]

Hello, Ivan.

> 1. How system views are going to be exposed? Is there any difference
> in comparison to other metrics?

We have a `SystemViewExporterSpi`.
Built-in implementations are `JmxSystemViewExporterSpi` and 
`SqlViewExporterSpi`.

> 2. What should be done to adopt RunningQueryManager to SystemView API?

I think we should replace `RunningQueryManager` with the special SystemView 
implementation.

> what is the difference between metrics and system views?

Actually, it's a very good question :)

System view is a collection of internal Ignite objects exported to a user.
Each system view is a table.

Metric is a value representing some instantaneous state of the internal Ignite 
object.
So its a "cell" of table.

We need metrics to build charts and history of processes.
We need system views to known what objects exist in node and its params.

В Пт, 04/10/2019 в 11:51 +0300, Ivan Pavlukhin пишет:
> Nikolay,
> 
> I checked the IEP [1]. Now it is more clear for me about SystemView
> API. Follow-up questions:
> 1. How system views are going to be exposed? Is there any difference
> in comparison to other metrics?
> 2. What should be done to adopt RunningQueryManager to SystemView API?
> 
> Also some bits for my understanding. I do not have a clear intuition
> what is the difference between metrics and system views? For example,
> how a system view is different from a metric holding a collection of
> values? And why they were introduced as a separate class?
> 
> [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> 
> чт, 3 окт. 2019 г. в 16:37, Nikolay Izhikov :
> > 
> > Hello, Ivan.
> > 
> > Thanks for feedback.
> > 
> > Initial IEP [1] naming was changed during code review.
> > I updated the IEP [1] with the current naming.
> > 
> > Can you take a look and check is all clear now?
> > 
> > [1] 
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> > 
> > В Ср, 02/10/2019 в 17:21 +0300, Ivan Pavlukhin пишет:
> > > Hi Nikolay,
> > > 
> > > Actually I do not fully understand what is SystemView API. I have not
> > > found it in IEP [1] (I searched for words "system" and "view").
> > > 
> > > RunningQueryManager is a component responsible for tracking running
> > > queries internally. This info is exposed to users as SQL view via
> > > SqlSystemViewRunningQueries. In the same package you can find a plenty
> > > of other SQL views.
> > > 
> > > [1] 
> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> > > 
> > > вт, 1 окт. 2019 г. в 06:42, Nikolay Izhikov :
> > > > 
> > > > Hello, Igniters.
> > > > 
> > > > Since the last release `RunningQueryManager` [1] was added.
> > > > It used to track a running query.
> > > > 
> > > > In IEP-35 [2] SystemView API was added.
> > > > SystemView API supposed to be used to track all kinds of internal 
> > > > Ignite objects.
> > > > 
> > > > I think this RunningQueryManager should be replaced [3] with the more 
> > > > unified SystemView API.
> > > > 
> > > > Any objections?
> > > > 
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-10754
> > > > [2] 
> > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> > > > [3] https://issues.apache.org/jira/browse/IGNITE-12223
> > > > [4] https://issues.apache.org/jira/browse/IGNITE-12224
> > > 
> > > 
> > > 
> 
> 
> 


signature.asc
Description: This is a digitally signed message part

Re: Metric showing how many nodes may safely leave the cluster

[Nikolay Izhikov]

Ivan.

> We shouldn't force users to configure external tools and write extra code for 
> basic things.

Actually, I don't agree with you.
Having external monitoring system for any production cluster is a *basic* thing.

Can you, please, define "basic things"?

> single method for the whole cluster

Can you clarify, what do you mean, exactly?
We have a ticket[1] to support metrics output via visor.sh.

My understanding: we should have an easy way to output metric values for each 
node in cluster.

[1] https://issues.apache.org/jira/browse/IGNITE-12191


В Пт, 04/10/2019 в 17:09 +0300, Ivan Rakov пишет:
> Max,
> 
> What if user simply don't have configured monitoring system?
> Knowing whether cluster will survive node shutdown is critical for any 
> administrator that performs any manipulations with cluster topology.
> Essential information should be easily accessed. We shouldn't force 
> users to configure external tools and write extra code for basic things.
> 
> Alex,
> 
> Thanks, that's exact metric we need.
> My point is that we should make it more accessible: via control.sh 
> command and single method for the whole cluster.
> 
> Best Regards,
> Ivan Rakov
> 
> On 04.10.2019 16:34, Alex Plehanov wrote:
> > Ivan, there already exist metric
> > CacheGroupMetricsMXBean#getMinimumNumberOfPartitionCopies, which shows the
> > current redundancy level for the cache group.
> > We can lose up to ( getMinimumNumberOfPartitionCopies-1) nodes without data
> > loss in this cache group.
> > 
> > пт, 4 окт. 2019 г. в 16:17, Ivan Rakov :
> > 
> > > Igniters,
> > > 
> > > I've seen numerous requests to find out an easy way to check whether is
> > > it safe to turn off cluster node. As we know, in Ignite protection from
> > > sudden node shutdown is implemented through keeping several backup
> > > copies of each partition. However, this guarantee can be weakened for a
> > > while in case cluster has recently experienced node restart and
> > > rebalancing process is still in progress.
> > > Example scenario is restarting nodes one by one in order to update a
> > > local configuration parameter. User restarts one node and rebalancing
> > > starts: when it will be completed, it will be safe to proceed (backup
> > > count=1). However, there's no transparent way to determine whether
> > > rebalancing is over.
> > >   From my perspective, it would be very helpful to:
> > > 1) Add information about rebalancing and number of free-to-go nodes to
> > > ./control.sh --state command.
> > > Examples of output:
> > > 
> > > > Cluster  ID: 125a6dce-74b1-4ee7-a453-c58f23f1f8fc
> > > > Cluster tag: new_tag
> > > > 
> > > 
> > > 
> > > > Cluster is active
> > > > All partitions are up-to-date.
> > > > 3 node(s) can safely leave the cluster without partition loss.
> > > > Cluster  ID: 125a6dce-74b1-4ee7-a453-c58f23f1f8fc
> > > > Cluster tag: new_tag
> > > > 
> > > 
> > > 
> > > > Cluster is active
> > > > Rebalancing is in progress.
> > > > 1 node(s) can safely leave the cluster without partition loss.
> > > 
> > > 2) Provide the same information via ClusterMetrics. For example:
> > > ClusterMetrics#isRebalanceInProgress // boolean
> > > ClusterMetrics#getSafeToLeaveNodesCount // int
> > > 
> > > Here I need to mention that this information can be calculated from
> > > existing rebalance metrics (see CacheMetrics#*rebalance*). However, I
> > > still think that we need more simple and understandable flag whether
> > > cluster is in danger of data loss. Another point is that current metrics
> > > are bound to specific cache, which makes this information even harder to
> > > analyze.
> > > 
> > > Thoughts?
> > > 
> > > --
> > > Best Regards,
> > > Ivan Rakov
> > > 
> > > 


signature.asc
Description: This is a digitally signed message part