Re: [DISCUSSION] Reserve partitions for CacheQueries

[Ivan Pavlukhin]

> Then, if there are no objections, the short-term plan is:

Sounds ok for me.

> AFAIR Scan and SQL queries implementations are totally different. Could you 
> tell me how Index queries fit there?

I suppose my questions was misleading. Actually I would like to know
how code is organized today. AFAIR SQL and scan queries has their own
codebases (messages, merging results from different nodes and etc).
Does IndexQuery has separate codebase? Does it share some code with
ScanQuery on a "query" layer (higher than BTree layer)?

2021-12-10 13:25 GMT+03:00, Maksim Timonin :
> Hi!
>
>> Existing users may already have some logic in place to handle
> inconsistencies
>
> Pavel, I'm not aware of such users but your comment makes sense. So, I' OK
> with adding an option for ScanQuery. Naming of the option is debatable. The
> name "reservePartitions" looks good, but we actually already reserve a
> partition when it is explicitly specified in ScanQuery. Then, it can be a
> bit misleading in the case of explicitly setting this param to `false` with
> a specified partition. But we can just mention it in javadocs, that the
> setting affects only full scan.
>
>> AFAIR Scan and SQL queries implementations are totally different. Could
> you tell me how Index queries fit there?
>
> IndexQuery scans indexes like SQL does with few optimizations of traversing
> BPlusTree, also there are some differences in query processing, also
> IndexQuery provides sorted results by default. But I don't expect
> inconsistency in results for SQL / IndexQuery for similar queries.
> Actually, I should add tests for that and fix failures if any.
>
>> In an ideal world it would be great to have only one API
>
> I think it's possible to teach SQL to switch to cache queries for some
> cases, or provide such an opportunity with hints or explicit functions. And
> these queries might be parts of an SQL query plan. Or we can go even
> deeper, maybe it could be like Apache Spark stages, when we can build our
> plan with different types of queries, and the same type provides to
> users the opportunity to run only specific stages: DataFrame.sql() or
> DataFrame.filter(inline=true/false).
>
>> Perhaps IndexQuery should also cover regular entry iteration cases
>
> It's possible too. IndexQuery provides an opportunity to scan the PK index,
> then it can start ScanQuery under the hood for some cases.
>
> But anyway, to make them run through the single API, we should provide the
> same guarantees.
>
> Then, if there are no objections, the short-term plan is:
> 1. Implement partition reservation for IndexQuery.
> 2. Add the option for ScanQuery.
> 3. Make tests that show that IndexQuery / SQL / ScanQuery is replaceable
> for some types of queries in terms of result consistency.
> 4. Then discuss again, how we can integrate them together.
>
>
> On Fri, Dec 10, 2021 at 11:41 AM Ivan Pavlukhin 
> wrote:
>
>> Actually I brought a point about using SQL queries instead of scan
>> queries because I worry about inconsistency between different
>> implementations. AFAIR Scan and SQL queries implementations are
>> totally different. Could you tell me how Index queries fit there?
>>
>> My general ideas are as follows:
>> 1. In an ideal world it would be great to have only one API (to rule
>> them all) and implementation to cover all cases (i.e. scan, index,
>> SQL). Also, I wonder how other vendors tackle this?
>> 2. New IndexQuery might not mimic ScanQuery behavior but instead have
>> a correct/expected one (including partition reservation). Perhaps
>> IndexQuery should also cover regular entry iteration (scan) cases and
>> become a new ground for generalized scan/index queries.
>>
>> 2021-12-09 17:08 GMT+03:00, Pavel Tupitsyn :
>> > IndexQuery is experimental, so we can indeed make it reserve partitions
>> by
>> > default. No objections here.
>> >
>> > But with ScanQuery, I think it's better to add a "reservePartitions"
>> > property so that default behavior is not affected.
>> > Existing users may already have some logic in place to handle
>> > inconsistencies. So if we enable reservation by default,
>> > they'll get an unnecessary performance degradation.
>> >
>> > On Thu, Dec 9, 2021 at 4:42 PM Maksim Timonin 
>> > wrote:
>> >
>> >> Hi, Ivan, Pavel! Thanks for your responses.
>> >>
>> >> > But is there a real need/benefit from using scan queries over
>> primitive
>> >> SQL queries today
>> >>
>> >> In addition to Pavel's response, I'd like to add about IndexQuery.
>> >> This
>> >> query is in experimental status, but it shows great performance and
>> beats
>> >> SQL for *some* type of queries. For IndexQuery we can leverage on
>> >> understanding that we use only index and apply some optimizations:
>> >> skip
>> >> boundaries check (h2 Select.isConditionMet), apply filtering by inline
>> IO
>> >> instead of extracting fields from BinaryObjects, also we skip some H2
>> >> related stuff. Also it's not implemented yet, but for IndexQuery we
>> >> can
>> >> access data page

Re: [DISCUSSION] Reserve partitions for CacheQueries

[Maksim Timonin]

Hi, Ivan!

> Does IndexQuery has separate codebase? Does it share some code with
ScanQuery

Yes, IndexQuery mostly shares processing with ScanQuery:
requests/responses, a remote filter. Reducer (merge of query results) on an
initiator node has its own implementation for IndexQuery (MergeSort), but
it's built into existing ScanQuery processing.

On Mon, Dec 13, 2021 at 11:00 AM Ivan Pavlukhin  wrote:

> > Then, if there are no objections, the short-term plan is:
>
> Sounds ok for me.
>
> > AFAIR Scan and SQL queries implementations are totally different. Could
> you tell me how Index queries fit there?
>
> I suppose my questions was misleading. Actually I would like to know
> how code is organized today. AFAIR SQL and scan queries has their own
> codebases (messages, merging results from different nodes and etc).
> Does IndexQuery has separate codebase? Does it share some code with
> ScanQuery on a "query" layer (higher than BTree layer)?
>
> 2021-12-10 13:25 GMT+03:00, Maksim Timonin :
> > Hi!
> >
> >> Existing users may already have some logic in place to handle
> > inconsistencies
> >
> > Pavel, I'm not aware of such users but your comment makes sense. So, I'
> OK
> > with adding an option for ScanQuery. Naming of the option is debatable.
> The
> > name "reservePartitions" looks good, but we actually already reserve a
> > partition when it is explicitly specified in ScanQuery. Then, it can be a
> > bit misleading in the case of explicitly setting this param to `false`
> with
> > a specified partition. But we can just mention it in javadocs, that the
> > setting affects only full scan.
> >
> >> AFAIR Scan and SQL queries implementations are totally different. Could
> > you tell me how Index queries fit there?
> >
> > IndexQuery scans indexes like SQL does with few optimizations of
> traversing
> > BPlusTree, also there are some differences in query processing, also
> > IndexQuery provides sorted results by default. But I don't expect
> > inconsistency in results for SQL / IndexQuery for similar queries.
> > Actually, I should add tests for that and fix failures if any.
> >
> >> In an ideal world it would be great to have only one API
> >
> > I think it's possible to teach SQL to switch to cache queries for some
> > cases, or provide such an opportunity with hints or explicit functions.
> And
> > these queries might be parts of an SQL query plan. Or we can go even
> > deeper, maybe it could be like Apache Spark stages, when we can build our
> > plan with different types of queries, and the same type provides to
> > users the opportunity to run only specific stages: DataFrame.sql() or
> > DataFrame.filter(inline=true/false).
> >
> >> Perhaps IndexQuery should also cover regular entry iteration cases
> >
> > It's possible too. IndexQuery provides an opportunity to scan the PK
> index,
> > then it can start ScanQuery under the hood for some cases.
> >
> > But anyway, to make them run through the single API, we should provide
> the
> > same guarantees.
> >
> > Then, if there are no objections, the short-term plan is:
> > 1. Implement partition reservation for IndexQuery.
> > 2. Add the option for ScanQuery.
> > 3. Make tests that show that IndexQuery / SQL / ScanQuery is replaceable
> > for some types of queries in terms of result consistency.
> > 4. Then discuss again, how we can integrate them together.
> >
> >
> > On Fri, Dec 10, 2021 at 11:41 AM Ivan Pavlukhin 
> > wrote:
> >
> >> Actually I brought a point about using SQL queries instead of scan
> >> queries because I worry about inconsistency between different
> >> implementations. AFAIR Scan and SQL queries implementations are
> >> totally different. Could you tell me how Index queries fit there?
> >>
> >> My general ideas are as follows:
> >> 1. In an ideal world it would be great to have only one API (to rule
> >> them all) and implementation to cover all cases (i.e. scan, index,
> >> SQL). Also, I wonder how other vendors tackle this?
> >> 2. New IndexQuery might not mimic ScanQuery behavior but instead have
> >> a correct/expected one (including partition reservation). Perhaps
> >> IndexQuery should also cover regular entry iteration (scan) cases and
> >> become a new ground for generalized scan/index queries.
> >>
> >> 2021-12-09 17:08 GMT+03:00, Pavel Tupitsyn :
> >> > IndexQuery is experimental, so we can indeed make it reserve
> partitions
> >> by
> >> > default. No objections here.
> >> >
> >> > But with ScanQuery, I think it's better to add a "reservePartitions"
> >> > property so that default behavior is not affected.
> >> > Existing users may already have some logic in place to handle
> >> > inconsistencies. So if we enable reservation by default,
> >> > they'll get an unnecessary performance degradation.
> >> >
> >> > On Thu, Dec 9, 2021 at 4:42 PM Maksim Timonin <
> timoninma...@apache.org>
> >> > wrote:
> >> >
> >> >> Hi, Ivan, Pavel! Thanks for your responses.
> >> >>
> >> >> > But is there a real need/benefit from using scan queries ove

[DISCUSSION] Reject join of nodes with different character encodings

[Mikhail Petrov]

Igniters,

Recently we faced the problem that if the cluster consists of nodes 
running in the JVM with different encodings, many issues arise.
The root cause of the mentioned issues is components that use 
`String#getBytes()` and `new String()`, which relies on the 
system default encoding. Thus, if a string is deserialized on a node 
with a different encoding from the one that serialized it, the 
deserialized string can be different from the original one.


For example:

Serialization/deserialization of string in communication messages may be 
broken for some strings on nodes running in a JVM with a different 
encoding as DirectByteBufferStreamImplV2 uses String#getBytes() to 
serialize strings - [1]


Or the IgniteAuthenticationProcessor can compute different security IDs 
for the user on different nodes in this case - [2]


What do you think, if we solve this problem globally, by rejecting to 
join nodes that run on JVMs with different encodings?


As a result, we will be sure that all cluster nodes have the same 
encoding and all related problems will be solved.


[1] - https://issues.apache.org/jira/browse/IGNITE-16106
[2] - https://issues.apache.org/jira/browse/IGNITE-16068

--
Mikhail

Re: 0-day CVE in log4j

[Stephen Darlington]

Another workaround appears to be using the -Dlog4j2.formatMsgNoLookups=true 
option. Also, “Java versions greater than 6u211, 7u201, 8u191, and 11.0.1 are 
less affected by this attack vector, at least in theory, because the JNDI can't 
load remote code using LDAP.”

(https://arstechnica.com/information-technology/2021/12/minecraft-and-other-apps-face-serious-threat-from-new-code-execution-bug/)

> On 12 Dec 2021, at 10:56, Dmitriy Pavlov  wrote:
> 
> Hi Igniters,
> 
> Preliminary: change of the log4j version does not affect any tests
> (Alexander Nikolaev, correct me if I'm wrong).
> 
> If you're using embedded Ignite, it's perfectly possible to enforce jog4j2
> dependency to be 2.15.0 in your project final pom.xml or build.gradle or
> any other build system properties.
> 
> https://issues.apache.org/jira/browse/IGNITE-16101 ticket seems to be
> a blocker for 2.12. But for now, as a workaround, it's possible to select
> the latest version manually.
> 
> Sincerely,
> Dmitriy Pavlov
> 
> сб, 11 дек. 2021 г. в 09:47, Nikita Amelchev :
> 
>> Hello.
>> 
>> The issue to update dependency was created:
>> https://issues.apache.org/jira/browse/IGNITE-16101
>> 
>> I want to include it to the 2.12 scope.
>> 
>> сб, 11 дек. 2021 г., 09:19 Raymond Wilson :
>> 
>>> All
>>> 
>>> This blew up today: CVE-2021-44228 (
>>> 
>>> 
>> https://www.bleepingcomputer.com/news/security/new-zero-day-exploit-for-log4j-java-library-is-an-enterprise-nightmare/
>>> )
>>> 
>>> Will there be a risk assessment with respect to Ignite for this CVE?
>>> 
>>> Thanks,
>>> Raymond.
>>> 
>>> --
>>> 
>>> Raymond Wilson
>>> Trimble Distinguished Engineer, Civil Construction Software (CCS)
>>> 11 Birmingham Drive | Christchurch, New Zealand
>>> raymond_wil...@trimble.com
>>> 
>>> <
>>> 
>> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
 
>>> 
>> 

Re: [DISCUSSION] Reject join of nodes with different character encodings

[Ivan Daschinsky]

Khm, maybe a better variant is  to enforce all strings to be encoded in
UTF-8?
AFAIK multi OS cluster is a quite common case.


пн, 13 дек. 2021 г. в 11:36, Mikhail Petrov :

> Igniters,
>
> Recently we faced the problem that if the cluster consists of nodes
> running in the JVM with different encodings, many issues arise.
> The root cause of the mentioned issues is components that use
> `String#getBytes()` and `new String()`, which relies on the
> system default encoding. Thus, if a string is deserialized on a node
> with a different encoding from the one that serialized it, the
> deserialized string can be different from the original one.
>
> For example:
>
> Serialization/deserialization of string in communication messages may be
> broken for some strings on nodes running in a JVM with a different
> encoding as DirectByteBufferStreamImplV2 uses String#getBytes() to
> serialize strings - [1]
>
> Or the IgniteAuthenticationProcessor can compute different security IDs
> for the user on different nodes in this case - [2]
>
> What do you think, if we solve this problem globally, by rejecting to
> join nodes that run on JVMs with different encodings?
>
> As a result, we will be sure that all cluster nodes have the same
> encoding and all related problems will be solved.
>
> [1] - https://issues.apache.org/jira/browse/IGNITE-16106
> [2] - https://issues.apache.org/jira/browse/IGNITE-16068
>
> --
> Mikhail
>
>

-- 
Sincerely yours, Ivan Daschinskiy

Re: [DISCUSSION] Reject join of nodes with different character encodings

[Nikolay Izhikov]

Hello, Ivan.

UTF-8 can’t encode all UNICODE characters.

> 13 дек. 2021 г., в 12:49, Ivan Daschinsky  написал(а):
> 
> Khm, maybe a better variant is  to enforce all strings to be encoded in
> UTF-8?
> AFAIK multi OS cluster is a quite common case.
> 
> 
> пн, 13 дек. 2021 г. в 11:36, Mikhail Petrov :
> 
>> Igniters,
>> 
>> Recently we faced the problem that if the cluster consists of nodes
>> running in the JVM with different encodings, many issues arise.
>> The root cause of the mentioned issues is components that use
>> `String#getBytes()` and `new String()`, which relies on the
>> system default encoding. Thus, if a string is deserialized on a node
>> with a different encoding from the one that serialized it, the
>> deserialized string can be different from the original one.
>> 
>> For example:
>> 
>> Serialization/deserialization of string in communication messages may be
>> broken for some strings on nodes running in a JVM with a different
>> encoding as DirectByteBufferStreamImplV2 uses String#getBytes() to
>> serialize strings - [1]
>> 
>> Or the IgniteAuthenticationProcessor can compute different security IDs
>> for the user on different nodes in this case - [2]
>> 
>> What do you think, if we solve this problem globally, by rejecting to
>> join nodes that run on JVMs with different encodings?
>> 
>> As a result, we will be sure that all cluster nodes have the same
>> encoding and all related problems will be solved.
>> 
>> [1] - https://issues.apache.org/jira/browse/IGNITE-16106
>> [2] - https://issues.apache.org/jira/browse/IGNITE-16068
>> 
>> --
>> Mikhail
>> 
>> 
> 
> -- 
> Sincerely yours, Ivan Daschinskiy

Re: [DISCUSSION] Reject join of nodes with different character encodings

[Ivan Daschinsky]

UTF-8 is already a default encoding in our BinaryObject format. So I am
for unification.

пн, 13 дек. 2021 г. в 12:50, Nikolay Izhikov :

> Hello, Ivan.
>
> UTF-8 can’t encode all UNICODE characters.
>
> > 13 дек. 2021 г., в 12:49, Ivan Daschinsky 
> написал(а):
> >
> > Khm, maybe a better variant is  to enforce all strings to be encoded in
> > UTF-8?
> > AFAIK multi OS cluster is a quite common case.
> >
> >
> > пн, 13 дек. 2021 г. в 11:36, Mikhail Petrov :
> >
> >> Igniters,
> >>
> >> Recently we faced the problem that if the cluster consists of nodes
> >> running in the JVM with different encodings, many issues arise.
> >> The root cause of the mentioned issues is components that use
> >> `String#getBytes()` and `new String()`, which relies on the
> >> system default encoding. Thus, if a string is deserialized on a node
> >> with a different encoding from the one that serialized it, the
> >> deserialized string can be different from the original one.
> >>
> >> For example:
> >>
> >> Serialization/deserialization of string in communication messages may be
> >> broken for some strings on nodes running in a JVM with a different
> >> encoding as DirectByteBufferStreamImplV2 uses String#getBytes() to
> >> serialize strings - [1]
> >>
> >> Or the IgniteAuthenticationProcessor can compute different security IDs
> >> for the user on different nodes in this case - [2]
> >>
> >> What do you think, if we solve this problem globally, by rejecting to
> >> join nodes that run on JVMs with different encodings?
> >>
> >> As a result, we will be sure that all cluster nodes have the same
> >> encoding and all related problems will be solved.
> >>
> >> [1] - https://issues.apache.org/jira/browse/IGNITE-16106
> >> [2] - https://issues.apache.org/jira/browse/IGNITE-16068
> >>
> >> --
> >> Mikhail
> >>
> >>
> >
> > --
> > Sincerely yours, Ivan Daschinskiy
>
>

-- 
Sincerely yours, Ivan Daschinskiy

Re: [DISCUSSION] Reserve partitions for CacheQueries

[Ivan Pavlukhin]

Thanks for details!

2021-12-13 11:14 GMT+03:00, Maksim Timonin :
> Hi, Ivan!
>
>> Does IndexQuery has separate codebase? Does it share some code with
> ScanQuery
>
> Yes, IndexQuery mostly shares processing with ScanQuery:
> requests/responses, a remote filter. Reducer (merge of query results) on an
> initiator node has its own implementation for IndexQuery (MergeSort), but
> it's built into existing ScanQuery processing.
>
> On Mon, Dec 13, 2021 at 11:00 AM Ivan Pavlukhin 
> wrote:
>
>> > Then, if there are no objections, the short-term plan is:
>>
>> Sounds ok for me.
>>
>> > AFAIR Scan and SQL queries implementations are totally different. Could
>> you tell me how Index queries fit there?
>>
>> I suppose my questions was misleading. Actually I would like to know
>> how code is organized today. AFAIR SQL and scan queries has their own
>> codebases (messages, merging results from different nodes and etc).
>> Does IndexQuery has separate codebase? Does it share some code with
>> ScanQuery on a "query" layer (higher than BTree layer)?
>>
>> 2021-12-10 13:25 GMT+03:00, Maksim Timonin :
>> > Hi!
>> >
>> >> Existing users may already have some logic in place to handle
>> > inconsistencies
>> >
>> > Pavel, I'm not aware of such users but your comment makes sense. So, I'
>> OK
>> > with adding an option for ScanQuery. Naming of the option is debatable.
>> The
>> > name "reservePartitions" looks good, but we actually already reserve a
>> > partition when it is explicitly specified in ScanQuery. Then, it can be
>> > a
>> > bit misleading in the case of explicitly setting this param to `false`
>> with
>> > a specified partition. But we can just mention it in javadocs, that the
>> > setting affects only full scan.
>> >
>> >> AFAIR Scan and SQL queries implementations are totally different.
>> >> Could
>> > you tell me how Index queries fit there?
>> >
>> > IndexQuery scans indexes like SQL does with few optimizations of
>> traversing
>> > BPlusTree, also there are some differences in query processing, also
>> > IndexQuery provides sorted results by default. But I don't expect
>> > inconsistency in results for SQL / IndexQuery for similar queries.
>> > Actually, I should add tests for that and fix failures if any.
>> >
>> >> In an ideal world it would be great to have only one API
>> >
>> > I think it's possible to teach SQL to switch to cache queries for some
>> > cases, or provide such an opportunity with hints or explicit functions.
>> And
>> > these queries might be parts of an SQL query plan. Or we can go even
>> > deeper, maybe it could be like Apache Spark stages, when we can build
>> > our
>> > plan with different types of queries, and the same type provides to
>> > users the opportunity to run only specific stages: DataFrame.sql() or
>> > DataFrame.filter(inline=true/false).
>> >
>> >> Perhaps IndexQuery should also cover regular entry iteration cases
>> >
>> > It's possible too. IndexQuery provides an opportunity to scan the PK
>> index,
>> > then it can start ScanQuery under the hood for some cases.
>> >
>> > But anyway, to make them run through the single API, we should provide
>> the
>> > same guarantees.
>> >
>> > Then, if there are no objections, the short-term plan is:
>> > 1. Implement partition reservation for IndexQuery.
>> > 2. Add the option for ScanQuery.
>> > 3. Make tests that show that IndexQuery / SQL / ScanQuery is
>> > replaceable
>> > for some types of queries in terms of result consistency.
>> > 4. Then discuss again, how we can integrate them together.
>> >
>> >
>> > On Fri, Dec 10, 2021 at 11:41 AM Ivan Pavlukhin 
>> > wrote:
>> >
>> >> Actually I brought a point about using SQL queries instead of scan
>> >> queries because I worry about inconsistency between different
>> >> implementations. AFAIR Scan and SQL queries implementations are
>> >> totally different. Could you tell me how Index queries fit there?
>> >>
>> >> My general ideas are as follows:
>> >> 1. In an ideal world it would be great to have only one API (to rule
>> >> them all) and implementation to cover all cases (i.e. scan, index,
>> >> SQL). Also, I wonder how other vendors tackle this?
>> >> 2. New IndexQuery might not mimic ScanQuery behavior but instead have
>> >> a correct/expected one (including partition reservation). Perhaps
>> >> IndexQuery should also cover regular entry iteration (scan) cases and
>> >> become a new ground for generalized scan/index queries.
>> >>
>> >> 2021-12-09 17:08 GMT+03:00, Pavel Tupitsyn :
>> >> > IndexQuery is experimental, so we can indeed make it reserve
>> partitions
>> >> by
>> >> > default. No objections here.
>> >> >
>> >> > But with ScanQuery, I think it's better to add a "reservePartitions"
>> >> > property so that default behavior is not affected.
>> >> > Existing users may already have some logic in place to handle
>> >> > inconsistencies. So if we enable reservation by default,
>> >> > they'll get an unnecessary performance degradation.
>> >> >
>> >> > On Thu, Dec 9, 2021

Re: [DISCUSSION] Reject join of nodes with different character encodings

[Ivan Pavlukhin]

> UTF-8 can’t encode all UNICODE characters.

Nikolay, could you please elaborate? My understanding is that encoding
we speak about matters for conversion from byte arrays to strings.
Does Java String support all unicode characters and particularly does
it support more characters than UTF-8 (I am not saying here that java
String uses UTF-8)?

2021-12-13 12:56 GMT+03:00, Ivan Daschinsky :
> UTF-8 is already a default encoding in our BinaryObject format. So I am
> for unification.
>
> пн, 13 дек. 2021 г. в 12:50, Nikolay Izhikov :
>
>> Hello, Ivan.
>>
>> UTF-8 can’t encode all UNICODE characters.
>>
>> > 13 дек. 2021 г., в 12:49, Ivan Daschinsky 
>> написал(а):
>> >
>> > Khm, maybe a better variant is  to enforce all strings to be encoded in
>> > UTF-8?
>> > AFAIK multi OS cluster is a quite common case.
>> >
>> >
>> > пн, 13 дек. 2021 г. в 11:36, Mikhail Petrov :
>> >
>> >> Igniters,
>> >>
>> >> Recently we faced the problem that if the cluster consists of nodes
>> >> running in the JVM with different encodings, many issues arise.
>> >> The root cause of the mentioned issues is components that use
>> >> `String#getBytes()` and `new String()`, which relies on
>> >> the
>> >> system default encoding. Thus, if a string is deserialized on a node
>> >> with a different encoding from the one that serialized it, the
>> >> deserialized string can be different from the original one.
>> >>
>> >> For example:
>> >>
>> >> Serialization/deserialization of string in communication messages may
>> >> be
>> >> broken for some strings on nodes running in a JVM with a different
>> >> encoding as DirectByteBufferStreamImplV2 uses String#getBytes() to
>> >> serialize strings - [1]
>> >>
>> >> Or the IgniteAuthenticationProcessor can compute different security
>> >> IDs
>> >> for the user on different nodes in this case - [2]
>> >>
>> >> What do you think, if we solve this problem globally, by rejecting to
>> >> join nodes that run on JVMs with different encodings?
>> >>
>> >> As a result, we will be sure that all cluster nodes have the same
>> >> encoding and all related problems will be solved.
>> >>
>> >> [1] - https://issues.apache.org/jira/browse/IGNITE-16106
>> >> [2] - https://issues.apache.org/jira/browse/IGNITE-16068
>> >>
>> >> --
>> >> Mikhail
>> >>
>> >>
>> >
>> > --
>> > Sincerely yours, Ivan Daschinskiy
>>
>>
>
> --
> Sincerely yours, Ivan Daschinskiy
>


-- 

Best regards,
Ivan Pavlukhin

Re: [DISCUSSION] Reject join of nodes with different character encodings

[Mikhail Petrov]

Ivan Daschinsky,


better variant is  to enforce all strings to be encoded in
UTF-8


I agree that it is possible way to go. But now multiple components 
independently serialize strings for their needs and use default encoding 
for this.
For example  DirectByteBufferStreamImplV2#writeString, 
MetaStorage#writeRaw and so on. Even if we fix all this cases we cannot 
guarantee that described above problem will not arise again.


Also it seems to be easy for the user  to  specify encoding for the 
Ignite Java process manually - through `file.encoding` system property.


Ivan Pavlukhin,

I guess Nikolay is talking about the problem with UTF-8 in case string 
contains unpaired surrogate symbols (e.g. used for encoding in UTF-16). 
In this case UTF-8 fails to serialize this string correctly since 
unpaired surrogates characters are forbidden  in UTF-8. Though this 
problem was solved for binary marshaller - see 
`BinaryWriterExImpl#doWriteString`  and `BinaryUtils#strToUtf8Bytes`


On 13.12.2021 13:57, Ivan Pavlukhin wrote:

UTF-8 can’t encode all UNICODE characters.

Nikolay, could you please elaborate? My understanding is that encoding
we speak about matters for conversion from byte arrays to strings.
Does Java String support all unicode characters and particularly does
it support more characters than UTF-8 (I am not saying here that java
String uses UTF-8)?

2021-12-13 12:56 GMT+03:00, Ivan Daschinsky :

UTF-8 is already a default encoding in our BinaryObject format. So I am
for unification.

пн, 13 дек. 2021 г. в 12:50, Nikolay Izhikov :


Hello, Ivan.

UTF-8 can’t encode all UNICODE characters.


13 дек. 2021 г., в 12:49, Ivan Daschinsky 

написал(а):

Khm, maybe a better variant is  to enforce all strings to be encoded in
UTF-8?
AFAIK multi OS cluster is a quite common case.


пн, 13 дек. 2021 г. в 11:36, Mikhail Petrov :


Igniters,

Recently we faced the problem that if the cluster consists of nodes
running in the JVM with different encodings, many issues arise.
The root cause of the mentioned issues is components that use
`String#getBytes()` and `new String()`, which relies on
the
system default encoding. Thus, if a string is deserialized on a node
with a different encoding from the one that serialized it, the
deserialized string can be different from the original one.

For example:

Serialization/deserialization of string in communication messages may
be
broken for some strings on nodes running in a JVM with a different
encoding as DirectByteBufferStreamImplV2 uses String#getBytes() to
serialize strings - [1]

Or the IgniteAuthenticationProcessor can compute different security
IDs
for the user on different nodes in this case - [2]

What do you think, if we solve this problem globally, by rejecting to
join nodes that run on JVMs with different encodings?

As a result, we will be sure that all cluster nodes have the same
encoding and all related problems will be solved.

[1] - https://issues.apache.org/jira/browse/IGNITE-16106
[2] - https://issues.apache.org/jira/browse/IGNITE-16068

--
Mikhail



--
Sincerely yours, Ivan Daschinskiy



--
Sincerely yours, Ivan Daschinskiy

Re: [DISCUSSION] Reject join of nodes with different character encodings

[Nikolay Izhikov]

> Does Java String support all unicode characters and particularly does it 
> support more characters than UTF-8

It’s not about Java, it’s about UTF-8 standard.

Please, take a look at [1] 

> In November 2003, UTF-8 was restricted by RFC 3629 to match the constraints 
> of the UTF-16 character encoding: explicitly prohibiting code points 
> corresponding to the high and low surrogate characters removed more than 3% 
> of the three-byte sequences, and ending at U+10 removed more than 48% of 
> the four-byte sequences and all five- and six-byte sequences.

And [2] 

> The definition of UTF-8 prohibits encoding character numbers between U+D800 
> and U+DFFF, which are reserved for use with the UTF-16 encoding form (as 
> surrogate pairs) and do not directly represent characters.

Actually, we already has some modes to support this restriction of UTF-8.
Please, take a look at BinaryUtils#utf8BytesToStr [3]


[1] https://en.wikipedia.org/wiki/UTF-8
[2] https://datatracker.ietf.org/doc/html/rfc3629
[3] 
https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/binary/BinaryUtils.java#L2387

> 13 дек. 2021 г., в 13:57, Ivan Pavlukhin  написал(а):
> 
>> UTF-8 can’t encode all UNICODE characters.
> 
> Nikolay, could you please elaborate? My understanding is that encoding
> we speak about matters for conversion from byte arrays to strings.
> Does Java String support all unicode characters and particularly does
> it support more characters than UTF-8 (I am not saying here that java
> String uses UTF-8)?
> 
> 2021-12-13 12:56 GMT+03:00, Ivan Daschinsky :
>> UTF-8 is already a default encoding in our BinaryObject format. So I am
>> for unification.
>> 
>> пн, 13 дек. 2021 г. в 12:50, Nikolay Izhikov :
>> 
>>> Hello, Ivan.
>>> 
>>> UTF-8 can’t encode all UNICODE characters.
>>> 
 13 дек. 2021 г., в 12:49, Ivan Daschinsky 
>>> написал(а):
 
 Khm, maybe a better variant is  to enforce all strings to be encoded in
 UTF-8?
 AFAIK multi OS cluster is a quite common case.
 
 
 пн, 13 дек. 2021 г. в 11:36, Mikhail Petrov :
 
> Igniters,
> 
> Recently we faced the problem that if the cluster consists of nodes
> running in the JVM with different encodings, many issues arise.
> The root cause of the mentioned issues is components that use
> `String#getBytes()` and `new String()`, which relies on
> the
> system default encoding. Thus, if a string is deserialized on a node
> with a different encoding from the one that serialized it, the
> deserialized string can be different from the original one.
> 
> For example:
> 
> Serialization/deserialization of string in communication messages may
> be
> broken for some strings on nodes running in a JVM with a different
> encoding as DirectByteBufferStreamImplV2 uses String#getBytes() to
> serialize strings - [1]
> 
> Or the IgniteAuthenticationProcessor can compute different security
> IDs
> for the user on different nodes in this case - [2]
> 
> What do you think, if we solve this problem globally, by rejecting to
> join nodes that run on JVMs with different encodings?
> 
> As a result, we will be sure that all cluster nodes have the same
> encoding and all related problems will be solved.
> 
> [1] - https://issues.apache.org/jira/browse/IGNITE-16106
> [2] - https://issues.apache.org/jira/browse/IGNITE-16068
> 
> --
> Mikhail
> 
> 
 
 --
 Sincerely yours, Ivan Daschinskiy
>>> 
>>> 
>> 
>> --
>> Sincerely yours, Ivan Daschinskiy
>> 
> 
> 
> -- 
> 
> Best regards,
> Ivan Pavlukhin

Re: 0-day CVE in log4j

[Anton Vinogradov]

Folks,

My 200 rubles here,
> I want to include it to the 2.12 scope.
Why not 2.11.1 as well?
We should provide a fixed version for current customers asap.
2.12 require migration, while 2.11.1 can be applied as-is.


On Mon, Dec 13, 2021 at 12:18 PM Stephen Darlington <
stephen.darling...@gridgain.com> wrote:

> Another workaround appears to be using the
> -Dlog4j2.formatMsgNoLookups=true option. Also, “Java versions greater than
> 6u211, 7u201, 8u191, and 11.0.1 are less affected by this attack vector, at
> least in theory, because the JNDI can't load remote code using LDAP.”
>
> (
> https://arstechnica.com/information-technology/2021/12/minecraft-and-other-apps-face-serious-threat-from-new-code-execution-bug/
> )
>
> > On 12 Dec 2021, at 10:56, Dmitriy Pavlov  wrote:
> >
> > Hi Igniters,
> >
> > Preliminary: change of the log4j version does not affect any tests
> > (Alexander Nikolaev, correct me if I'm wrong).
> >
> > If you're using embedded Ignite, it's perfectly possible to enforce
> jog4j2
> > dependency to be 2.15.0 in your project final pom.xml or build.gradle or
> > any other build system properties.
> >
> > https://issues.apache.org/jira/browse/IGNITE-16101 ticket seems to be
> > a blocker for 2.12. But for now, as a workaround, it's possible to select
> > the latest version manually.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > сб, 11 дек. 2021 г. в 09:47, Nikita Amelchev :
> >
> >> Hello.
> >>
> >> The issue to update dependency was created:
> >> https://issues.apache.org/jira/browse/IGNITE-16101
> >>
> >> I want to include it to the 2.12 scope.
> >>
> >> сб, 11 дек. 2021 г., 09:19 Raymond Wilson :
> >>
> >>> All
> >>>
> >>> This blew up today: CVE-2021-44228 (
> >>>
> >>>
> >>
> https://www.bleepingcomputer.com/news/security/new-zero-day-exploit-for-log4j-java-library-is-an-enterprise-nightmare/
> >>> )
> >>>
> >>> Will there be a risk assessment with respect to Ignite for this CVE?
> >>>
> >>> Thanks,
> >>> Raymond.
> >>>
> >>> --
> >>> 
> >>> Raymond Wilson
> >>> Trimble Distinguished Engineer, Civil Construction Software (CCS)
> >>> 11 Birmingham Drive | Christchurch, New Zealand
> >>> raymond_wil...@trimble.com
> >>>
> >>> <
> >>>
> >>
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> 
> >>>
> >>
>
>
>

Re: Apache Ignite 2.12 RELEASE [Time, Scope, Manager]

[Nikita Amelchev]

Hello.

I have cherry-picked the blocked [1] to the 2.12. It updates the log4j
version to 2.15.

The release is blocked by one blocker issue with auth.

[1] https://issues.apache.org/jira/browse/IGNITE-16101
[2] https://issues.apache.org/jira/browse/IGNITE-16068

чт, 9 дек. 2021 г. в 12:44, Nikita Amelchev :
>
> Hello, Nikolay.
>
> I have cherry-picked the issue fix.
>
> Thank you.
>
> ср, 8 дек. 2021 г. в 18:47, Nikolay Izhikov :
> >
> > Hello.
> >
> > Let’s include [1] in 2.12 scope.
> >
> > After migrations authentication processor to security API we have an issue.
> > Persistent data region should exists on client node if authentication is 
> > enabled.
> >
> > It seems very annoying for the users.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-15969
> >
> > > 2 дек. 2021 г., в 19:50, Nikita Amelchev  
> > > написал(а):
> > >
> > > I would like to formally announce code freeze for 2.12.0. Only
> > > blockers are accepted to be included to the scope. [1]
> > >
> > > We have one blocker issue [2] that will be fixed soon.
> > >
> > > [1] 
> > > https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P3.Stabilization
> > > [2] https://issues.apache.org/jira/browse/IGNITE-15966
> > >
> > > чт, 2 дек. 2021 г. в 16:01, Nikita Amelchev :
> > >>
> > >> Hi, Igniters.
> > >>
> > >> The release is blocked by the auth issue [1] discussed above. The
> > >> patch will be prepared at the nearest time.
> > >>
> > >> I want to include the useful issue that adds the ability to configure
> > >> snapshot thread pool size if nobody minds.
> > >>
> > >> [1] https://issues.apache.org/jira/browse/IGNITE-15966
> > >> [2] https://issues.apache.org/jira/browse/IGNITE-16014
> > >>
> > >> пт, 26 нояб. 2021 г. в 11:44, Nikita Amelchev :
> > >>>
> > >>> Hello,
> > >>>
> > >>> I want to include the issue [1] to the 2.12 scope. It fixes some
> > >>> metrics according to the JSR107.
> > >>>
> > >>> Any objections?
> > >>>
> > >>> [1] https://issues.apache.org/jira/browse/IGNITE-16002 The remove
> > >>> cache method should update statistics if the method returns true
> > >>>
> > >>> чт, 25 нояб. 2021 г. в 17:12, Nikita Amelchev :
> > 
> >  Hello Ivan,
> > 
> >  +1, the issue can be cherry-picked to the 2.12.
> > 
> >  чт, 25 нояб. 2021 г. в 17:07, Ivan Bessonov :
> > >
> > > Hello everyone,
> > >
> > > is there a chance to add this issue to the release scope? [1]
> > > Currently, defragmentation of certain types of indexes can corrupt 
> > > them.
> > > Fix is straightforward and should not cause any problems.
> > > Thank you!
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-15968
> > >
> > > вт, 23 нояб. 2021 г. в 20:44, Nikita Amelchev :
> > >
> > >> Ivan,
> > >>
> > >> I have cherry-picked the issue:
> > >> https://issues.apache.org/jira/browse/IGNITE-15767
> > >>
> > >> вт, 23 нояб. 2021 г. в 20:31, Nikolay Izhikov :
> > >>>
> >  [1] https://issues.apache.org/jira/browse/IGNITE-15951
> > >>>
> > >>> Cherry-picked to 2.12
> > >>>
> >  23 нояб. 2021 г., в 16:38, Nikita Amelchev 
> > >> написал(а):
> > 
> >  Hi, +1 to cherry-pick.
> > 
> >  вт, 23 нояб. 2021 г. в 15:25, Ivan Daschinsky 
> >  :
> > >
> > > Hi! Lets add one simple bug fix, but very useful
> > >
> > > https://issues.apache.org/jira/browse/IGNITE-15767
> > >
> > > пн, 22 нояб. 2021 г. в 15:22, Pavel Tupitsyn 
> > > :
> > >
> > >> Yes, let's fix those auth issues, there are so many complaints on
> > >> the user
> > >> list.
> > >>
> > >> On Mon, Nov 22, 2021 at 1:03 PM Mikhail Petrov <
> > >> pmgheap@gmail.com>
> > >> wrote:
> > >>
> > >>> Igniters,
> > >>> it seems that issues [1] and [2] that were discovered recently 
> > >>> are
> > >>> blockers for the 2.12 release.
> > >>>
> > >>> [1] https://issues.apache.org/jira/browse/IGNITE-15951
> > >>> [2] https://issues.apache.org/jira/browse/IGNITE-15966
> > >>>
> > >>> On 22.11.2021 11:04, Nikolay Izhikov wrote:
> >  Hello.
> > 
> >  One more tiny fix that will improve java thin client 
> >  performance in
> > >>> certain cases [1]
> >  I think it worth to cherry-pick it to 2.12 release.
> > 
> >  Any objections?
> > 
> >  [1] https://issues.apache.org/jira/browse/IGNITE-15924
> > 
> > > 19 нояб. 2021 г., в 14:56, Nikita Amelchev 
> > > 
> > >>> написал(а):
> > >
> > > Vladimir, +1.
> > >
> > > I have cherry-picked the issue.
> > >
> > > пт, 19 нояб. 2021 г. в 14:53, Steshin Vladimir <
>

Re: Apache Ignite 2.12 RELEASE [Time, Scope, Manager]

[Nikolay Izhikov]

Nikita.

I don’t think IGNITE-16068 is a release blocker.

1. It only happens when authentication enabled.
2. There is at lease 2 workarounds:
a. Use the same file.encoding for all Ignite nodes.
b. Use only latin characters in user login.

I propose to postpone this ticket and move on with the release.

> 13 дек. 2021 г., в 15:28, Nikita Amelchev  написал(а):
> 
> Hello.
> 
> I have cherry-picked the blocked [1] to the 2.12. It updates the log4j
> version to 2.15.
> 
> The release is blocked by one blocker issue with auth.
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-16101
> [2] https://issues.apache.org/jira/browse/IGNITE-16068
> 
> чт, 9 дек. 2021 г. в 12:44, Nikita Amelchev :
>> 
>> Hello, Nikolay.
>> 
>> I have cherry-picked the issue fix.
>> 
>> Thank you.
>> 
>> ср, 8 дек. 2021 г. в 18:47, Nikolay Izhikov :
>>> 
>>> Hello.
>>> 
>>> Let’s include [1] in 2.12 scope.
>>> 
>>> After migrations authentication processor to security API we have an issue.
>>> Persistent data region should exists on client node if authentication is 
>>> enabled.
>>> 
>>> It seems very annoying for the users.
>>> 
>>> [1] https://issues.apache.org/jira/browse/IGNITE-15969
>>> 
 2 дек. 2021 г., в 19:50, Nikita Amelchev  написал(а):
 
 I would like to formally announce code freeze for 2.12.0. Only
 blockers are accepted to be included to the scope. [1]
 
 We have one blocker issue [2] that will be fixed soon.
 
 [1] 
 https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P3.Stabilization
 [2] https://issues.apache.org/jira/browse/IGNITE-15966
 
 чт, 2 дек. 2021 г. в 16:01, Nikita Amelchev :
> 
> Hi, Igniters.
> 
> The release is blocked by the auth issue [1] discussed above. The
> patch will be prepared at the nearest time.
> 
> I want to include the useful issue that adds the ability to configure
> snapshot thread pool size if nobody minds.
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-15966
> [2] https://issues.apache.org/jira/browse/IGNITE-16014
> 
> пт, 26 нояб. 2021 г. в 11:44, Nikita Amelchev :
>> 
>> Hello,
>> 
>> I want to include the issue [1] to the 2.12 scope. It fixes some
>> metrics according to the JSR107.
>> 
>> Any objections?
>> 
>> [1] https://issues.apache.org/jira/browse/IGNITE-16002 The remove
>> cache method should update statistics if the method returns true
>> 
>> чт, 25 нояб. 2021 г. в 17:12, Nikita Amelchev :
>>> 
>>> Hello Ivan,
>>> 
>>> +1, the issue can be cherry-picked to the 2.12.
>>> 
>>> чт, 25 нояб. 2021 г. в 17:07, Ivan Bessonov :
 
 Hello everyone,
 
 is there a chance to add this issue to the release scope? [1]
 Currently, defragmentation of certain types of indexes can corrupt 
 them.
 Fix is straightforward and should not cause any problems.
 Thank you!
 
 [1] https://issues.apache.org/jira/browse/IGNITE-15968
 
 вт, 23 нояб. 2021 г. в 20:44, Nikita Amelchev :
 
> Ivan,
> 
> I have cherry-picked the issue:
> https://issues.apache.org/jira/browse/IGNITE-15767
> 
> вт, 23 нояб. 2021 г. в 20:31, Nikolay Izhikov :
>> 
>>> [1] https://issues.apache.org/jira/browse/IGNITE-15951
>> 
>> Cherry-picked to 2.12
>> 
>>> 23 нояб. 2021 г., в 16:38, Nikita Amelchev 
> написал(а):
>>> 
>>> Hi, +1 to cherry-pick.
>>> 
>>> вт, 23 нояб. 2021 г. в 15:25, Ivan Daschinsky :
 
 Hi! Lets add one simple bug fix, but very useful
 
 https://issues.apache.org/jira/browse/IGNITE-15767
 
 пн, 22 нояб. 2021 г. в 15:22, Pavel Tupitsyn 
 :
 
> Yes, let's fix those auth issues, there are so many complaints on
> the user
> list.
> 
> On Mon, Nov 22, 2021 at 1:03 PM Mikhail Petrov <
> pmgheap@gmail.com>
> wrote:
> 
>> Igniters,
>> it seems that issues [1] and [2] that were discovered recently 
>> are
>> blockers for the 2.12 release.
>> 
>> [1] https://issues.apache.org/jira/browse/IGNITE-15951
>> [2] https://issues.apache.org/jira/browse/IGNITE-15966
>> 
>> On 22.11.2021 11:04, Nikolay Izhikov wrote:
>>> Hello.
>>> 
>>> One more tiny fix that will improve java thin client 
>>> performance in
>> certain cases [1]
>>> I think it worth to cherry-pick it to 2.12 release.
>>> 
>>> Any objections?
>>> 
>>> [1] https://issues.apache.org/jira/browse/IGNITE-1592

Re: Ignite Website UI and UX improvements

[Mauricio Stekl]

Hi all,

I'm happy to let you know the new website is ready, and I will start
merging the code and do the launch today. Please avoid committing new
changes to the current website until this process is ready.

I will share an update once it's live.

Thanks!

Mauricio

On Tue, Nov 9, 2021 at 9:00 PM Denis Magda  wrote:

> Igniters,
>
> I'm pleased to share a piece of good news with you. The next version of the
> website is ready and deployed on staging:
>
> https://test2.heavenweb.ru/
> username: apache
> passwd: ignite
>
> In the course of the next 5-10 days, I'll be collaborating with the design
> and dev teams on required improvements. Please pass on any feedback that
> you might have.
>
> -
> Denis
>
>
> On Mon, Sep 13, 2021 at 8:46 AM Denis Magda  wrote:
>
> > Hi Saikat,
> >
> > Thanks for the suggestions. I'll pass them over to the design team.
> >
> >
> > -
> > Denis
> >
> >
> > On Sun, Sep 12, 2021 at 10:43 AM Saikat Maitra 
> > wrote:
> >
> >> Hi Denis, Kseniya
> >>
> >> Thank you for leading the effort in designing and revamping the website.
> >>
> >> I wanted to share a few thoughts and ideas:
> >> 1. Will it be suitable if the rocket exhaust fire/gases are replaced
> with
> >> the Ignite spark from the Ignite logo. I really like the red Ignite
> spark.
> >> It may symbolize that the rocket's journey is powered / fueled by
> Ignite.
> >> 2. Can we rewrite the sentence "We appreciate any help and track every
> >> contribution" to "We appreciate any help and value every contribution".
> >> The
> >> word track felt a bit intrusive like if the user's actions are
> monitored.
> >> 3. Similarly here we may replace the word track "Top contributors are
> >> tracked at Github" to "Top contributors @ Github".
> >> 4. If it is ok with PMC members and committers then we can use face
> shots
> >> for committers and PMC members like here
> >> https://kafka.apache.org/committers.
> >> It will be nice to see a face and recognize them in person when we all
> >> meet
> >> again in person.
> >>
> >> Warm regards,
> >> Saikat
> >>
> >>
> >>
> >> On Wed, Sep 8, 2021 at 11:34 AM Denis Magda  wrote:
> >>
> >> > Folks, an update for you. We keep prototyping new pages and you can
> >> find a
> >> > reference to the mockups in the ticket.
> >> >
> >> > Need to hear your opinion on one of the mockups. Kseniya and I worked
> >> with
> >> > the design studio on a Community page, and this is what we have
> >> prepared so
> >> > far:
> >> > https://app.moqups.com/4WR00aUZ8z/view/page/a7e1c85f0
> >> >
> >> > The page tells a brief story about the project, has a "meet the
> >> community"
> >> > section, explains how to start contributing or ask a question. All
> those
> >> > sections will have anchors and will be added as separate menu items
> for
> >> > quick navigation.
> >> >
> >> > Any suggestions before we send the mockup to editors and designers?
> >> >
> >> > --
> >> > Denis
> >> >
> >> > -
> >> > Denis
> >> >
> >> > On Mon, Aug 16, 2021 at 4:35 PM Denis Magda 
> wrote:
> >> >
> >> > > Folks,
> >> > >
> >> > > In early 2021 a 3rd-party vendor helped us to conduct a usability
> >> audit
> >> > of
> >> > > the Ignite website. The audit revealed significant user experience
> and
> >> > > structure-related gaps that make it harder to understand Ignite,
> >> > navigate,
> >> > > consume content.
> >> > >
> >> > > We've just started bringing those changes to life:
> >> > > https://issues.apache.org/jira/browse/IGNITE-15320
> >> > >
> >> > > I'll keep you posted on the progress and intermediate results. Let
> me
> >> > know
> >> > > if:
> >> > > a) you have any questions or suggestions
> >> > > b) you want to be involved in this effort and brief the team (the
> same
> >> > > vendor) on the structure/content of the community and other pages. I
> >> meet
> >> > > with the folks several days a week to help them finish the project
> by
> >> > late
> >> > > September- early October.
> >> > >
> >> > > --
> >> > > Denis
> >> > >
> >> > > -
> >> > > Denis
> >> > >
> >> >
> >>
> >
>

Re: 0-day CVE in log4j

[Maxim Muzafarov]

+1 for the 2.11.1

On Mon, 13 Dec 2021 at 15:21, Anton Vinogradov  wrote:
>
> Folks,
>
> My 200 rubles here,
> > I want to include it to the 2.12 scope.
> Why not 2.11.1 as well?
> We should provide a fixed version for current customers asap.
> 2.12 require migration, while 2.11.1 can be applied as-is.
>
>
> On Mon, Dec 13, 2021 at 12:18 PM Stephen Darlington <
> stephen.darling...@gridgain.com> wrote:
>
> > Another workaround appears to be using the
> > -Dlog4j2.formatMsgNoLookups=true option. Also, “Java versions greater than
> > 6u211, 7u201, 8u191, and 11.0.1 are less affected by this attack vector, at
> > least in theory, because the JNDI can't load remote code using LDAP.”
> >
> > (
> > https://arstechnica.com/information-technology/2021/12/minecraft-and-other-apps-face-serious-threat-from-new-code-execution-bug/
> > )
> >
> > > On 12 Dec 2021, at 10:56, Dmitriy Pavlov  wrote:
> > >
> > > Hi Igniters,
> > >
> > > Preliminary: change of the log4j version does not affect any tests
> > > (Alexander Nikolaev, correct me if I'm wrong).
> > >
> > > If you're using embedded Ignite, it's perfectly possible to enforce
> > jog4j2
> > > dependency to be 2.15.0 in your project final pom.xml or build.gradle or
> > > any other build system properties.
> > >
> > > https://issues.apache.org/jira/browse/IGNITE-16101 ticket seems to be
> > > a blocker for 2.12. But for now, as a workaround, it's possible to select
> > > the latest version manually.
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > сб, 11 дек. 2021 г. в 09:47, Nikita Amelchev :
> > >
> > >> Hello.
> > >>
> > >> The issue to update dependency was created:
> > >> https://issues.apache.org/jira/browse/IGNITE-16101
> > >>
> > >> I want to include it to the 2.12 scope.
> > >>
> > >> сб, 11 дек. 2021 г., 09:19 Raymond Wilson :
> > >>
> > >>> All
> > >>>
> > >>> This blew up today: CVE-2021-44228 (
> > >>>
> > >>>
> > >>
> > https://www.bleepingcomputer.com/news/security/new-zero-day-exploit-for-log4j-java-library-is-an-enterprise-nightmare/
> > >>> )
> > >>>
> > >>> Will there be a risk assessment with respect to Ignite for this CVE?
> > >>>
> > >>> Thanks,
> > >>> Raymond.
> > >>>
> > >>> --
> > >>> 
> > >>> Raymond Wilson
> > >>> Trimble Distinguished Engineer, Civil Construction Software (CCS)
> > >>> 11 Birmingham Drive | Christchurch, New Zealand
> > >>> raymond_wil...@trimble.com
> > >>>
> > >>> <
> > >>>
> > >>
> > https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > 
> > >>>
> > >>
> >
> >
> >

Re: 0-day CVE in log4j

[Ivan Daschinsky]

Unfortunately, we need patch our Log4j2 adapter in order to work with
log4j-2.15
So there is no choice other than to release 2.11.1

пн, 13 дек. 2021 г. в 15:21, Anton Vinogradov :

> Folks,
>
> My 200 rubles here,
> > I want to include it to the 2.12 scope.
> Why not 2.11.1 as well?
> We should provide a fixed version for current customers asap.
> 2.12 require migration, while 2.11.1 can be applied as-is.
>
>
> On Mon, Dec 13, 2021 at 12:18 PM Stephen Darlington <
> stephen.darling...@gridgain.com> wrote:
>
> > Another workaround appears to be using the
> > -Dlog4j2.formatMsgNoLookups=true option. Also, “Java versions greater
> than
> > 6u211, 7u201, 8u191, and 11.0.1 are less affected by this attack vector,
> at
> > least in theory, because the JNDI can't load remote code using LDAP.”
> >
> > (
> >
> https://arstechnica.com/information-technology/2021/12/minecraft-and-other-apps-face-serious-threat-from-new-code-execution-bug/
> > )
> >
> > > On 12 Dec 2021, at 10:56, Dmitriy Pavlov  wrote:
> > >
> > > Hi Igniters,
> > >
> > > Preliminary: change of the log4j version does not affect any tests
> > > (Alexander Nikolaev, correct me if I'm wrong).
> > >
> > > If you're using embedded Ignite, it's perfectly possible to enforce
> > jog4j2
> > > dependency to be 2.15.0 in your project final pom.xml or build.gradle
> or
> > > any other build system properties.
> > >
> > > https://issues.apache.org/jira/browse/IGNITE-16101 ticket seems to be
> > > a blocker for 2.12. But for now, as a workaround, it's possible to
> select
> > > the latest version manually.
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > сб, 11 дек. 2021 г. в 09:47, Nikita Amelchev :
> > >
> > >> Hello.
> > >>
> > >> The issue to update dependency was created:
> > >> https://issues.apache.org/jira/browse/IGNITE-16101
> > >>
> > >> I want to include it to the 2.12 scope.
> > >>
> > >> сб, 11 дек. 2021 г., 09:19 Raymond Wilson  >:
> > >>
> > >>> All
> > >>>
> > >>> This blew up today: CVE-2021-44228 (
> > >>>
> > >>>
> > >>
> >
> https://www.bleepingcomputer.com/news/security/new-zero-day-exploit-for-log4j-java-library-is-an-enterprise-nightmare/
> > >>> )
> > >>>
> > >>> Will there be a risk assessment with respect to Ignite for this CVE?
> > >>>
> > >>> Thanks,
> > >>> Raymond.
> > >>>
> > >>> --
> > >>> 
> > >>> Raymond Wilson
> > >>> Trimble Distinguished Engineer, Civil Construction Software (CCS)
> > >>> 11 Birmingham Drive | Christchurch, New Zealand
> > >>> raymond_wil...@trimble.com
> > >>>
> > >>> <
> > >>>
> > >>
> >
> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > 
> > >>>
> > >>
> >
> >
> >
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: 0-day CVE in log4j

[Maxim Muzafarov]

Folks,

Should we describe all the WA available for the issue [1]? There is
already a lot of information about CVE, and nevertheless, it will not
be superfluous.

[1] https://issues.apache.org/jira/browse/IGNITE-16101

On Mon, 13 Dec 2021 at 15:37, Ivan Daschinsky  wrote:
>
> Unfortunately, we need patch our Log4j2 adapter in order to work with
> log4j-2.15
> So there is no choice other than to release 2.11.1
>
> пн, 13 дек. 2021 г. в 15:21, Anton Vinogradov :
>
> > Folks,
> >
> > My 200 rubles here,
> > > I want to include it to the 2.12 scope.
> > Why not 2.11.1 as well?
> > We should provide a fixed version for current customers asap.
> > 2.12 require migration, while 2.11.1 can be applied as-is.
> >
> >
> > On Mon, Dec 13, 2021 at 12:18 PM Stephen Darlington <
> > stephen.darling...@gridgain.com> wrote:
> >
> > > Another workaround appears to be using the
> > > -Dlog4j2.formatMsgNoLookups=true option. Also, “Java versions greater
> > than
> > > 6u211, 7u201, 8u191, and 11.0.1 are less affected by this attack vector,
> > at
> > > least in theory, because the JNDI can't load remote code using LDAP.”
> > >
> > > (
> > >
> > https://arstechnica.com/information-technology/2021/12/minecraft-and-other-apps-face-serious-threat-from-new-code-execution-bug/
> > > )
> > >
> > > > On 12 Dec 2021, at 10:56, Dmitriy Pavlov  wrote:
> > > >
> > > > Hi Igniters,
> > > >
> > > > Preliminary: change of the log4j version does not affect any tests
> > > > (Alexander Nikolaev, correct me if I'm wrong).
> > > >
> > > > If you're using embedded Ignite, it's perfectly possible to enforce
> > > jog4j2
> > > > dependency to be 2.15.0 in your project final pom.xml or build.gradle
> > or
> > > > any other build system properties.
> > > >
> > > > https://issues.apache.org/jira/browse/IGNITE-16101 ticket seems to be
> > > > a blocker for 2.12. But for now, as a workaround, it's possible to
> > select
> > > > the latest version manually.
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > > сб, 11 дек. 2021 г. в 09:47, Nikita Amelchev :
> > > >
> > > >> Hello.
> > > >>
> > > >> The issue to update dependency was created:
> > > >> https://issues.apache.org/jira/browse/IGNITE-16101
> > > >>
> > > >> I want to include it to the 2.12 scope.
> > > >>
> > > >> сб, 11 дек. 2021 г., 09:19 Raymond Wilson  > >:
> > > >>
> > > >>> All
> > > >>>
> > > >>> This blew up today: CVE-2021-44228 (
> > > >>>
> > > >>>
> > > >>
> > >
> > https://www.bleepingcomputer.com/news/security/new-zero-day-exploit-for-log4j-java-library-is-an-enterprise-nightmare/
> > > >>> )
> > > >>>
> > > >>> Will there be a risk assessment with respect to Ignite for this CVE?
> > > >>>
> > > >>> Thanks,
> > > >>> Raymond.
> > > >>>
> > > >>> --
> > > >>> 
> > > >>> Raymond Wilson
> > > >>> Trimble Distinguished Engineer, Civil Construction Software (CCS)
> > > >>> 11 Birmingham Drive | Christchurch, New Zealand
> > > >>> raymond_wil...@trimble.com
> > > >>>
> > > >>> <
> > > >>>
> > > >>
> > >
> > https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch
> > > 
> > > >>>
> > > >>
> > >
> > >
> > >
> >
>
>
> --
> Sincerely yours, Ivan Daschinskiy

Re: Apache Ignite 2.12 RELEASE [Time, Scope, Manager]

[Pavel Tupitsyn]

Agree with Nikolay, let's proceed with the release.

On Mon, Dec 13, 2021 at 3:31 PM Nikolay Izhikov  wrote:

> Nikita.
>
> I don’t think IGNITE-16068 is a release blocker.
>
> 1. It only happens when authentication enabled.
> 2. There is at lease 2 workarounds:
> a. Use the same file.encoding for all Ignite nodes.
> b. Use only latin characters in user login.
>
> I propose to postpone this ticket and move on with the release.
>
> > 13 дек. 2021 г., в 15:28, Nikita Amelchev 
> написал(а):
> >
> > Hello.
> >
> > I have cherry-picked the blocked [1] to the 2.12. It updates the log4j
> > version to 2.15.
> >
> > The release is blocked by one blocker issue with auth.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-16101
> > [2] https://issues.apache.org/jira/browse/IGNITE-16068
> >
> > чт, 9 дек. 2021 г. в 12:44, Nikita Amelchev :
> >>
> >> Hello, Nikolay.
> >>
> >> I have cherry-picked the issue fix.
> >>
> >> Thank you.
> >>
> >> ср, 8 дек. 2021 г. в 18:47, Nikolay Izhikov :
> >>>
> >>> Hello.
> >>>
> >>> Let’s include [1] in 2.12 scope.
> >>>
> >>> After migrations authentication processor to security API we have an
> issue.
> >>> Persistent data region should exists on client node if authentication
> is enabled.
> >>>
> >>> It seems very annoying for the users.
> >>>
> >>> [1] https://issues.apache.org/jira/browse/IGNITE-15969
> >>>
>  2 дек. 2021 г., в 19:50, Nikita Amelchev 
> написал(а):
> 
>  I would like to formally announce code freeze for 2.12.0. Only
>  blockers are accepted to be included to the scope. [1]
> 
>  We have one blocker issue [2] that will be fixed soon.
> 
>  [1]
> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P3.Stabilization
>  [2] https://issues.apache.org/jira/browse/IGNITE-15966
> 
>  чт, 2 дек. 2021 г. в 16:01, Nikita Amelchev :
> >
> > Hi, Igniters.
> >
> > The release is blocked by the auth issue [1] discussed above. The
> > patch will be prepared at the nearest time.
> >
> > I want to include the useful issue that adds the ability to configure
> > snapshot thread pool size if nobody minds.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-15966
> > [2] https://issues.apache.org/jira/browse/IGNITE-16014
> >
> > пт, 26 нояб. 2021 г. в 11:44, Nikita Amelchev  >:
> >>
> >> Hello,
> >>
> >> I want to include the issue [1] to the 2.12 scope. It fixes some
> >> metrics according to the JSR107.
> >>
> >> Any objections?
> >>
> >> [1] https://issues.apache.org/jira/browse/IGNITE-16002 The remove
> >> cache method should update statistics if the method returns true
> >>
> >> чт, 25 нояб. 2021 г. в 17:12, Nikita Amelchev  >:
> >>>
> >>> Hello Ivan,
> >>>
> >>> +1, the issue can be cherry-picked to the 2.12.
> >>>
> >>> чт, 25 нояб. 2021 г. в 17:07, Ivan Bessonov  >:
> 
>  Hello everyone,
> 
>  is there a chance to add this issue to the release scope? [1]
>  Currently, defragmentation of certain types of indexes can
> corrupt them.
>  Fix is straightforward and should not cause any problems.
>  Thank you!
> 
>  [1] https://issues.apache.org/jira/browse/IGNITE-15968
> 
>  вт, 23 нояб. 2021 г. в 20:44, Nikita Amelchev <
> namelc...@apache.org>:
> 
> > Ivan,
> >
> > I have cherry-picked the issue:
> > https://issues.apache.org/jira/browse/IGNITE-15767
> >
> > вт, 23 нояб. 2021 г. в 20:31, Nikolay Izhikov <
> nizhi...@apache.org>:
> >>
> >>> [1] https://issues.apache.org/jira/browse/IGNITE-15951
> >>
> >> Cherry-picked to 2.12
> >>
> >>> 23 нояб. 2021 г., в 16:38, Nikita Amelchev <
> namelc...@apache.org>
> > написал(а):
> >>>
> >>> Hi, +1 to cherry-pick.
> >>>
> >>> вт, 23 нояб. 2021 г. в 15:25, Ivan Daschinsky <
> ivanda...@gmail.com>:
> 
>  Hi! Lets add one simple bug fix, but very useful
> 
>  https://issues.apache.org/jira/browse/IGNITE-15767
> 
>  пн, 22 нояб. 2021 г. в 15:22, Pavel Tupitsyn <
> ptupit...@apache.org>:
> 
> > Yes, let's fix those auth issues, there are so many
> complaints on
> > the user
> > list.
> >
> > On Mon, Nov 22, 2021 at 1:03 PM Mikhail Petrov <
> > pmgheap@gmail.com>
> > wrote:
> >
> >> Igniters,
> >> it seems that issues [1] and [2] that were discovered
> recently are
> >> blockers for the 2.12 release.
> >>
> >> [1] https://issues.apache.org/jira/browse/IGNITE-15951
> >> [2] https://issues.apache.org/jira/browse/IGNITE-15966
> >>
> 

Re: Re[2]: NUMA aware allocator, PR review request

[Ivan Daschinsky]

Thanks all for review, merged into master.

пн, 6 дек. 2021 г. в 23:26, Ivan Daschinsky :

> You are not wrong, it is built from source, every night. And every TC run.
> I don't understand why numa allocator cannot be treated the same. Moreover,
> it is built using maven, with maven plugin and just needs gcc and
> libnuma-dev. All of theese are already on TC agents and build are ready. I
> didn't see any difficulties in building.
>
> But I still don't understand how to organize Anton's proposal. It seems
> that if we follow this way, we cannot release allocator in 2.13
>
> пн, 6 дек. 2021 г., 23:18 Ilya Kasnacheev :
>
>> Hello!
>>
>> Maybe I am wrong, but ODBC installer is built from source and may be
>> improved from release to release.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пн, 6 дек. 2021 г. в 20:41, Ivan Daschinsky :
>>
>> > Only one reason -- nowadays amost all hardware platforms uses NUMA
>> >
>> > Another reason -- there is no any release process of extensions.
>> >
>> >
>> > BTW, apache ignite release is shipped with odbc binary installer for
>> > windows. And nobody complains about it.
>> >
>> > But may be listen to others?
>> >
>> > пн, 6 дек. 2021 г., 19:49 Anton Vinogradov :
>> >
>> > > Any reason to release the same cpp sources for each release?
>> > > Any reason to increase the requirements amount for each new release?
>> > > Any reason to increase release complexity and duration?
>> > > All answers are "definitely no"
>> > >
>> > > What we should do is to release cpp part once and use it as a
>> dependency.
>> > > Extensions are a good location.
>> > >
>> > > On Mon, Dec 6, 2021 at 3:11 PM Zhenya Stanilovsky
>> > >  wrote:
>> > >
>> > > >
>> > > >
>> > > > +1 with Ivan, let`s store it in core product just because it looks
>> > > > like inalienable functionality and release cycle of extensions a
>> little
>> > > bit
>> > > > different.
>> > > >
>> > > >
>> > > >
>> > > > >Anton, I disagree.
>> > > > >
>> > > > >1. This should be released with main distro.
>> > > > >2. This should not be abandoned.
>> > > > >3. There is not any release process in ignite-extensions.
>> > > > >4. Everything is working now and working good.
>> > > > >
>> > > > >
>> > > > >So lets do not do this :)
>> > > > >
>> > > > >пн, 6 дек. 2021 г. в 14:49, Anton Vinogradov < a...@apache.org >:
>> > > > >
>> > > > >> Let's move all GCC-related parts to ignite-extensions, release,
>> and
>> > > use
>> > > > >> them as a maven dependency.
>> > > > >>
>> > > > >>
>> > > > >> On Fri, Dec 3, 2021 at 1:08 PM Ivan Daschinsky <
>> > ivanda...@gmail.com
>> > > >
>> > > > >> wrote:
>> > > > >>
>> > > > >> > Ok, TC suite is ready [1].
>> > > > >> > If there is no objections, I will merge it soon.
>> > > > >> >
>> > > > >> > Possible concerns -- now it is required to install
>> > build-essentials
>> > > > and
>> > > > >> > libnuma-dev in order to build ignite on 64 bit linux.
>> > > > >> > I suppose that this is not a big deal, but maybe someone will
>> > > > contradict?
>> > > > >> >
>> > > > >> >
>> > > > >> > [1] --
>> > > > >> >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_NumaAllocator/?mode=builds
>> > > > >> >
>> > > > >> > чт, 2 дек. 2021 г. в 12:03, Ivan Daschinsky <
>> ivanda...@gmail.com
>> > > >:
>> > > > >> >
>> > > > >> > > >> Our runs show about 7-10 speedup,
>> > > > >> > > Sorry, typo 7-10% speedup
>> > > > >> > >
>> > > > >> > > чт, 2 дек. 2021 г. в 12:01, Ivan Daschinsky <
>> > ivanda...@gmail.com
>> > > > >:
>> > > > >> > >
>> > > > >> > >> Andrey, thanks!
>> > > > >> > >>
>> > > > >> > >> This allocator can be tested on every NUMA system.
>> > > > >> > >> Our runs show about 7-10 speedup, if we use allocattor with
>> > > > >> interleaved
>> > > > >> > >> strategy + -XX:+UseNUMA.
>> > > > >> > >> But unfortunately our yardstick benches doesn't use offheap
>> a
>> > > lot,
>> > > > >> > >> usually above one Gb.
>> > > > >> > >> We trying to do more benches with real data and share them,
>> > > > possibly
>> > > > >> in
>> > > > >> > >> meetup.
>> > > > >> > >>
>> > > > >> > >> AFAIK, GG lab servers are two-sockets machines, aren't
>> they? So
>> > > it
>> > > > is
>> > > > >> > >> worth to run benches with a lot data on them, using
>> > > > >> > >> allocator with interleaved strategy (you can skip specifying
>> > numa
>> > > > >> nodes,
>> > > > >> > >> by default it will use all available) and use -XX:+UseNUMA
>> jvm
>> > > > >> > >> flag.
>> > > > >> > >>
>> > > > >> > >>
>> > > > >> > >>
>> > > > >> > >> чт, 2 дек. 2021 г. в 11:48, Andrey Mashenkov <
>> > > > >> >  andrey.mashen...@gmail.com
>> > > > >> > >> >:
>> > > > >> > >>
>> > > > >> > >>> Ivan,
>> > > > >> > >>>
>> > > > >> > >>> Great job. PR looks good.
>> > > > >> > >>>
>> > > > >> > >>> This allocator in interleaved mode and passing
>> `-XX:+UseNUMA`
>> > > > flag to
>> > > > >> > jvm
>> > > > >> > >>> > show promising results on yardstick benches.
>> Technically, G1
>> > >

Re: [DISCUSSION] Reject join of nodes with different character encodings

[Ivan Daschinsky]

>> But now multiple components
>> independently serialize strings for their needs and use default encoding
>> for this.
>> For example  DirectByteBufferStreamImplV2#writeString,
>> MetaStorage#writeRaw and so on
We should fix all of them.

>> BinaryUtils#utf8BytesToStr
Lets use this everywhere.

As for me, I'm expecting a way more problem with enforcing rule to fail,
rather than enforcing all components to use UTF-8
Some weird cases  (surrogate pairs) we can (I strongly believe it is OK)
simply do not consider at all.

пн, 13 дек. 2021 г. в 15:15, Nikolay Izhikov :

> > Does Java String support all unicode characters and particularly does it
> support more characters than UTF-8
>
> It’s not about Java, it’s about UTF-8 standard.
>
> Please, take a look at [1]
>
> > In November 2003, UTF-8 was restricted by RFC 3629 to match the
> constraints of the UTF-16 character encoding: explicitly prohibiting code
> points corresponding to the high and low surrogate characters removed more
> than 3% of the three-byte sequences, and ending at U+10 removed more
> than 48% of the four-byte sequences and all five- and six-byte sequences.
>
> And [2]
>
> > The definition of UTF-8 prohibits encoding character numbers between
> U+D800 and U+DFFF, which are reserved for use with the UTF-16 encoding form
> (as surrogate pairs) and do not directly represent characters.
>
> Actually, we already has some modes to support this restriction of UTF-8.
> Please, take a look at BinaryUtils#utf8BytesToStr [3]
>
>
> [1] https://en.wikipedia.org/wiki/UTF-8
> [2] https://datatracker.ietf.org/doc/html/rfc3629
> [3]
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/binary/BinaryUtils.java#L2387
>
> > 13 дек. 2021 г., в 13:57, Ivan Pavlukhin 
> написал(а):
> >
> >> UTF-8 can’t encode all UNICODE characters.
> >
> > Nikolay, could you please elaborate? My understanding is that encoding
> > we speak about matters for conversion from byte arrays to strings.
> > Does Java String support all unicode characters and particularly does
> > it support more characters than UTF-8 (I am not saying here that java
> > String uses UTF-8)?
> >
> > 2021-12-13 12:56 GMT+03:00, Ivan Daschinsky :
> >> UTF-8 is already a default encoding in our BinaryObject format. So
> I am
> >> for unification.
> >>
> >> пн, 13 дек. 2021 г. в 12:50, Nikolay Izhikov :
> >>
> >>> Hello, Ivan.
> >>>
> >>> UTF-8 can’t encode all UNICODE characters.
> >>>
>  13 дек. 2021 г., в 12:49, Ivan Daschinsky 
> >>> написал(а):
> 
>  Khm, maybe a better variant is  to enforce all strings to be encoded
> in
>  UTF-8?
>  AFAIK multi OS cluster is a quite common case.
> 
> 
>  пн, 13 дек. 2021 г. в 11:36, Mikhail Petrov :
> 
> > Igniters,
> >
> > Recently we faced the problem that if the cluster consists of nodes
> > running in the JVM with different encodings, many issues arise.
> > The root cause of the mentioned issues is components that use
> > `String#getBytes()` and `new String()`, which relies on
> > the
> > system default encoding. Thus, if a string is deserialized on a node
> > with a different encoding from the one that serialized it, the
> > deserialized string can be different from the original one.
> >
> > For example:
> >
> > Serialization/deserialization of string in communication messages may
> > be
> > broken for some strings on nodes running in a JVM with a different
> > encoding as DirectByteBufferStreamImplV2 uses String#getBytes() to
> > serialize strings - [1]
> >
> > Or the IgniteAuthenticationProcessor can compute different security
> > IDs
> > for the user on different nodes in this case - [2]
> >
> > What do you think, if we solve this problem globally, by rejecting to
> > join nodes that run on JVMs with different encodings?
> >
> > As a result, we will be sure that all cluster nodes have the same
> > encoding and all related problems will be solved.
> >
> > [1] - https://issues.apache.org/jira/browse/IGNITE-16106
> > [2] - https://issues.apache.org/jira/browse/IGNITE-16068
> >
> > --
> > Mikhail
> >
> >
> 
>  --
>  Sincerely yours, Ivan Daschinskiy
> >>>
> >>>
> >>
> >> --
> >> Sincerely yours, Ivan Daschinskiy
> >>
> >
> >
> > --
> >
> > Best regards,
> > Ivan Pavlukhin
>
>

-- 
Sincerely yours, Ivan Daschinskiy

Re: [DISCUSSION] Reject join of nodes with different character encodings

[Ilya Kasnacheev]

Hello!

We already have a warning about this, see IgniteKernal.checkFileEncoding()

Regards,
-- 
Ilya Kasnacheev


пн, 13 дек. 2021 г. в 16:26, Ivan Daschinsky :

> >> But now multiple components
> >> independently serialize strings for their needs and use default encoding
> >> for this.
> >> For example  DirectByteBufferStreamImplV2#writeString,
> >> MetaStorage#writeRaw and so on
> We should fix all of them.
>
> >> BinaryUtils#utf8BytesToStr
> Lets use this everywhere.
>
> As for me, I'm expecting a way more problem with enforcing rule to fail,
> rather than enforcing all components to use UTF-8
> Some weird cases  (surrogate pairs) we can (I strongly believe it is OK)
> simply do not consider at all.
>
> пн, 13 дек. 2021 г. в 15:15, Nikolay Izhikov :
>
> > > Does Java String support all unicode characters and particularly does
> it
> > support more characters than UTF-8
> >
> > It’s not about Java, it’s about UTF-8 standard.
> >
> > Please, take a look at [1]
> >
> > > In November 2003, UTF-8 was restricted by RFC 3629 to match the
> > constraints of the UTF-16 character encoding: explicitly prohibiting code
> > points corresponding to the high and low surrogate characters removed
> more
> > than 3% of the three-byte sequences, and ending at U+10 removed more
> > than 48% of the four-byte sequences and all five- and six-byte sequences.
> >
> > And [2]
> >
> > > The definition of UTF-8 prohibits encoding character numbers between
> > U+D800 and U+DFFF, which are reserved for use with the UTF-16 encoding
> form
> > (as surrogate pairs) and do not directly represent characters.
> >
> > Actually, we already has some modes to support this restriction of UTF-8.
> > Please, take a look at BinaryUtils#utf8BytesToStr [3]
> >
> >
> > [1] https://en.wikipedia.org/wiki/UTF-8
> > [2] https://datatracker.ietf.org/doc/html/rfc3629
> > [3]
> >
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/binary/BinaryUtils.java#L2387
> >
> > > 13 дек. 2021 г., в 13:57, Ivan Pavlukhin 
> > написал(а):
> > >
> > >> UTF-8 can’t encode all UNICODE characters.
> > >
> > > Nikolay, could you please elaborate? My understanding is that encoding
> > > we speak about matters for conversion from byte arrays to strings.
> > > Does Java String support all unicode characters and particularly does
> > > it support more characters than UTF-8 (I am not saying here that java
> > > String uses UTF-8)?
> > >
> > > 2021-12-13 12:56 GMT+03:00, Ivan Daschinsky :
> > >> UTF-8 is already a default encoding in our BinaryObject format. So
> > I am
> > >> for unification.
> > >>
> > >> пн, 13 дек. 2021 г. в 12:50, Nikolay Izhikov :
> > >>
> > >>> Hello, Ivan.
> > >>>
> > >>> UTF-8 can’t encode all UNICODE characters.
> > >>>
> >  13 дек. 2021 г., в 12:49, Ivan Daschinsky 
> > >>> написал(а):
> > 
> >  Khm, maybe a better variant is  to enforce all strings to be encoded
> > in
> >  UTF-8?
> >  AFAIK multi OS cluster is a quite common case.
> > 
> > 
> >  пн, 13 дек. 2021 г. в 11:36, Mikhail Petrov  >:
> > 
> > > Igniters,
> > >
> > > Recently we faced the problem that if the cluster consists of nodes
> > > running in the JVM with different encodings, many issues arise.
> > > The root cause of the mentioned issues is components that use
> > > `String#getBytes()` and `new String()`, which relies on
> > > the
> > > system default encoding. Thus, if a string is deserialized on a
> node
> > > with a different encoding from the one that serialized it, the
> > > deserialized string can be different from the original one.
> > >
> > > For example:
> > >
> > > Serialization/deserialization of string in communication messages
> may
> > > be
> > > broken for some strings on nodes running in a JVM with a different
> > > encoding as DirectByteBufferStreamImplV2 uses String#getBytes() to
> > > serialize strings - [1]
> > >
> > > Or the IgniteAuthenticationProcessor can compute different security
> > > IDs
> > > for the user on different nodes in this case - [2]
> > >
> > > What do you think, if we solve this problem globally, by rejecting
> to
> > > join nodes that run on JVMs with different encodings?
> > >
> > > As a result, we will be sure that all cluster nodes have the same
> > > encoding and all related problems will be solved.
> > >
> > > [1] - https://issues.apache.org/jira/browse/IGNITE-16106
> > > [2] - https://issues.apache.org/jira/browse/IGNITE-16068
> > >
> > > --
> > > Mikhail
> > >
> > >
> > 
> >  --
> >  Sincerely yours, Ivan Daschinskiy
> > >>>
> > >>>
> > >>
> > >> --
> > >> Sincerely yours, Ivan Daschinskiy
> > >>
> > >
> > >
> > > --
> > >
> > > Best regards,
> > > Ivan Pavlukhin
> >
> >
>
> --
> Sincerely yours, Ivan Daschinskiy
>

Re: Ignite Website UI and UX improvements

[Mauricio Stekl]

Hi,
I completed merging the code and the new website is live now.  Also updated
the wiki instructions to build and test the site locally:
https://cwiki.apache.org/confluence/display/IGNITE/Website+Development

Please let me know if you have any questions or comments.

Best,
Mauricio


On Mon, Dec 13, 2021 at 9:31 AM Mauricio Stekl  wrote:

> Hi all,
>
> I'm happy to let you know the new website is ready, and I will start
> merging the code and do the launch today. Please avoid committing new
> changes to the current website until this process is ready.
>
> I will share an update once it's live.
>
> Thanks!
>
> Mauricio
>
> On Tue, Nov 9, 2021 at 9:00 PM Denis Magda  wrote:
>
>> Igniters,
>>
>> I'm pleased to share a piece of good news with you. The next version of
>> the
>> website is ready and deployed on staging:
>>
>> https://test2.heavenweb.ru/
>> username: apache
>> passwd: ignite
>>
>> In the course of the next 5-10 days, I'll be collaborating with the design
>> and dev teams on required improvements. Please pass on any feedback that
>> you might have.
>>
>> -
>> Denis
>>
>>
>> On Mon, Sep 13, 2021 at 8:46 AM Denis Magda  wrote:
>>
>> > Hi Saikat,
>> >
>> > Thanks for the suggestions. I'll pass them over to the design team.
>> >
>> >
>> > -
>> > Denis
>> >
>> >
>> > On Sun, Sep 12, 2021 at 10:43 AM Saikat Maitra > >
>> > wrote:
>> >
>> >> Hi Denis, Kseniya
>> >>
>> >> Thank you for leading the effort in designing and revamping the
>> website.
>> >>
>> >> I wanted to share a few thoughts and ideas:
>> >> 1. Will it be suitable if the rocket exhaust fire/gases are replaced
>> with
>> >> the Ignite spark from the Ignite logo. I really like the red Ignite
>> spark.
>> >> It may symbolize that the rocket's journey is powered / fueled by
>> Ignite.
>> >> 2. Can we rewrite the sentence "We appreciate any help and track every
>> >> contribution" to "We appreciate any help and value every contribution".
>> >> The
>> >> word track felt a bit intrusive like if the user's actions are
>> monitored.
>> >> 3. Similarly here we may replace the word track "Top contributors are
>> >> tracked at Github" to "Top contributors @ Github".
>> >> 4. If it is ok with PMC members and committers then we can use face
>> shots
>> >> for committers and PMC members like here
>> >> https://kafka.apache.org/committers.
>> >> It will be nice to see a face and recognize them in person when we all
>> >> meet
>> >> again in person.
>> >>
>> >> Warm regards,
>> >> Saikat
>> >>
>> >>
>> >>
>> >> On Wed, Sep 8, 2021 at 11:34 AM Denis Magda  wrote:
>> >>
>> >> > Folks, an update for you. We keep prototyping new pages and you can
>> >> find a
>> >> > reference to the mockups in the ticket.
>> >> >
>> >> > Need to hear your opinion on one of the mockups. Kseniya and I worked
>> >> with
>> >> > the design studio on a Community page, and this is what we have
>> >> prepared so
>> >> > far:
>> >> > https://app.moqups.com/4WR00aUZ8z/view/page/a7e1c85f0
>> >> >
>> >> > The page tells a brief story about the project, has a "meet the
>> >> community"
>> >> > section, explains how to start contributing or ask a question. All
>> those
>> >> > sections will have anchors and will be added as separate menu items
>> for
>> >> > quick navigation.
>> >> >
>> >> > Any suggestions before we send the mockup to editors and designers?
>> >> >
>> >> > --
>> >> > Denis
>> >> >
>> >> > -
>> >> > Denis
>> >> >
>> >> > On Mon, Aug 16, 2021 at 4:35 PM Denis Magda 
>> wrote:
>> >> >
>> >> > > Folks,
>> >> > >
>> >> > > In early 2021 a 3rd-party vendor helped us to conduct a usability
>> >> audit
>> >> > of
>> >> > > the Ignite website. The audit revealed significant user experience
>> and
>> >> > > structure-related gaps that make it harder to understand Ignite,
>> >> > navigate,
>> >> > > consume content.
>> >> > >
>> >> > > We've just started bringing those changes to life:
>> >> > > https://issues.apache.org/jira/browse/IGNITE-15320
>> >> > >
>> >> > > I'll keep you posted on the progress and intermediate results. Let
>> me
>> >> > know
>> >> > > if:
>> >> > > a) you have any questions or suggestions
>> >> > > b) you want to be involved in this effort and brief the team (the
>> same
>> >> > > vendor) on the structure/content of the community and other pages.
>> I
>> >> meet
>> >> > > with the folks several days a week to help them finish the project
>> by
>> >> > late
>> >> > > September- early October.
>> >> > >
>> >> > > --
>> >> > > Denis
>> >> > >
>> >> > > -
>> >> > > Denis
>> >> > >
>> >> >
>> >>
>> >
>>
>

Re: [DISCUSSION] Reject join of nodes with different character encodings

[Ivan Pavlukhin]

> I guess Nikolay is talking about the problem with UTF-8 in case string 
> contains unpaired surrogate symbols

Folks, give me a clue why it is a problem? Naively it seems to be a
good restriction rather than problem. What problems can it cause in
practice?

2021-12-13 16:32 GMT+03:00, Ilya Kasnacheev :
> Hello!
>
> We already have a warning about this, see IgniteKernal.checkFileEncoding()
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 13 дек. 2021 г. в 16:26, Ivan Daschinsky :
>
>> >> But now multiple components
>> >> independently serialize strings for their needs and use default
>> >> encoding
>> >> for this.
>> >> For example  DirectByteBufferStreamImplV2#writeString,
>> >> MetaStorage#writeRaw and so on
>> We should fix all of them.
>>
>> >> BinaryUtils#utf8BytesToStr
>> Lets use this everywhere.
>>
>> As for me, I'm expecting a way more problem with enforcing rule to fail,
>> rather than enforcing all components to use UTF-8
>> Some weird cases  (surrogate pairs) we can (I strongly believe it is OK)
>> simply do not consider at all.
>>
>> пн, 13 дек. 2021 г. в 15:15, Nikolay Izhikov :
>>
>> > > Does Java String support all unicode characters and particularly does
>> it
>> > support more characters than UTF-8
>> >
>> > It’s not about Java, it’s about UTF-8 standard.
>> >
>> > Please, take a look at [1]
>> >
>> > > In November 2003, UTF-8 was restricted by RFC 3629 to match the
>> > constraints of the UTF-16 character encoding: explicitly prohibiting
>> > code
>> > points corresponding to the high and low surrogate characters removed
>> more
>> > than 3% of the three-byte sequences, and ending at U+10 removed
>> > more
>> > than 48% of the four-byte sequences and all five- and six-byte
>> > sequences.
>> >
>> > And [2]
>> >
>> > > The definition of UTF-8 prohibits encoding character numbers between
>> > U+D800 and U+DFFF, which are reserved for use with the UTF-16 encoding
>> form
>> > (as surrogate pairs) and do not directly represent characters.
>> >
>> > Actually, we already has some modes to support this restriction of
>> > UTF-8.
>> > Please, take a look at BinaryUtils#utf8BytesToStr [3]
>> >
>> >
>> > [1] https://en.wikipedia.org/wiki/UTF-8
>> > [2] https://datatracker.ietf.org/doc/html/rfc3629
>> > [3]
>> >
>> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/binary/BinaryUtils.java#L2387
>> >
>> > > 13 дек. 2021 г., в 13:57, Ivan Pavlukhin 
>> > написал(а):
>> > >
>> > >> UTF-8 can’t encode all UNICODE characters.
>> > >
>> > > Nikolay, could you please elaborate? My understanding is that
>> > > encoding
>> > > we speak about matters for conversion from byte arrays to strings.
>> > > Does Java String support all unicode characters and particularly does
>> > > it support more characters than UTF-8 (I am not saying here that java
>> > > String uses UTF-8)?
>> > >
>> > > 2021-12-13 12:56 GMT+03:00, Ivan Daschinsky :
>> > >> UTF-8 is already a default encoding in our BinaryObject format.
>> > >> So
>> > I am
>> > >> for unification.
>> > >>
>> > >> пн, 13 дек. 2021 г. в 12:50, Nikolay Izhikov :
>> > >>
>> > >>> Hello, Ivan.
>> > >>>
>> > >>> UTF-8 can’t encode all UNICODE characters.
>> > >>>
>> >  13 дек. 2021 г., в 12:49, Ivan Daschinsky 
>> > >>> написал(а):
>> > 
>> >  Khm, maybe a better variant is  to enforce all strings to be
>> >  encoded
>> > in
>> >  UTF-8?
>> >  AFAIK multi OS cluster is a quite common case.
>> > 
>> > 
>> >  пн, 13 дек. 2021 г. в 11:36, Mikhail Petrov > >:
>> > 
>> > > Igniters,
>> > >
>> > > Recently we faced the problem that if the cluster consists of
>> > > nodes
>> > > running in the JVM with different encodings, many issues arise.
>> > > The root cause of the mentioned issues is components that use
>> > > `String#getBytes()` and `new String()`, which relies
>> > > on
>> > > the
>> > > system default encoding. Thus, if a string is deserialized on a
>> node
>> > > with a different encoding from the one that serialized it, the
>> > > deserialized string can be different from the original one.
>> > >
>> > > For example:
>> > >
>> > > Serialization/deserialization of string in communication messages
>> may
>> > > be
>> > > broken for some strings on nodes running in a JVM with a
>> > > different
>> > > encoding as DirectByteBufferStreamImplV2 uses String#getBytes()
>> > > to
>> > > serialize strings - [1]
>> > >
>> > > Or the IgniteAuthenticationProcessor can compute different
>> > > security
>> > > IDs
>> > > for the user on different nodes in this case - [2]
>> > >
>> > > What do you think, if we solve this problem globally, by
>> > > rejecting
>> to
>> > > join nodes that run on JVMs with different encodings?
>> > >
>> > > As a result, we will be sure that all cluster nodes have the same
>> > > encoding and all related problems will be so

Re: [DISCUSSION] Reject join of nodes with different character encodings

[Mikhail Petrov]

Ivan, string with unpaired surrogates symbols are serialized and 
deserialized by java UTF-8 decoder successfully but the result does not 
match the initial string. It may result in that if the user's login 
contains these symbols, it will be distorted after deserialization and 
the user will not be able to log in. I understand that it is a quite 
rare case.
Anyway, the way to solve this problem was introduced here - 
https://issues.apache.org/jira/browse/IGNITE-3098


Frankly, it is not the topic I would like to discuss now. The main 
question is - should we restrict the join of nodes with different 
encodings or just fix all places where implicit default encoding is used 
and specify the explicit one as Ivan Daschinsky suggested?


From my point of view, it is better to reject nodes with different 
encodings (especially after Ilya Kasnacheev mentioned that we already 
have a warning  "Differing character encodings across cluster may lead 
to erratic behavior"). It will help to avoid "erratic behavior", not 
just warn about it. It is important since the problems related to string 
encoding can occur in different components and the cause of them is not 
always obvious.


WDYT?

On 13.12.2021 20:01, Ivan Pavlukhin wrote:

I guess Nikolay is talking about the problem with UTF-8 in case string contains 
unpaired surrogate symbols

Folks, give me a clue why it is a problem? Naively it seems to be a
good restriction rather than problem. What problems can it cause in
practice?

2021-12-13 16:32 GMT+03:00, Ilya Kasnacheev :

Hello!

We already have a warning about this, see IgniteKernal.checkFileEncoding()

Regards,
--
Ilya Kasnacheev


пн, 13 дек. 2021 г. в 16:26, Ivan Daschinsky :


But now multiple components
independently serialize strings for their needs and use default
encoding
for this.
For example  DirectByteBufferStreamImplV2#writeString,
MetaStorage#writeRaw and so on

We should fix all of them.


BinaryUtils#utf8BytesToStr

Lets use this everywhere.

As for me, I'm expecting a way more problem with enforcing rule to fail,
rather than enforcing all components to use UTF-8
Some weird cases  (surrogate pairs) we can (I strongly believe it is OK)
simply do not consider at all.

пн, 13 дек. 2021 г. в 15:15, Nikolay Izhikov :


Does Java String support all unicode characters and particularly does

it

support more characters than UTF-8

It’s not about Java, it’s about UTF-8 standard.

Please, take a look at [1]


In November 2003, UTF-8 was restricted by RFC 3629 to match the

constraints of the UTF-16 character encoding: explicitly prohibiting
code
points corresponding to the high and low surrogate characters removed

more

than 3% of the three-byte sequences, and ending at U+10 removed
more
than 48% of the four-byte sequences and all five- and six-byte
sequences.

And [2]


The definition of UTF-8 prohibits encoding character numbers between

U+D800 and U+DFFF, which are reserved for use with the UTF-16 encoding

form

(as surrogate pairs) and do not directly represent characters.

Actually, we already has some modes to support this restriction of
UTF-8.
Please, take a look at BinaryUtils#utf8BytesToStr [3]


[1] https://en.wikipedia.org/wiki/UTF-8
[2] https://datatracker.ietf.org/doc/html/rfc3629
[3]


https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/binary/BinaryUtils.java#L2387

13 дек. 2021 г., в 13:57, Ivan Pavlukhin 

написал(а):

UTF-8 can’t encode all UNICODE characters.

Nikolay, could you please elaborate? My understanding is that
encoding
we speak about matters for conversion from byte arrays to strings.
Does Java String support all unicode characters and particularly does
it support more characters than UTF-8 (I am not saying here that java
String uses UTF-8)?

2021-12-13 12:56 GMT+03:00, Ivan Daschinsky :

UTF-8 is already a default encoding in our BinaryObject format.
So

I am

for unification.

пн, 13 дек. 2021 г. в 12:50, Nikolay Izhikov :


Hello, Ivan.

UTF-8 can’t encode all UNICODE characters.


13 дек. 2021 г., в 12:49, Ivan Daschinsky 

написал(а):

Khm, maybe a better variant is  to enforce all strings to be
encoded

in

UTF-8?
AFAIK multi OS cluster is a quite common case.


пн, 13 дек. 2021 г. в 11:36, Mikhail Petrov 
:

Igniters,

Recently we faced the problem that if the cluster consists of
nodes
running in the JVM with different encodings, many issues arise.
The root cause of the mentioned issues is components that use
`String#getBytes()` and `new String()`, which relies
on
the
system default encoding. Thus, if a string is deserialized on a

node

with a different encoding from the one that serialized it, the
deserialized string can be different from the original one.

For example:

Serialization/deserialization of string in communication messages

may

be
broken for some strings on nodes running in a JVM with a
different
encoding as DirectByteBufferStreamImplV2 uses String#getBytes()
to
serialize strings - [1]

Or the Ig

New Ignite Website: Released and Check the Dev Instructions

[Denis Magda]

Igniters,

Please welcome the new Ignite website: https://ignite.apache.org
Hope you like it!

The website uses gulpjs to simplify the dev/maintenance cycles, thus, I
would encourage all of you who release Ignite (and update the downloads
page) or edit any other pages to check the updated instruction:
https://cwiki.apache.org/confluence/display/IGNITE/Website+Development


-
Denis

Re: [DISCUSSION] Reject join of nodes with different character encodings

[Ivan Daschinsky]

Unpaited surrogates are emoji symbols. One should be completely insane to
use emojis in login.

пн, 13 дек. 2021 г., 21:30 Mikhail Petrov :

> Ivan, string with unpaired surrogates symbols are serialized and
> deserialized by java UTF-8 decoder successfully but the result does not
> match the initial string. It may result in that if the user's login
> contains these symbols, it will be distorted after deserialization and
> the user will not be able to log in. I understand that it is a quite
> rare case.
> Anyway, the way to solve this problem was introduced here -
> https://issues.apache.org/jira/browse/IGNITE-3098
>
> Frankly, it is not the topic I would like to discuss now. The main
> question is - should we restrict the join of nodes with different
> encodings or just fix all places where implicit default encoding is used
> and specify the explicit one as Ivan Daschinsky suggested?
>
>  From my point of view, it is better to reject nodes with different
> encodings (especially after Ilya Kasnacheev mentioned that we already
> have a warning  "Differing character encodings across cluster may lead
> to erratic behavior"). It will help to avoid "erratic behavior", not
> just warn about it. It is important since the problems related to string
> encoding can occur in different components and the cause of them is not
> always obvious.
>
> WDYT?
>
> On 13.12.2021 20:01, Ivan Pavlukhin wrote:
> >> I guess Nikolay is talking about the problem with UTF-8 in case string
> contains unpaired surrogate symbols
> > Folks, give me a clue why it is a problem? Naively it seems to be a
> > good restriction rather than problem. What problems can it cause in
> > practice?
> >
> > 2021-12-13 16:32 GMT+03:00, Ilya Kasnacheev :
> >> Hello!
> >>
> >> We already have a warning about this, see
> IgniteKernal.checkFileEncoding()
> >>
> >> Regards,
> >> --
> >> Ilya Kasnacheev
> >>
> >>
> >> пн, 13 дек. 2021 г. в 16:26, Ivan Daschinsky :
> >>
> > But now multiple components
> > independently serialize strings for their needs and use default
> > encoding
> > for this.
> > For example  DirectByteBufferStreamImplV2#writeString,
> > MetaStorage#writeRaw and so on
> >>> We should fix all of them.
> >>>
> > BinaryUtils#utf8BytesToStr
> >>> Lets use this everywhere.
> >>>
> >>> As for me, I'm expecting a way more problem with enforcing rule to
> fail,
> >>> rather than enforcing all components to use UTF-8
> >>> Some weird cases  (surrogate pairs) we can (I strongly believe it is
> OK)
> >>> simply do not consider at all.
> >>>
> >>> пн, 13 дек. 2021 г. в 15:15, Nikolay Izhikov :
> >>>
> > Does Java String support all unicode characters and particularly does
> >>> it
>  support more characters than UTF-8
> 
>  It’s not about Java, it’s about UTF-8 standard.
> 
>  Please, take a look at [1]
> 
> > In November 2003, UTF-8 was restricted by RFC 3629 to match the
>  constraints of the UTF-16 character encoding: explicitly prohibiting
>  code
>  points corresponding to the high and low surrogate characters removed
> >>> more
>  than 3% of the three-byte sequences, and ending at U+10 removed
>  more
>  than 48% of the four-byte sequences and all five- and six-byte
>  sequences.
> 
>  And [2]
> 
> > The definition of UTF-8 prohibits encoding character numbers between
>  U+D800 and U+DFFF, which are reserved for use with the UTF-16 encoding
> >>> form
>  (as surrogate pairs) and do not directly represent characters.
> 
>  Actually, we already has some modes to support this restriction of
>  UTF-8.
>  Please, take a look at BinaryUtils#utf8BytesToStr [3]
> 
> 
>  [1] https://en.wikipedia.org/wiki/UTF-8
>  [2] https://datatracker.ietf.org/doc/html/rfc3629
>  [3]
> 
> >>>
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/binary/BinaryUtils.java#L2387
> > 13 дек. 2021 г., в 13:57, Ivan Pavlukhin 
>  написал(а):
> >> UTF-8 can’t encode all UNICODE characters.
> > Nikolay, could you please elaborate? My understanding is that
> > encoding
> > we speak about matters for conversion from byte arrays to strings.
> > Does Java String support all unicode characters and particularly does
> > it support more characters than UTF-8 (I am not saying here that java
> > String uses UTF-8)?
> >
> > 2021-12-13 12:56 GMT+03:00, Ivan Daschinsky :
> >> UTF-8 is already a default encoding in our BinaryObject format.
> >> So
>  I am
> >> for unification.
> >>
> >> пн, 13 дек. 2021 г. в 12:50, Nikolay Izhikov :
> >>
> >>> Hello, Ivan.
> >>>
> >>> UTF-8 can’t encode all UNICODE characters.
> >>>
>  13 дек. 2021 г., в 12:49, Ivan Daschinsky 
> >>> написал(а):
>  Khm, maybe a better variant is  to enforce all strings to be
>  encoded
>  in
>  UT

Re: New Ignite Website: Released and Check the Dev Instructions

[Mauricio Stekl]

Hi,
I noticed the "Extensions" section was not merged correctly into the
Downloads page. I have created a ticket
https://issues.apache.org/jira/browse/IGNITE-16112, and we'll be solving
this as soon as possible. My apologies for the inconvenience

Best,
Mauricio

On Mon, Dec 13, 2021 at 4:46 PM Denis Magda  wrote:

> Igniters,
>
> Please welcome the new Ignite website: https://ignite.apache.org
> Hope you like it!
>
> The website uses gulpjs to simplify the dev/maintenance cycles, thus, I
> would encourage all of you who release Ignite (and update the downloads
> page) or edit any other pages to check the updated instruction:
> https://cwiki.apache.org/confluence/display/IGNITE/Website+Development
>
>
> -
> Denis
>

Re: Apache Ignite 2.12 RELEASE [Time, Scope, Manager]

[Nikita Amelchev]

+1 to move these auth issues to the next release. I will prepare RC at
the nearest time.

пн, 13 дек. 2021 г. в 16:13, Pavel Tupitsyn :
>
> Agree with Nikolay, let's proceed with the release.
>
> On Mon, Dec 13, 2021 at 3:31 PM Nikolay Izhikov  wrote:
>
> > Nikita.
> >
> > I don’t think IGNITE-16068 is a release blocker.
> >
> > 1. It only happens when authentication enabled.
> > 2. There is at lease 2 workarounds:
> > a. Use the same file.encoding for all Ignite nodes.
> > b. Use only latin characters in user login.
> >
> > I propose to postpone this ticket and move on with the release.
> >
> > > 13 дек. 2021 г., в 15:28, Nikita Amelchev 
> > написал(а):
> > >
> > > Hello.
> > >
> > > I have cherry-picked the blocked [1] to the 2.12. It updates the log4j
> > > version to 2.15.
> > >
> > > The release is blocked by one blocker issue with auth.
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-16101
> > > [2] https://issues.apache.org/jira/browse/IGNITE-16068
> > >
> > > чт, 9 дек. 2021 г. в 12:44, Nikita Amelchev :
> > >>
> > >> Hello, Nikolay.
> > >>
> > >> I have cherry-picked the issue fix.
> > >>
> > >> Thank you.
> > >>
> > >> ср, 8 дек. 2021 г. в 18:47, Nikolay Izhikov :
> > >>>
> > >>> Hello.
> > >>>
> > >>> Let’s include [1] in 2.12 scope.
> > >>>
> > >>> After migrations authentication processor to security API we have an
> > issue.
> > >>> Persistent data region should exists on client node if authentication
> > is enabled.
> > >>>
> > >>> It seems very annoying for the users.
> > >>>
> > >>> [1] https://issues.apache.org/jira/browse/IGNITE-15969
> > >>>
> >  2 дек. 2021 г., в 19:50, Nikita Amelchev 
> > написал(а):
> > 
> >  I would like to formally announce code freeze for 2.12.0. Only
> >  blockers are accepted to be included to the scope. [1]
> > 
> >  We have one blocker issue [2] that will be fixed soon.
> > 
> >  [1]
> > https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P3.Stabilization
> >  [2] https://issues.apache.org/jira/browse/IGNITE-15966
> > 
> >  чт, 2 дек. 2021 г. в 16:01, Nikita Amelchev :
> > >
> > > Hi, Igniters.
> > >
> > > The release is blocked by the auth issue [1] discussed above. The
> > > patch will be prepared at the nearest time.
> > >
> > > I want to include the useful issue that adds the ability to configure
> > > snapshot thread pool size if nobody minds.
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-15966
> > > [2] https://issues.apache.org/jira/browse/IGNITE-16014
> > >
> > > пт, 26 нояб. 2021 г. в 11:44, Nikita Amelchev  > >:
> > >>
> > >> Hello,
> > >>
> > >> I want to include the issue [1] to the 2.12 scope. It fixes some
> > >> metrics according to the JSR107.
> > >>
> > >> Any objections?
> > >>
> > >> [1] https://issues.apache.org/jira/browse/IGNITE-16002 The remove
> > >> cache method should update statistics if the method returns true
> > >>
> > >> чт, 25 нояб. 2021 г. в 17:12, Nikita Amelchev  > >:
> > >>>
> > >>> Hello Ivan,
> > >>>
> > >>> +1, the issue can be cherry-picked to the 2.12.
> > >>>
> > >>> чт, 25 нояб. 2021 г. в 17:07, Ivan Bessonov  > >:
> > 
> >  Hello everyone,
> > 
> >  is there a chance to add this issue to the release scope? [1]
> >  Currently, defragmentation of certain types of indexes can
> > corrupt them.
> >  Fix is straightforward and should not cause any problems.
> >  Thank you!
> > 
> >  [1] https://issues.apache.org/jira/browse/IGNITE-15968
> > 
> >  вт, 23 нояб. 2021 г. в 20:44, Nikita Amelchev <
> > namelc...@apache.org>:
> > 
> > > Ivan,
> > >
> > > I have cherry-picked the issue:
> > > https://issues.apache.org/jira/browse/IGNITE-15767
> > >
> > > вт, 23 нояб. 2021 г. в 20:31, Nikolay Izhikov <
> > nizhi...@apache.org>:
> > >>
> > >>> [1] https://issues.apache.org/jira/browse/IGNITE-15951
> > >>
> > >> Cherry-picked to 2.12
> > >>
> > >>> 23 нояб. 2021 г., в 16:38, Nikita Amelchev <
> > namelc...@apache.org>
> > > написал(а):
> > >>>
> > >>> Hi, +1 to cherry-pick.
> > >>>
> > >>> вт, 23 нояб. 2021 г. в 15:25, Ivan Daschinsky <
> > ivanda...@gmail.com>:
> > 
> >  Hi! Lets add one simple bug fix, but very useful
> > 
> >  https://issues.apache.org/jira/browse/IGNITE-15767
> > 
> >  пн, 22 нояб. 2021 г. в 15:22, Pavel Tupitsyn <
> > ptupit...@apache.org>:
> > 
> > > Yes, let's fix those auth issues, there are so many
> > complaints on
> > > the user
> > > list.
> > >
> > > On Mon, Nov 22, 2021 at 1:03 PM Mikhail Petrov <

Re: New Ignite Website: Released and Check the Dev Instructions

[Erlan Aytpaev]

Hi Igniters,

I've added the "Extensions" section on the "Downloads" page. Please review
my pull request https://github.com/apache/ignite-website/pull/91.
This is my first commit. And I'm going to continue contributing updates on
the website.

Best Regards,
Erlan.

On Tue, Dec 14, 2021 at 9:55 AM Mauricio Stekl 
wrote:

> Hi,
> I noticed the "Extensions" section was not merged correctly into the
> Downloads page. I have created a ticket
> https://issues.apache.org/jira/browse/IGNITE-16112, and we'll be solving
> this as soon as possible. My apologies for the inconvenience
>
> Best,
> Mauricio
>
> On Mon, Dec 13, 2021 at 4:46 PM Denis Magda  wrote:
>
> > Igniters,
> >
> > Please welcome the new Ignite website: https://ignite.apache.org
> > Hope you like it!
> >
> > The website uses gulpjs to simplify the dev/maintenance cycles, thus, I
> > would encourage all of you who release Ignite (and update the downloads
> > page) or edit any other pages to check the updated instruction:
> > https://cwiki.apache.org/confluence/display/IGNITE/Website+Development
> >
> >
> > -
> > Denis
> >
>

Re: [DISCUSSION] Reject join of nodes with different character encodings

[Ivan Pavlukhin]

Do encodings in question somehow influence on actual stored data
(bytes)? If so, using an implicit platform encoding sounds quite
dangerous. Moving data between servers (or perhaps even rebalancing)
can lead to bad consequences. Anyways, IMHO an implicit encoding is
not good, but sensible default is quite robust.

2021-12-13 23:07 GMT+03:00, Ivan Daschinsky :
> Unpaited surrogates are emoji symbols. One should be completely insane to
> use emojis in login.
>
> пн, 13 дек. 2021 г., 21:30 Mikhail Petrov :
>
>> Ivan, string with unpaired surrogates symbols are serialized and
>> deserialized by java UTF-8 decoder successfully but the result does not
>> match the initial string. It may result in that if the user's login
>> contains these symbols, it will be distorted after deserialization and
>> the user will not be able to log in. I understand that it is a quite
>> rare case.
>> Anyway, the way to solve this problem was introduced here -
>> https://issues.apache.org/jira/browse/IGNITE-3098
>>
>> Frankly, it is not the topic I would like to discuss now. The main
>> question is - should we restrict the join of nodes with different
>> encodings or just fix all places where implicit default encoding is used
>> and specify the explicit one as Ivan Daschinsky suggested?
>>
>>  From my point of view, it is better to reject nodes with different
>> encodings (especially after Ilya Kasnacheev mentioned that we already
>> have a warning  "Differing character encodings across cluster may lead
>> to erratic behavior"). It will help to avoid "erratic behavior", not
>> just warn about it. It is important since the problems related to string
>> encoding can occur in different components and the cause of them is not
>> always obvious.
>>
>> WDYT?
>>
>> On 13.12.2021 20:01, Ivan Pavlukhin wrote:
>> >> I guess Nikolay is talking about the problem with UTF-8 in case string
>> contains unpaired surrogate symbols
>> > Folks, give me a clue why it is a problem? Naively it seems to be a
>> > good restriction rather than problem. What problems can it cause in
>> > practice?
>> >
>> > 2021-12-13 16:32 GMT+03:00, Ilya Kasnacheev
>> > :
>> >> Hello!
>> >>
>> >> We already have a warning about this, see
>> IgniteKernal.checkFileEncoding()
>> >>
>> >> Regards,
>> >> --
>> >> Ilya Kasnacheev
>> >>
>> >>
>> >> пн, 13 дек. 2021 г. в 16:26, Ivan Daschinsky :
>> >>
>> > But now multiple components
>> > independently serialize strings for their needs and use default
>> > encoding
>> > for this.
>> > For example  DirectByteBufferStreamImplV2#writeString,
>> > MetaStorage#writeRaw and so on
>> >>> We should fix all of them.
>> >>>
>> > BinaryUtils#utf8BytesToStr
>> >>> Lets use this everywhere.
>> >>>
>> >>> As for me, I'm expecting a way more problem with enforcing rule to
>> fail,
>> >>> rather than enforcing all components to use UTF-8
>> >>> Some weird cases  (surrogate pairs) we can (I strongly believe it is
>> OK)
>> >>> simply do not consider at all.
>> >>>
>> >>> пн, 13 дек. 2021 г. в 15:15, Nikolay Izhikov :
>> >>>
>> > Does Java String support all unicode characters and particularly
>> > does
>> >>> it
>>  support more characters than UTF-8
>> 
>>  It’s not about Java, it’s about UTF-8 standard.
>> 
>>  Please, take a look at [1]
>> 
>> > In November 2003, UTF-8 was restricted by RFC 3629 to match the
>>  constraints of the UTF-16 character encoding: explicitly prohibiting
>>  code
>>  points corresponding to the high and low surrogate characters
>>  removed
>> >>> more
>>  than 3% of the three-byte sequences, and ending at U+10 removed
>>  more
>>  than 48% of the four-byte sequences and all five- and six-byte
>>  sequences.
>> 
>>  And [2]
>> 
>> > The definition of UTF-8 prohibits encoding character numbers
>> > between
>>  U+D800 and U+DFFF, which are reserved for use with the UTF-16
>>  encoding
>> >>> form
>>  (as surrogate pairs) and do not directly represent characters.
>> 
>>  Actually, we already has some modes to support this restriction of
>>  UTF-8.
>>  Please, take a look at BinaryUtils#utf8BytesToStr [3]
>> 
>> 
>>  [1] https://en.wikipedia.org/wiki/UTF-8
>>  [2] https://datatracker.ietf.org/doc/html/rfc3629
>>  [3]
>> 
>> >>>
>> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/binary/BinaryUtils.java#L2387
>> > 13 дек. 2021 г., в 13:57, Ivan Pavlukhin 
>>  написал(а):
>> >> UTF-8 can’t encode all UNICODE characters.
>> > Nikolay, could you please elaborate? My understanding is that
>> > encoding
>> > we speak about matters for conversion from byte arrays to strings.
>> > Does Java String support all unicode characters and particularly
>> > does
>> > it support more characters than UTF-8 (I am not saying here that
>> > java
>> > String uses UTF-8)?
>> >
>> >