[jira] [Created] (IGNITE-20373) Fix IO threading model
Alexey Scherbakov created IGNITE-20373: -- Summary: Fix IO threading model Key: IGNITE-20373 URL: https://issues.apache.org/jira/browse/IGNITE-20373 Project: Ignite Issue Type: Improvement Affects Versions: 3.0 Reporter: Alexey Scherbakov Fix For: 3.0 Currently IO is resubmitted to inbound pool for further processing (even there are corner cases then a message handler called in IO thread) It introduces additional latency in message processing because each component other uses it's own pools. addMessageHandler should be extended with a 3-d argument: a pool for submitting a callback for execution. Some kind of message header can be passed along the way for precise pool selection (similar to jraft IO model) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-20373) Fix IO threading model
[ https://issues.apache.org/jira/browse/IGNITE-20373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Scherbakov updated IGNITE-20373: --- Labels: ignite-3 ignite3_performance (was: ) > Fix IO threading model > -- > > Key: IGNITE-20373 > URL: https://issues.apache.org/jira/browse/IGNITE-20373 > Project: Ignite > Issue Type: Improvement >Affects Versions: 3.0 >Reporter: Alexey Scherbakov >Priority: Major > Labels: ignite-3, ignite3_performance > Fix For: 3.0 > > > Currently IO is resubmitted to inbound pool for further processing (even > there are corner cases then a message handler called in IO thread) > It introduces additional latency in message processing because each component > other uses it's own pools. > addMessageHandler should be extended with a 3-d argument: a pool for > submitting a callback for execution. > Some kind of message header can be passed along the way for precise pool > selection (similar to jraft IO model) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-20373) Fix IO threading model
[ https://issues.apache.org/jira/browse/IGNITE-20373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Scherbakov updated IGNITE-20373: --- Description: Currently IO is resubmitted to inbound pool for further processing (even there are corner cases then a message handler called in IO thread) It introduces additional latency in message processing because each component other uses it's own pools. addMessageHandler should be extended with a 3-d argument: a pool for submitting a callback for execution. Some kind of message header can be passed along the way for precise pool selection (similar to jraft IO model) inbound pool should be removed was: Currently IO is resubmitted to inbound pool for further processing (even there are corner cases then a message handler called in IO thread) It introduces additional latency in message processing because each component other uses it's own pools. addMessageHandler should be extended with a 3-d argument: a pool for submitting a callback for execution. Some kind of message header can be passed along the way for precise pool selection (similar to jraft IO model) > Fix IO threading model > -- > > Key: IGNITE-20373 > URL: https://issues.apache.org/jira/browse/IGNITE-20373 > Project: Ignite > Issue Type: Improvement >Affects Versions: 3.0 >Reporter: Alexey Scherbakov >Priority: Major > Labels: ignite-3, ignite3_performance > Fix For: 3.0 > > > Currently IO is resubmitted to inbound pool for further processing (even > there are corner cases then a message handler called in IO thread) > It introduces additional latency in message processing because each component > other uses it's own pools. > addMessageHandler should be extended with a 3-d argument: a pool for > submitting a callback for execution. > Some kind of message header can be passed along the way for precise pool > selection (similar to jraft IO model) > inbound pool should be removed -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-14818) Implement compression of short primitives
[
https://issues.apache.org/jira/browse/IGNITE-14818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17762623#comment-17762623
]
Roman Puchkovskiy commented on IGNITE-14818:
The patch looks good to me
> Implement compression of short primitives
> -
>
> Key: IGNITE-14818
> URL: https://issues.apache.org/jira/browse/IGNITE-14818
> Project: Ignite
> Issue Type: Improvement
> Components: networking
>Reporter: Aleksandr Polovtcev
>Assignee: Ivan Bessonov
>Priority: Minor
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> Network message type consists of two {{short}} values: the message group type
> and the message type. During serialization, these values are written as
> non-compressed (2 bytes each), which can be improved to reduce the network
> pressure by compressing these {{short}} values the same way we currently do
> for {{int}} and {{long}} (see {{DirectByteBufferStreamImplV1}} for details)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IGNITE-20374) ItFileTransferTest fails on Windows
Ivan Gagarkin created IGNITE-20374:
--
Summary: ItFileTransferTest fails on Windows
Key: IGNITE-20374
URL: https://issues.apache.org/jira/browse/IGNITE-20374
Project: Ignite
Issue Type: Bug
Reporter: Ivan Gagarkin
Some tests in the scope of the ItFileTransferTest
[downloadFilesWhenDoNotHaveAccessToWrite, downloadNonReadableFiles,
uploadFilesWhenDoNotHaveAccessToWrite, uploadNonReadableFiles] fail with the
following error:
{{}}
{code:java}
java.lang.AssertionError: Expected: a future that completes with an exception
that is an instance of
org.apache.ignite.internal.network.file.exception.FileTransferException and
error message that contains a string containing "Failed to create a file
transfer stream" but: was completed successfully java.lang.AssertionError:
Expected: a future that completes with an exception that is an instance of
org.apache.ignite.internal.network.file.exception.FileTransferException and
error message that contains a string containing "Failed to create a file
transfer stream" but: was completed successfully{code}
{{}}
The reason of these failures on the Windows platform can be caused by the fact
that File.setReadable() method may not work as is.
{quote}{{A convenience method to set the owner's read permission for this
abstract pathname.
On some platforms it may be possible to start the Java virtual machine with
special privileges
that allow it to read files that are marked as unreadable.}}
{quote}
I think these tests can be disabled on Windows using @DisabledOnOs(OS.WINDOWS)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-20374) ItFileTransferTest fails on Windows
[
https://issues.apache.org/jira/browse/IGNITE-20374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Gagarkin updated IGNITE-20374:
---
Description:
Some tests in the scope of the ItFileTransferTest
[downloadFilesWhenDoNotHaveAccessToWrite, downloadNonReadableFiles,
uploadFilesWhenDoNotHaveAccessToWrite, uploadNonReadableFiles] fail with the
following error:
{code:java}
java.lang.AssertionError: Expected: a future that completes with an exception
that is an instance of
org.apache.ignite.internal.network.file.exception.FileTransferException and
error message that contains a string containing "Failed to create a file
transfer stream" but: was completed successfully java.lang.AssertionError:
Expected: a future that completes with an exception that is an instance of
org.apache.ignite.internal.network.file.exception.FileTransferException and
error message that contains a string containing "Failed to create a file
transfer stream" but: was completed successfully{code}
The reason of these failures on the Windows platform can be caused by the fact
that File.setReadable() method may not work as is.
{quote}A convenience method to set the owner's read permission for this
abstract pathname.
On some platforms it may be possible to start the Java virtual machine with
special privileges
that allow it to read files that are marked as unreadable.
{quote}
I think these tests can be disabled on Windows using @DisabledOnOs(OS.WINDOWS)
was:
Some tests in the scope of the ItFileTransferTest
[downloadFilesWhenDoNotHaveAccessToWrite, downloadNonReadableFiles,
uploadFilesWhenDoNotHaveAccessToWrite, uploadNonReadableFiles] fail with the
following error:
{{}}
{code:java}
java.lang.AssertionError: Expected: a future that completes with an exception
that is an instance of
org.apache.ignite.internal.network.file.exception.FileTransferException and
error message that contains a string containing "Failed to create a file
transfer stream" but: was completed successfully java.lang.AssertionError:
Expected: a future that completes with an exception that is an instance of
org.apache.ignite.internal.network.file.exception.FileTransferException and
error message that contains a string containing "Failed to create a file
transfer stream" but: was completed successfully{code}
{{}}
The reason of these failures on the Windows platform can be caused by the fact
that File.setReadable() method may not work as is.
{quote}{{A convenience method to set the owner's read permission for this
abstract pathname.
On some platforms it may be possible to start the Java virtual machine with
special privileges
that allow it to read files that are marked as unreadable.}}
{quote}
I think these tests can be disabled on Windows using @DisabledOnOs(OS.WINDOWS)
> ItFileTransferTest fails on Windows
> ---
>
> Key: IGNITE-20374
> URL: https://issues.apache.org/jira/browse/IGNITE-20374
> Project: Ignite
> Issue Type: Bug
>Reporter: Ivan Gagarkin
>Priority: Major
> Labels: ignite-3
>
> Some tests in the scope of the ItFileTransferTest
> [downloadFilesWhenDoNotHaveAccessToWrite, downloadNonReadableFiles,
> uploadFilesWhenDoNotHaveAccessToWrite, uploadNonReadableFiles] fail with the
> following error:
> {code:java}
> java.lang.AssertionError: Expected: a future that completes with an exception
> that is an instance of
> org.apache.ignite.internal.network.file.exception.FileTransferException and
> error message that contains a string containing "Failed to create a file
> transfer stream" but: was completed successfully java.lang.AssertionError:
> Expected: a future that completes with an exception that is an instance of
> org.apache.ignite.internal.network.file.exception.FileTransferException and
> error message that contains a string containing "Failed to create a file
> transfer stream" but: was completed successfully{code}
> The reason of these failures on the Windows platform can be caused by the
> fact that File.setReadable() method may not work as is.
> {quote}A convenience method to set the owner's read permission for this
> abstract pathname.
> On some platforms it may be possible to start the Java virtual machine with
> special privileges
> that allow it to read files that are marked as unreadable.
> {quote}
> I think these tests can be disabled on Windows using @DisabledOnOs(OS.WINDOWS)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IGNITE-20375) Thin 3.0: Review packArrayHeader usage
Pavel Tupitsyn created IGNITE-20375:
---
Summary: Thin 3.0: Review packArrayHeader usage
Key: IGNITE-20375
URL: https://issues.apache.org/jira/browse/IGNITE-20375
Project: Ignite
Issue Type: Improvement
Components: thin client
Affects Versions: 3.0.0-beta1
Reporter: Pavel Tupitsyn
Assignee: Pavel Tupitsyn
Fix For: 3.0.0-beta2
Ignite 3.x client protocol uses MsgPack for message serialization. In some
places we use *packArrayHeader* incorrectly, for example,
[ClientCompute.packJob|https://github.com/apache/ignite-3/blob/fa8d626f57e3497289c0aaa1bfa8efd19f1042f7/modules/client/src/main/java/org/apache/ignite/internal/client/compute/ClientCompute.java#L330]:
{code:java}
w.packArrayHeader(units.size());
for (DeploymentUnit unit : units) {
w.packString(unit.name());
w.packString(unit.version().render());
}
{code}
We pack 2N items for an array of size N, and in some cases those items can be
of different type.
This creates issues with some MsgPack implemetations, such as the one used in
C++ client.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-20375) Thin 3.0: Review packArrayHeader usage
[
https://issues.apache.org/jira/browse/IGNITE-20375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pavel Tupitsyn updated IGNITE-20375:
Description:
Ignite 3.x client protocol uses MsgPack for message serialization. In some
places we use *packArrayHeader* incorrectly, for example,
[ClientCompute.packJob|https://github.com/apache/ignite-3/blob/fa8d626f57e3497289c0aaa1bfa8efd19f1042f7/modules/client/src/main/java/org/apache/ignite/internal/client/compute/ClientCompute.java#L330]:
{code:java}
w.packArrayHeader(units.size());
for (DeploymentUnit unit : units) {
w.packString(unit.name());
w.packString(unit.version().render());
}
{code}
We pack 2N items for an array of size N, and in some cases those items can be
of different type, which breaks the rules of MsgPack protocol.
This creates issues with some MsgPack implemetations, such as the one used in
C++ client (msgpack-c).
was:
Ignite 3.x client protocol uses MsgPack for message serialization. In some
places we use *packArrayHeader* incorrectly, for example,
[ClientCompute.packJob|https://github.com/apache/ignite-3/blob/fa8d626f57e3497289c0aaa1bfa8efd19f1042f7/modules/client/src/main/java/org/apache/ignite/internal/client/compute/ClientCompute.java#L330]:
{code:java}
w.packArrayHeader(units.size());
for (DeploymentUnit unit : units) {
w.packString(unit.name());
w.packString(unit.version().render());
}
{code}
We pack 2N items for an array of size N, and in some cases those items can be
of different type.
This creates issues with some MsgPack implemetations, such as the one used in
C++ client.
> Thin 3.0: Review packArrayHeader usage
> --
>
> Key: IGNITE-20375
> URL: https://issues.apache.org/jira/browse/IGNITE-20375
> Project: Ignite
> Issue Type: Improvement
> Components: thin client
>Affects Versions: 3.0.0-beta1
>Reporter: Pavel Tupitsyn
>Assignee: Pavel Tupitsyn
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> Ignite 3.x client protocol uses MsgPack for message serialization. In some
> places we use *packArrayHeader* incorrectly, for example,
> [ClientCompute.packJob|https://github.com/apache/ignite-3/blob/fa8d626f57e3497289c0aaa1bfa8efd19f1042f7/modules/client/src/main/java/org/apache/ignite/internal/client/compute/ClientCompute.java#L330]:
> {code:java}
> w.packArrayHeader(units.size());
> for (DeploymentUnit unit : units) {
> w.packString(unit.name());
> w.packString(unit.version().render());
> }
> {code}
> We pack 2N items for an array of size N, and in some cases those items can be
> of different type, which breaks the rules of MsgPack protocol.
> This creates issues with some MsgPack implemetations, such as the one used in
> C++ client (msgpack-c).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (IGNITE-20374) ItFileTransferTest fails on Windows
[
https://issues.apache.org/jira/browse/IGNITE-20374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Gagarkin reassigned IGNITE-20374:
--
Assignee: Ivan Gagarkin
> ItFileTransferTest fails on Windows
> ---
>
> Key: IGNITE-20374
> URL: https://issues.apache.org/jira/browse/IGNITE-20374
> Project: Ignite
> Issue Type: Bug
>Reporter: Ivan Gagarkin
>Assignee: Ivan Gagarkin
>Priority: Major
> Labels: ignite-3
>
> Some tests in the scope of the ItFileTransferTest
> [downloadFilesWhenDoNotHaveAccessToWrite, downloadNonReadableFiles,
> uploadFilesWhenDoNotHaveAccessToWrite, uploadNonReadableFiles] fail with the
> following error:
> {code:java}
> java.lang.AssertionError: Expected: a future that completes with an exception
> that is an instance of
> org.apache.ignite.internal.network.file.exception.FileTransferException and
> error message that contains a string containing "Failed to create a file
> transfer stream" but: was completed successfully java.lang.AssertionError:
> Expected: a future that completes with an exception that is an instance of
> org.apache.ignite.internal.network.file.exception.FileTransferException and
> error message that contains a string containing "Failed to create a file
> transfer stream" but: was completed successfully{code}
> The reason of these failures on the Windows platform can be caused by the
> fact that File.setReadable() method may not work as is.
> {quote}A convenience method to set the owner's read permission for this
> abstract pathname.
> On some platforms it may be possible to start the Java virtual machine with
> special privileges
> that allow it to read files that are marked as unreadable.
> {quote}
> I think these tests can be disabled on Windows using @DisabledOnOs(OS.WINDOWS)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-20375) Thin 3.0: Review packArrayHeader usage
[
https://issues.apache.org/jira/browse/IGNITE-20375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pavel Tupitsyn updated IGNITE-20375:
Description:
Ignite 3.x client protocol uses MsgPack for message serialization. In some
places we use *packArrayHeader* incorrectly, for example,
[ClientCompute.packJob|https://github.com/apache/ignite-3/blob/fa8d626f57e3497289c0aaa1bfa8efd19f1042f7/modules/client/src/main/java/org/apache/ignite/internal/client/compute/ClientCompute.java#L330]:
{code:java}
w.packArrayHeader(units.size());
for (DeploymentUnit unit : units) {
w.packString(unit.name());
w.packString(unit.version().render());
}
{code}
We pack 2N items for an array of size N, and in some cases those items can be
of different type, which breaks the rules of MsgPack protocol.
This creates issues with some MsgPack implemetations, such as the one used in
C++ client (msgpack-c):
* Exactly N items must be present
* Entire array is deserialized at once (can't read large arrays one by one)
Proposal: don't use *packArrayHeader* at all, replace with *packInt*, which
allows more flexibility.
was:
Ignite 3.x client protocol uses MsgPack for message serialization. In some
places we use *packArrayHeader* incorrectly, for example,
[ClientCompute.packJob|https://github.com/apache/ignite-3/blob/fa8d626f57e3497289c0aaa1bfa8efd19f1042f7/modules/client/src/main/java/org/apache/ignite/internal/client/compute/ClientCompute.java#L330]:
{code:java}
w.packArrayHeader(units.size());
for (DeploymentUnit unit : units) {
w.packString(unit.name());
w.packString(unit.version().render());
}
{code}
We pack 2N items for an array of size N, and in some cases those items can be
of different type, which breaks the rules of MsgPack protocol.
This creates issues with some MsgPack implemetations, such as the one used in
C++ client (msgpack-c).
> Thin 3.0: Review packArrayHeader usage
> --
>
> Key: IGNITE-20375
> URL: https://issues.apache.org/jira/browse/IGNITE-20375
> Project: Ignite
> Issue Type: Improvement
> Components: thin client
>Affects Versions: 3.0.0-beta1
>Reporter: Pavel Tupitsyn
>Assignee: Pavel Tupitsyn
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> Ignite 3.x client protocol uses MsgPack for message serialization. In some
> places we use *packArrayHeader* incorrectly, for example,
> [ClientCompute.packJob|https://github.com/apache/ignite-3/blob/fa8d626f57e3497289c0aaa1bfa8efd19f1042f7/modules/client/src/main/java/org/apache/ignite/internal/client/compute/ClientCompute.java#L330]:
> {code:java}
> w.packArrayHeader(units.size());
> for (DeploymentUnit unit : units) {
> w.packString(unit.name());
> w.packString(unit.version().render());
> }
> {code}
> We pack 2N items for an array of size N, and in some cases those items can be
> of different type, which breaks the rules of MsgPack protocol.
> This creates issues with some MsgPack implemetations, such as the one used in
> C++ client (msgpack-c):
> * Exactly N items must be present
> * Entire array is deserialized at once (can't read large arrays one by one)
> Proposal: don't use *packArrayHeader* at all, replace with *packInt*, which
> allows more flexibility.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IGNITE-20376) ItServerDataStreamerTest.testManyItems is flaky: PrimaryReplicaMissException
Pavel Tupitsyn created IGNITE-20376: --- Summary: ItServerDataStreamerTest.testManyItems is flaky: PrimaryReplicaMissException Key: IGNITE-20376 URL: https://issues.apache.org/jira/browse/IGNITE-20376 Project: Ignite Issue Type: Bug Affects Versions: 3.0.0-beta1 Reporter: Pavel Tupitsyn Assignee: Pavel Tupitsyn Fix For: 3.0.0-beta2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-20376) ItServerDataStreamerTest.testManyItems is flaky: PrimaryReplicaMissException
[ https://issues.apache.org/jira/browse/IGNITE-20376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Tupitsyn updated IGNITE-20376: Attachment: ignite-20376.txt > ItServerDataStreamerTest.testManyItems is flaky: PrimaryReplicaMissException > > > Key: IGNITE-20376 > URL: https://issues.apache.org/jira/browse/IGNITE-20376 > Project: Ignite > Issue Type: Bug >Affects Versions: 3.0.0-beta1 >Reporter: Pavel Tupitsyn >Assignee: Pavel Tupitsyn >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Attachments: ignite-20376.txt > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-20376) ItServerDataStreamerTest.testManyItems is flaky: PrimaryReplicaMissException
[
https://issues.apache.org/jira/browse/IGNITE-20376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pavel Tupitsyn updated IGNITE-20376:
Description:
{code}
java.util.concurrent.CompletionException:
org.apache.ignite.internal.replicator.exception.PrimaryReplicaMissException:
IGN-REP-6 TraceId:72a5da42-c429-447a-aaf3-86c1137994dd The primary replica has
changed because the term has been changed [expectedPrimaryReplicaTerm=1,
currentPrimaryReplicaTerm=2]
{code}
> ItServerDataStreamerTest.testManyItems is flaky: PrimaryReplicaMissException
>
>
> Key: IGNITE-20376
> URL: https://issues.apache.org/jira/browse/IGNITE-20376
> Project: Ignite
> Issue Type: Bug
>Affects Versions: 3.0.0-beta1
>Reporter: Pavel Tupitsyn
>Assignee: Pavel Tupitsyn
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
> Attachments: ignite-20376.txt
>
>
> {code}
> java.util.concurrent.CompletionException:
> org.apache.ignite.internal.replicator.exception.PrimaryReplicaMissException:
> IGN-REP-6 TraceId:72a5da42-c429-447a-aaf3-86c1137994dd The primary replica
> has changed because the term has been changed [expectedPrimaryReplicaTerm=1,
> currentPrimaryReplicaTerm=2]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IGNITE-20377) Rename term to enlistmentConsistencyToken where primary Replica is used
Alexander Lapin created IGNITE-20377: Summary: Rename term to enlistmentConsistencyToken where primary Replica is used Key: IGNITE-20377 URL: https://issues.apache.org/jira/browse/IGNITE-20377 Project: Ignite Issue Type: Improvement Reporter: Alexander Lapin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-20377) Rename term to enlistmentConsistencyToken where primary Replica is used
[ https://issues.apache.org/jira/browse/IGNITE-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-20377: - Ignite Flags: (was: Docs Required,Release Notes Required) > Rename term to enlistmentConsistencyToken where primary Replica is used > --- > > Key: IGNITE-20377 > URL: https://issues.apache.org/jira/browse/IGNITE-20377 > Project: Ignite > Issue Type: Improvement >Reporter: Alexander Lapin >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-20377) Rename term to enlistmentConsistencyToken where primary Replica is used
[ https://issues.apache.org/jira/browse/IGNITE-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-20377: - Description: h3. Motivation Within IGNITE-18856 primary replica calculation was switched from raft leader collocation to primary replicas provided by placement driver service, however within replica requests, exceptions, tx context naming term still used (despite the fact that instead real term it holds lease.startTime). So, it's required to rename term wherever possible. h3. Definition of Done * Term should be renamed to enlistment consistency token or similar wherever leases are used. > Rename term to enlistmentConsistencyToken where primary Replica is used > --- > > Key: IGNITE-20377 > URL: https://issues.apache.org/jira/browse/IGNITE-20377 > Project: Ignite > Issue Type: Improvement >Reporter: Alexander Lapin >Priority: Major > > h3. Motivation > Within IGNITE-18856 primary replica calculation was switched from raft leader > collocation to primary replicas provided by placement driver service, however > within replica requests, exceptions, tx context naming term still used > (despite the fact that instead real term it holds lease.startTime). So, it's > required to rename term wherever possible. > h3. Definition of Done > * Term should be renamed to enlistment consistency token or similar wherever > leases are used. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-20377) Rename term to enlistmentConsistencyToken where primary Replica is used
[ https://issues.apache.org/jira/browse/IGNITE-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-20377: - Labels: ignite-3 (was: ) > Rename term to enlistmentConsistencyToken where primary Replica is used > --- > > Key: IGNITE-20377 > URL: https://issues.apache.org/jira/browse/IGNITE-20377 > Project: Ignite > Issue Type: Improvement >Reporter: Alexander Lapin >Priority: Major > Labels: ignite-3 > > h3. Motivation > Within IGNITE-18856 primary replica calculation was switched from raft leader > collocation to primary replicas provided by placement driver service, however > within replica requests, exceptions, tx context naming term still used > (despite the fact that instead real term it holds lease.startTime). So, it's > required to rename term wherever possible. > h3. Definition of Done > * Term should be renamed to enlistment consistency token or similar wherever > leases are used. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-20377) Rename term to enlistmentConsistencyToken wherever primary replica is used
[ https://issues.apache.org/jira/browse/IGNITE-20377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-20377: - Summary: Rename term to enlistmentConsistencyToken wherever primary replica is used (was: Rename term to enlistmentConsistencyToken where primary Replica is used) > Rename term to enlistmentConsistencyToken wherever primary replica is used > -- > > Key: IGNITE-20377 > URL: https://issues.apache.org/jira/browse/IGNITE-20377 > Project: Ignite > Issue Type: Improvement >Reporter: Alexander Lapin >Priority: Major > Labels: ignite-3 > > h3. Motivation > Within IGNITE-18856 primary replica calculation was switched from raft leader > collocation to primary replicas provided by placement driver service, however > within replica requests, exceptions, tx context naming term still used > (despite the fact that instead real term it holds lease.startTime). So, it's > required to rename term wherever possible. > h3. Definition of Done > * Term should be renamed to enlistment consistency token or similar wherever > leases are used. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-20354) Reduce code duplication in commit cleanup
[ https://issues.apache.org/jira/browse/IGNITE-20354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17762688#comment-17762688 ] Denis Chudov commented on IGNITE-20354: --- [~ksizov] LGTM. > Reduce code duplication in commit cleanup > - > > Key: IGNITE-20354 > URL: https://issues.apache.org/jira/browse/IGNITE-20354 > Project: Ignite > Issue Type: Task >Reporter: Kirill Sizov >Assignee: Kirill Sizov >Priority: Major > Labels: ignite-3 > > PartitionListener and PartitionReplicaListener have the storage cleanup code > duplicated, any change to it should be introduced in both places. > *Implementation details* > There is already a StorageUpdateHandler that was created to reuse the storage > access logic. > The storage commit cleanup should be moved into this class from both > *Listener classes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-20378) Observable timestamp should be adjusted for DDL operations
[ https://issues.apache.org/jira/browse/IGNITE-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-20378: - Ignite Flags: (was: Docs Required,Release Notes Required) > Observable timestamp should be adjusted for DDL operations > -- > > Key: IGNITE-20378 > URL: https://issues.apache.org/jira/browse/IGNITE-20378 > Project: Ignite > Issue Type: Improvement >Reporter: Alexander Lapin >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-20378) Observable timestamp should be adjusted for DDL operations
[ https://issues.apache.org/jira/browse/IGNITE-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-20378: - Labels: ignite-3 (was: ) > Observable timestamp should be adjusted for DDL operations > -- > > Key: IGNITE-20378 > URL: https://issues.apache.org/jira/browse/IGNITE-20378 > Project: Ignite > Issue Type: Improvement >Reporter: Alexander Lapin >Priority: Major > Labels: ignite-3 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-20378) Observable timestamp should be adjusted for DDL operations
Alexander Lapin created IGNITE-20378: Summary: Observable timestamp should be adjusted for DDL operations Key: IGNITE-20378 URL: https://issues.apache.org/jira/browse/IGNITE-20378 Project: Ignite Issue Type: Improvement Reporter: Alexander Lapin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-20378) Observable timestamp should be adjusted for DDL operations
[
https://issues.apache.org/jira/browse/IGNITE-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Lapin updated IGNITE-20378:
-
Description:
h3. Motivation
Both client and embedded nodes should have an ability to see same-actor schema
adjustments though RO requests.
{code:java}
client1.sql(CREATE TABLE t1);
client.beginRO()
client.sql(SELECT * from t1); // Given request should see t1.
...
{code}
Definition o
> Observable timestamp should be adjusted for DDL operations
> --
>
> Key: IGNITE-20378
> URL: https://issues.apache.org/jira/browse/IGNITE-20378
> Project: Ignite
> Issue Type: Improvement
>Reporter: Alexander Lapin
>Priority: Major
> Labels: ignite-3
>
> h3. Motivation
> Both client and embedded nodes should have an ability to see same-actor
> schema adjustments though RO requests.
>
> {code:java}
> client1.sql(CREATE TABLE t1);
> client.beginRO()
> client.sql(SELECT * from t1); // Given request should see t1.
> ...
> {code}
> Definition o
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-20378) Observable timestamp should be adjusted for DDL operations
[
https://issues.apache.org/jira/browse/IGNITE-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Lapin updated IGNITE-20378:
-
Description:
h3. Motivation
Both client and embedded nodes should have an ability to see same-actor schema
adjustments though RO requests.
{code:java}
client1.sql(CREATE TABLE t1);
client.beginRO()
client.sql(SELECT * from t1); // Given request should see t1.
...
{code}
Definition o
Observable timestamp should be adjusted for DDL operations
was:
h3. Motivation
Both client and embedded nodes should have an ability to see same-actor schema
adjustments though RO requests.
{code:java}
client1.sql(CREATE TABLE t1);
client.beginRO()
client.sql(SELECT * from t1); // Given request should see t1.
...
{code}
Definition o
> Observable timestamp should be adjusted for DDL operations
> --
>
> Key: IGNITE-20378
> URL: https://issues.apache.org/jira/browse/IGNITE-20378
> Project: Ignite
> Issue Type: Improvement
>Reporter: Alexander Lapin
>Priority: Major
> Labels: ignite-3
>
> h3. Motivation
> Both client and embedded nodes should have an ability to see same-actor
> schema adjustments though RO requests.
>
> {code:java}
> client1.sql(CREATE TABLE t1);
> client.beginRO()
> client.sql(SELECT * from t1); // Given request should see t1.
> ...
> {code}
> Definition o
> Observable timestamp should be adjusted for DDL operations
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-20378) Both clients and embedded nodes should have an ability to see same-actor schema adjustments though RO requests
[
https://issues.apache.org/jira/browse/IGNITE-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Lapin updated IGNITE-20378:
-
Summary: Both clients and embedded nodes should have an ability to see
same-actor schema adjustments though RO requests (was: Observable timestamp
should be adjusted for DDL operations)
> Both clients and embedded nodes should have an ability to see same-actor
> schema adjustments though RO requests
>
>
> Key: IGNITE-20378
> URL: https://issues.apache.org/jira/browse/IGNITE-20378
> Project: Ignite
> Issue Type: Improvement
>Reporter: Alexander Lapin
>Priority: Major
> Labels: ignite-3
>
> h3. Motivation
> Both client and embedded nodes should have an ability to see same-actor
> schema adjustments though RO requests.
>
> {code:java}
> client1.sql(CREATE TABLE t1);
> client.beginRO()
> client.sql(SELECT * from t1); // Given request should see t1.
> ...
> {code}
> Definition o
> Observable timestamp should be adjusted for DDL operations
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-20378) Both clients and embedded nodes should have an ability to see same-actor schema adjustments though RO requests
[
https://issues.apache.org/jira/browse/IGNITE-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Lapin updated IGNITE-20378:
-
Description:
h3. Motivation
{code:java}
client1.sql(CREATE TABLE t1);
client.beginRO()
client.sql(SELECT * from t1); // Given request should see t1.
...
{code}
However it won't be true by default because we may step back for idle safe time
propagation period that might be greater than delay duration in metadata sync
logic.
h3. Definition of Done
* Both clients and embedded nodes should have an ability to see same-actor
schema adjustments though RO requests.
h3. Implementation Notes
Seems that it's required to adjust observable timestamp within DDL operations
or we may consider setting up DD greater than idle safe time propagation period.
was:
h3. Motivation
Both client and embedded nodes should have an ability to see same-actor schema
adjustments though RO requests.
{code:java}
client1.sql(CREATE TABLE t1);
client.beginRO()
client.sql(SELECT * from t1); // Given request should see t1.
...
{code}
Definition o
Observable timestamp should be adjusted for DDL operations
> Both clients and embedded nodes should have an ability to see same-actor
> schema adjustments though RO requests
>
>
> Key: IGNITE-20378
> URL: https://issues.apache.org/jira/browse/IGNITE-20378
> Project: Ignite
> Issue Type: Improvement
>Reporter: Alexander Lapin
>Priority: Major
> Labels: ignite-3
>
> h3. Motivation
> {code:java}
> client1.sql(CREATE TABLE t1);
> client.beginRO()
> client.sql(SELECT * from t1); // Given request should see t1.
> ...
> {code}
> However it won't be true by default because we may step back for idle safe
> time propagation period that might be greater than delay duration in metadata
> sync logic.
> h3. Definition of Done
> * Both clients and embedded nodes should have an ability to see same-actor
> schema adjustments though RO requests.
> h3. Implementation Notes
> Seems that it's required to adjust observable timestamp within DDL operations
> or we may consider setting up DD greater than idle safe time propagation
> period.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-20378) Both clients and embedded nodes should have an ability to see same-actor schema adjustments though RO requests
[
https://issues.apache.org/jira/browse/IGNITE-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Lapin updated IGNITE-20378:
-
Description:
h3. Motivation
{code:java}
client1.sql(CREATE TABLE t1);
client.beginRO()
client.sql(SELECT * from t1); // Given request should see t1.
...
{code}
However it won't be true by default because we may step back for idle safe time
propagation period that might be greater than delay duration in metadata sync
logic.
h3. Definition of Done
* Both clients and embedded nodes should have an ability to see same-actor
schema adjustments though RO requests.
h3. Implementation Notes
Seems that it's required to adjust observable timestamp within DDL operations
or we may consider setting DD greater than idle safe time propagation period.
was:
h3. Motivation
{code:java}
client1.sql(CREATE TABLE t1);
client.beginRO()
client.sql(SELECT * from t1); // Given request should see t1.
...
{code}
However it won't be true by default because we may step back for idle safe time
propagation period that might be greater than delay duration in metadata sync
logic.
h3. Definition of Done
* Both clients and embedded nodes should have an ability to see same-actor
schema adjustments though RO requests.
h3. Implementation Notes
Seems that it's required to adjust observable timestamp within DDL operations
or we may consider setting up DD greater than idle safe time propagation period.
> Both clients and embedded nodes should have an ability to see same-actor
> schema adjustments though RO requests
>
>
> Key: IGNITE-20378
> URL: https://issues.apache.org/jira/browse/IGNITE-20378
> Project: Ignite
> Issue Type: Improvement
>Reporter: Alexander Lapin
>Priority: Major
> Labels: ignite-3
>
> h3. Motivation
> {code:java}
> client1.sql(CREATE TABLE t1);
> client.beginRO()
> client.sql(SELECT * from t1); // Given request should see t1.
> ...
> {code}
> However it won't be true by default because we may step back for idle safe
> time propagation period that might be greater than delay duration in metadata
> sync logic.
> h3. Definition of Done
> * Both clients and embedded nodes should have an ability to see same-actor
> schema adjustments though RO requests.
> h3. Implementation Notes
> Seems that it's required to adjust observable timestamp within DDL operations
> or we may consider setting DD greater than idle safe time propagation period.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-20378) Both clients and embedded nodes should have an ability to see same-actor schema adjustments though RO requests
[
https://issues.apache.org/jira/browse/IGNITE-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Lapin updated IGNITE-20378:
-
Epic Link: IGNITE-19884
> Both clients and embedded nodes should have an ability to see same-actor
> schema adjustments though RO requests
>
>
> Key: IGNITE-20378
> URL: https://issues.apache.org/jira/browse/IGNITE-20378
> Project: Ignite
> Issue Type: Improvement
>Reporter: Alexander Lapin
>Priority: Major
> Labels: ignite-3
>
> h3. Motivation
> {code:java}
> client1.sql(CREATE TABLE t1);
> client.beginRO()
> client.sql(SELECT * from t1); // Given request should see t1.
> ...
> {code}
> However it won't be true by default because we may step back for idle safe
> time propagation period that might be greater than delay duration in metadata
> sync logic.
> h3. Definition of Done
> * Both clients and embedded nodes should have an ability to see same-actor
> schema adjustments though RO requests.
> h3. Implementation Notes
> Seems that it's required to adjust observable timestamp within DDL operations
> or we may consider setting DD greater than idle safe time propagation period.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-18499) Make IncomingSnapshotCopier behave closer to LocalSnapshotCopier
[
https://issues.apache.org/jira/browse/IGNITE-18499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17762698#comment-17762698
]
Roman Puchkovskiy commented on IGNITE-18499:
Item 4 was implemented in IGNITE-20256, so I removed the item from the list
> Make IncomingSnapshotCopier behave closer to LocalSnapshotCopier
>
>
> Key: IGNITE-18499
> URL: https://issues.apache.org/jira/browse/IGNITE-18499
> Project: Ignite
> Issue Type: Improvement
>Reporter: Roman Puchkovskiy
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> There are a few differences between {{IncomingSnapshotCopier}} and
> {{{}LocalSnapshotCopier{}}}. JRaft seems to be very sensitive to little
> changes in behavior because it relies on many invariants, some of them
> implicit, to function correctly. Hence it is best to have our copier mimic
> the stock one as fully as possible (where it's feasible).
> Here are the differences:
> # On cancellation, the copier should be transferred to an erroneous state
> (using {{{}setError(){}}})
> # On closure, the copier should be cancelled
> # The stock copier does not invoke {{join()}} on cancellation, but our
> implementation does. On the one hand, waiting seems to make sense because it
> allows to save retries, but it's a difference from the stock logic which
> might be dangerous.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-18499) Make IncomingSnapshotCopier behave closer to LocalSnapshotCopier
[
https://issues.apache.org/jira/browse/IGNITE-18499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Puchkovskiy updated IGNITE-18499:
---
Description:
There are a few differences between {{IncomingSnapshotCopier}} and
{{{}LocalSnapshotCopier{}}}. JRaft seems to be very sensitive to little changes
in behavior because it relies on many invariants, some of them implicit, to
function correctly. Hence it is best to have our copier mimic the stock one as
fully as possible (where it's feasible).
Here are the differences:
# On cancellation, the copier should be transferred to an erroneous state
(using {{{}setError(){}}})
# On closure, the copier should be cancelled
# The stock copier does not invoke {{join()}} on cancellation, but our
implementation does. On the one hand, waiting seems to make sense because it
allows to save retries, but it's a difference from the stock logic which might
be dangerous.
was:
There are a few differences between {{IncomingSnapshotCopier}} and
{{{}LocalSnapshotCopier{}}}. JRaft seems to be very sensitive to little changes
in behavior because it relies on many invariants, some of them implicit, to
function correctly. Hence it is best to have our copier mimic the stock one as
fully as possible (where it's feasible).
Here are the differences:
# On cancellation, the copier should be transferred to an erroneous state
(using {{{}setError(){}}})
# On closure, the copier should be cancelled
# The stock copier does not invoke {{join()}} on cancellation, but our
implementation does. On the one hand, waiting seems to make sense because it
allows to save retries, but it's a difference from the stock logic which might
be dangerous.
# We should only set error if it has not been set yet; but our copier
implementation usually does the opposite: if there is an error, it sets some
other error
> Make IncomingSnapshotCopier behave closer to LocalSnapshotCopier
>
>
> Key: IGNITE-18499
> URL: https://issues.apache.org/jira/browse/IGNITE-18499
> Project: Ignite
> Issue Type: Improvement
>Reporter: Roman Puchkovskiy
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> There are a few differences between {{IncomingSnapshotCopier}} and
> {{{}LocalSnapshotCopier{}}}. JRaft seems to be very sensitive to little
> changes in behavior because it relies on many invariants, some of them
> implicit, to function correctly. Hence it is best to have our copier mimic
> the stock one as fully as possible (where it's feasible).
> Here are the differences:
> # On cancellation, the copier should be transferred to an erroneous state
> (using {{{}setError(){}}})
> # On closure, the copier should be cancelled
> # The stock copier does not invoke {{join()}} on cancellation, but our
> implementation does. On the one hand, waiting seems to make sense because it
> allows to save retries, but it's a difference from the stock logic which
> might be dangerous.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-20354) Reduce code duplication in commit cleanup
[ https://issues.apache.org/jira/browse/IGNITE-20354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vyacheslav Koptilin updated IGNITE-20354: - Ignite Flags: (was: Docs Required,Release Notes Required) > Reduce code duplication in commit cleanup > - > > Key: IGNITE-20354 > URL: https://issues.apache.org/jira/browse/IGNITE-20354 > Project: Ignite > Issue Type: Task >Reporter: Kirill Sizov >Assignee: Kirill Sizov >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > PartitionListener and PartitionReplicaListener have the storage cleanup code > duplicated, any change to it should be introduced in both places. > *Implementation details* > There is already a StorageUpdateHandler that was created to reuse the storage > access logic. > The storage commit cleanup should be moved into this class from both > *Listener classes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-20373) Fix IO threading model
[ https://issues.apache.org/jira/browse/IGNITE-20373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Scherbakov updated IGNITE-20373: --- Description: Currently IO is resubmitted to inbound pool for further processing (even there are corner cases then a message handler called in IO thread) It introduces additional latency in message processing because each component other uses it's own pools. addMessageHandler should be extended with a 3-d argument: a pool for submitting a callback for execution. Some kind of message header can be passed along the way for precise pool selection (similar to jraft IO model) inbound pool should be removed IMPORTANT: delivery guaranties should be preserved: if a message A is send before B, B can't be processed on a receiver before A was: Currently IO is resubmitted to inbound pool for further processing (even there are corner cases then a message handler called in IO thread) It introduces additional latency in message processing because each component other uses it's own pools. addMessageHandler should be extended with a 3-d argument: a pool for submitting a callback for execution. Some kind of message header can be passed along the way for precise pool selection (similar to jraft IO model) inbound pool should be removed > Fix IO threading model > -- > > Key: IGNITE-20373 > URL: https://issues.apache.org/jira/browse/IGNITE-20373 > Project: Ignite > Issue Type: Improvement >Affects Versions: 3.0 >Reporter: Alexey Scherbakov >Priority: Major > Labels: ignite-3, ignite3_performance > Fix For: 3.0 > > > Currently IO is resubmitted to inbound pool for further processing (even > there are corner cases then a message handler called in IO thread) > It introduces additional latency in message processing because each component > other uses it's own pools. > addMessageHandler should be extended with a 3-d argument: a pool for > submitting a callback for execution. > Some kind of message header can be passed along the way for precise pool > selection (similar to jraft IO model) > inbound pool should be removed > IMPORTANT: delivery guaranties should be preserved: if a message A is send > before B, B can't be processed on a receiver before A -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-20373) Fix IO threading model
[ https://issues.apache.org/jira/browse/IGNITE-20373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Scherbakov updated IGNITE-20373: --- Description: Currently IO is resubmitted to inbound pool for further processing (even there are corner cases then a message handler called in IO thread) It makes message processing essentially single threaded and introduces additional lag due to multiple queues message transaction. addMessageHandler should be extended with a 3-d argument: a pool for submitting a callback for execution. Some kind of message header can be passed along the way for precise pool selection (similar to jraft IO model) inbound pool should be removed IMPORTANT: delivery guaranties should be preserved: if a message A is send before B, B can't be processed on a receiver before A was: Currently IO is resubmitted to inbound pool for further processing (even there are corner cases then a message handler called in IO thread) It introduces additional latency in message processing because each component other uses it's own pools. addMessageHandler should be extended with a 3-d argument: a pool for submitting a callback for execution. Some kind of message header can be passed along the way for precise pool selection (similar to jraft IO model) inbound pool should be removed IMPORTANT: delivery guaranties should be preserved: if a message A is send before B, B can't be processed on a receiver before A > Fix IO threading model > -- > > Key: IGNITE-20373 > URL: https://issues.apache.org/jira/browse/IGNITE-20373 > Project: Ignite > Issue Type: Improvement >Affects Versions: 3.0 >Reporter: Alexey Scherbakov >Priority: Major > Labels: ignite-3, ignite3_performance > Fix For: 3.0 > > > Currently IO is resubmitted to inbound pool for further processing (even > there are corner cases then a message handler called in IO thread) > It makes message processing essentially single threaded and introduces > additional lag due to multiple queues message transaction. > addMessageHandler should be extended with a 3-d argument: a pool for > submitting a callback for execution. > Some kind of message header can be passed along the way for precise pool > selection (similar to jraft IO model) > inbound pool should be removed > IMPORTANT: delivery guaranties should be preserved: if a message A is send > before B, B can't be processed on a receiver before A -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-20373) Fix IO threading model
[ https://issues.apache.org/jira/browse/IGNITE-20373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Scherbakov updated IGNITE-20373: --- Description: Currently IO is resubmitted to inboundExecutor for further processing (even there are corner cases then a message handler called in IO thread) It makes message processing essentially single threaded and introduces additional lag due to multiple queues message transaction. addMessageHandler should be extended with a 3-d argument: a pool for submitting a callback for execution. Some kind of message header can be passed along the way for precise pool selection (similar to jraft IO model) inbound pool should be removed IMPORTANT: delivery guaranties should be preserved: if a message A is send before B, B can't be processed on a receiver before A outboundExecutor also looks like a contention point - need to be addressed as well was: Currently IO is resubmitted to inbound pool for further processing (even there are corner cases then a message handler called in IO thread) It makes message processing essentially single threaded and introduces additional lag due to multiple queues message transaction. addMessageHandler should be extended with a 3-d argument: a pool for submitting a callback for execution. Some kind of message header can be passed along the way for precise pool selection (similar to jraft IO model) inbound pool should be removed IMPORTANT: delivery guaranties should be preserved: if a message A is send before B, B can't be processed on a receiver before A > Fix IO threading model > -- > > Key: IGNITE-20373 > URL: https://issues.apache.org/jira/browse/IGNITE-20373 > Project: Ignite > Issue Type: Improvement >Affects Versions: 3.0 >Reporter: Alexey Scherbakov >Priority: Major > Labels: ignite-3, ignite3_performance > Fix For: 3.0 > > > Currently IO is resubmitted to inboundExecutor for further processing (even > there are corner cases then a message handler called in IO thread) > It makes message processing essentially single threaded and introduces > additional lag due to multiple queues message transaction. > addMessageHandler should be extended with a 3-d argument: a pool for > submitting a callback for execution. > Some kind of message header can be passed along the way for precise pool > selection (similar to jraft IO model) > inbound pool should be removed > IMPORTANT: delivery guaranties should be preserved: if a message A is send > before B, B can't be processed on a receiver before A > outboundExecutor also looks like a contention point - need to be addressed as > well -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-20373) Fix IO threading model
[ https://issues.apache.org/jira/browse/IGNITE-20373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Scherbakov updated IGNITE-20373: --- Description: Currently IO is resubmitted to inboundExecutor for further processing (even there are corner cases then a message handler called in IO thread) It makes message processing essentially single threaded and introduces additional lag due to message transition between multiple queues. addMessageHandler should be extended with a 3-d argument: a pool for submitting a callback for execution. Some kind of message header can be passed along the way for precise pool selection (similar to jraft IO model) inbound pool should be removed IMPORTANT: delivery guaranties should be preserved: if a message A is send before B, B can't be processed on a receiver before A outboundExecutor also looks like a contention point - need to be addressed as well was: Currently IO is resubmitted to inboundExecutor for further processing (even there are corner cases then a message handler called in IO thread) It makes message processing essentially single threaded and introduces additional lag due to multiple queues message transaction. addMessageHandler should be extended with a 3-d argument: a pool for submitting a callback for execution. Some kind of message header can be passed along the way for precise pool selection (similar to jraft IO model) inbound pool should be removed IMPORTANT: delivery guaranties should be preserved: if a message A is send before B, B can't be processed on a receiver before A outboundExecutor also looks like a contention point - need to be addressed as well > Fix IO threading model > -- > > Key: IGNITE-20373 > URL: https://issues.apache.org/jira/browse/IGNITE-20373 > Project: Ignite > Issue Type: Improvement >Affects Versions: 3.0 >Reporter: Alexey Scherbakov >Priority: Major > Labels: ignite-3, ignite3_performance > Fix For: 3.0 > > > Currently IO is resubmitted to inboundExecutor for further processing (even > there are corner cases then a message handler called in IO thread) > It makes message processing essentially single threaded and introduces > additional lag due to message transition between multiple queues. > addMessageHandler should be extended with a 3-d argument: a pool for > submitting a callback for execution. > Some kind of message header can be passed along the way for precise pool > selection (similar to jraft IO model) > inbound pool should be removed > IMPORTANT: delivery guaranties should be preserved: if a message A is send > before B, B can't be processed on a receiver before A > outboundExecutor also looks like a contention point - need to be addressed as > well -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-20373) Fix IO threading model
[ https://issues.apache.org/jira/browse/IGNITE-20373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Scherbakov updated IGNITE-20373: --- Description: Currently IO is resubmitted to inboundExecutor for further processing (even there are corner cases then a message handler called in IO thread) It makes message processing essentially single threaded and introduces additional lag due to message transition between multiple queues. addMessageHandler should be extended with a 3-d argument: a pool for submitting a callback for execution, or an executorSelector like in jraft. inbound pool should be removed IMPORTANT: delivery guaranties should be preserved: if a message A is send before B, B can't be processed on a receiver before A outboundExecutor also looks like a contention point - need to be addressed as well was: Currently IO is resubmitted to inboundExecutor for further processing (even there are corner cases then a message handler called in IO thread) It makes message processing essentially single threaded and introduces additional lag due to message transition between multiple queues. addMessageHandler should be extended with a 3-d argument: a pool for submitting a callback for execution. Some kind of message header can be passed along the way for precise pool selection (similar to jraft IO model) inbound pool should be removed IMPORTANT: delivery guaranties should be preserved: if a message A is send before B, B can't be processed on a receiver before A outboundExecutor also looks like a contention point - need to be addressed as well > Fix IO threading model > -- > > Key: IGNITE-20373 > URL: https://issues.apache.org/jira/browse/IGNITE-20373 > Project: Ignite > Issue Type: Improvement >Affects Versions: 3.0 >Reporter: Alexey Scherbakov >Priority: Major > Labels: ignite-3, ignite3_performance > Fix For: 3.0 > > > Currently IO is resubmitted to inboundExecutor for further processing (even > there are corner cases then a message handler called in IO thread) > It makes message processing essentially single threaded and introduces > additional lag due to message transition between multiple queues. > addMessageHandler should be extended with a 3-d argument: a pool for > submitting a callback for execution, or an executorSelector like in jraft. > inbound pool should be removed > IMPORTANT: delivery guaranties should be preserved: if a message A is send > before B, B can't be processed on a receiver before A > outboundExecutor also looks like a contention point - need to be addressed as > well -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-20373) Fix IO threading model
[ https://issues.apache.org/jira/browse/IGNITE-20373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Scherbakov updated IGNITE-20373: --- Description: Currently IO is resubmitted to inboundExecutor for further processing (even there are corner cases then a message handler called in IO thread) It makes message processing essentially single threaded and introduces additional lag due to message transition between multiple queues. addMessageHandler should be extended with a 3-d argument: a pool for submitting a callback for execution, or an executorSelector like in jraft. inboundExecutor should be changed to striped, use more than one thread, and serve messages without explicit executor IMPORTANT: delivery guaranties should be preserved: if a message A is send before B, B can't be processed on a receiver before A outboundExecutor also looks like a contention point - need to be addressed as well was: Currently IO is resubmitted to inboundExecutor for further processing (even there are corner cases then a message handler called in IO thread) It makes message processing essentially single threaded and introduces additional lag due to message transition between multiple queues. addMessageHandler should be extended with a 3-d argument: a pool for submitting a callback for execution, or an executorSelector like in jraft. inbound pool should be removed IMPORTANT: delivery guaranties should be preserved: if a message A is send before B, B can't be processed on a receiver before A outboundExecutor also looks like a contention point - need to be addressed as well > Fix IO threading model > -- > > Key: IGNITE-20373 > URL: https://issues.apache.org/jira/browse/IGNITE-20373 > Project: Ignite > Issue Type: Improvement >Affects Versions: 3.0 >Reporter: Alexey Scherbakov >Priority: Major > Labels: ignite-3, ignite3_performance > Fix For: 3.0 > > > Currently IO is resubmitted to inboundExecutor for further processing (even > there are corner cases then a message handler called in IO thread) > It makes message processing essentially single threaded and introduces > additional lag due to message transition between multiple queues. > addMessageHandler should be extended with a 3-d argument: a pool for > submitting a callback for execution, or an executorSelector like in jraft. > inboundExecutor should be changed to striped, use more than one thread, and > serve messages without explicit executor > IMPORTANT: delivery guaranties should be preserved: if a message A is send > before B, B can't be processed on a receiver before A > outboundExecutor also looks like a contention point - need to be addressed as > well -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-20373) Fix IO threading model
[ https://issues.apache.org/jira/browse/IGNITE-20373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Scherbakov updated IGNITE-20373: --- Description: Currently IO is resubmitted to inboundExecutor for further processing (even there are corner cases then a message handler called in IO thread) It makes message processing essentially single threaded and introduces additional lag due to message transition between multiple queues. addMessageHandler should be extended with a 3-d argument: a pool for submitting a callback for execution, or an executorSelector like in jraft. inboundExecutor should be changed to striped, use more than one thread, and serve messages without explicit executor. Delivery guaranties should be preserved: if a message A is send before B, B can't be processed on a receiver before A outboundExecutor also looks like a contention point - need to be addressed as well was: Currently IO is resubmitted to inboundExecutor for further processing (even there are corner cases then a message handler called in IO thread) It makes message processing essentially single threaded and introduces additional lag due to message transition between multiple queues. addMessageHandler should be extended with a 3-d argument: a pool for submitting a callback for execution, or an executorSelector like in jraft. inboundExecutor should be changed to striped, use more than one thread, and serve messages without explicit executor IMPORTANT: delivery guaranties should be preserved: if a message A is send before B, B can't be processed on a receiver before A outboundExecutor also looks like a contention point - need to be addressed as well > Fix IO threading model > -- > > Key: IGNITE-20373 > URL: https://issues.apache.org/jira/browse/IGNITE-20373 > Project: Ignite > Issue Type: Improvement >Affects Versions: 3.0 >Reporter: Alexey Scherbakov >Priority: Major > Labels: ignite-3, ignite3_performance > Fix For: 3.0 > > > Currently IO is resubmitted to inboundExecutor for further processing (even > there are corner cases then a message handler called in IO thread) > It makes message processing essentially single threaded and introduces > additional lag due to message transition between multiple queues. > addMessageHandler should be extended with a 3-d argument: a pool for > submitting a callback for execution, or an executorSelector like in jraft. > inboundExecutor should be changed to striped, use more than one thread, and > serve messages without explicit executor. Delivery guaranties should be > preserved: if a message A is send before B, B can't be processed on a > receiver before A > outboundExecutor also looks like a contention point - need to be addressed as > well -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19106) Sql. Column namings are partially broken after upgrading to calcite 1.34
[
https://issues.apache.org/jira/browse/IGNITE-19106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Evgeny Stanilovsky updated IGNITE-19106:
Description:
After upgrading to calcite 1.34 SqlValidator#deriveAlias and overloading call
IgniteSqlValidator#deriveAlias is changed, thus requests like :
select v / 2 from t, return EXPR$ instead of column name derived from
IgniteSqlValidator#deriveAlias. Fast (near) fix looks like cover both such
queries:
Check annotation for: SqlValidatorImpl#deriveAlias
IgnitePlanner#validateAndGetTypeMetadata ->
{noformat}
public ValidationResult validateAndGetTypeMetadata(SqlNode sqlNode) {
SqlNode validatedNode = validator().validate(sqlNode);
RelDataType type = validator().getValidatedNodeType(validatedNode);
List> origins = validator().getFieldOrigins(validatedNode);
List derived = Collections.emptyList();
if (sqlNode instanceof SqlSelect) {
SelectScope list = validator().getRawSelectScope((SqlSelect)
sqlNode);
assert type.getFieldList().size() ==
list.getExpandedSelectList().size();
int cnt = 0;
derived = new ArrayList<>(list.getExpandedSelectList().size());
for (SqlNode node : list.getExpandedSelectList()) {
derived.add(validator().deriveAlias(node, cnt++));
}
}
return new ValidationResult(validatedNode, type, origins, derived);
}
{noformat}
and use this derived instead of aliases here:
PrepareServiceImpl#resultSetMetadata
*Be careful !* .net and cpp tests are involved too, check all issue number
occurrences.
was:
After upgrading to calcite 1.34 SqlValidator#deriveAlias and overloading call
IgniteSqlValidator#deriveAlias is changed, thus requests like :
select v / 2 from t, return EXPR$ instead of column name derived from
IgniteSqlValidator#deriveAlias. Fast (near) fix looks like cover both such
queries:
IgnitePlanner#validateAndGetTypeMetadata ->
{noformat}
public ValidationResult validateAndGetTypeMetadata(SqlNode sqlNode) {
SqlNode validatedNode = validator().validate(sqlNode);
RelDataType type = validator().getValidatedNodeType(validatedNode);
List> origins = validator().getFieldOrigins(validatedNode);
List derived = Collections.emptyList();
if (sqlNode instanceof SqlSelect) {
SelectScope list = validator().getRawSelectScope((SqlSelect)
sqlNode);
assert type.getFieldList().size() ==
list.getExpandedSelectList().size();
int cnt = 0;
derived = new ArrayList<>(list.getExpandedSelectList().size());
for (SqlNode node : list.getExpandedSelectList()) {
derived.add(validator().deriveAlias(node, cnt++));
}
}
return new ValidationResult(validatedNode, type, origins, derived);
}
{noformat}
and use this derived instead of aliases here:
PrepareServiceImpl#resultSetMetadata
*Be careful !* .net and cpp tests are involved too, check all issue number
occurrences.
> Sql. Column namings are partially broken after upgrading to calcite 1.34
>
>
> Key: IGNITE-19106
> URL: https://issues.apache.org/jira/browse/IGNITE-19106
> Project: Ignite
> Issue Type: Improvement
> Components: sql
>Affects Versions: 3.0.0-beta1
>Reporter: Evgeny Stanilovsky
>Assignee: Evgeny Stanilovsky
>Priority: Major
> Labels: calcite3-required, ignite-3
>
> After upgrading to calcite 1.34 SqlValidator#deriveAlias and overloading call
> IgniteSqlValidator#deriveAlias is changed, thus requests like :
> select v / 2 from t, return EXPR$ instead of column name derived from
> IgniteSqlValidator#deriveAlias. Fast (near) fix looks like cover both such
> queries:
> Check annotation for: SqlValidatorImpl#deriveAlias
> IgnitePlanner#validateAndGetTypeMetadata ->
> {noformat}
> public ValidationResult validateAndGetTypeMetadata(SqlNode sqlNode) {
> SqlNode validatedNode = validator().validate(sqlNode);
> RelDataType type = validator().getValidatedNodeType(validatedNode);
> List> origins =
> validator().getFieldOrigins(validatedNode);
> List derived = Collections.emptyList();
> if (sqlNode instanceof SqlSelect) {
> SelectScope list = validator().getRawSelectScope((SqlSelect)
> sqlNode);
> assert type.getFieldList().size() ==
> list.getExpandedSelectList().size();
> int cnt = 0;
> derived = new ArrayList<>(list.getExpandedSelectList().size());
> for (SqlNode node : list.getExpandedSelectList()) {
> derived.add(validator().deriveAlias(node, cnt++));
> }
> }
> return new ValidationResult(v
[jira] [Updated] (IGNITE-20373) Fix IO threading model
[ https://issues.apache.org/jira/browse/IGNITE-20373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Scherbakov updated IGNITE-20373: --- Description: Currently IO is resubmitted to inboundExecutor for further processing (even there are corner cases then a message handler called in IO thread) It makes message processing essentially single threaded and introduces additional lag due to message transition between multiple queues. addMessageHandler should be extended with a 3-d argument: a pool for submitting a callback for execution, or an executorSelector like in jraft. inboundExecutor should be changed to striped, use more than one thread, and serve messages without explicit executor. Delivery guaranties should be preserved: if a message A is send before B, B can't be processed on a receiver before A. A stripe is defined by sender-receiver pair (or can be user defined - TBD) outboundExecutor also looks like a contention point - need to be addressed as well was: Currently IO is resubmitted to inboundExecutor for further processing (even there are corner cases then a message handler called in IO thread) It makes message processing essentially single threaded and introduces additional lag due to message transition between multiple queues. addMessageHandler should be extended with a 3-d argument: a pool for submitting a callback for execution, or an executorSelector like in jraft. inboundExecutor should be changed to striped, use more than one thread, and serve messages without explicit executor. Delivery guaranties should be preserved: if a message A is send before B, B can't be processed on a receiver before A outboundExecutor also looks like a contention point - need to be addressed as well > Fix IO threading model > -- > > Key: IGNITE-20373 > URL: https://issues.apache.org/jira/browse/IGNITE-20373 > Project: Ignite > Issue Type: Improvement >Affects Versions: 3.0 >Reporter: Alexey Scherbakov >Priority: Major > Labels: ignite-3, ignite3_performance > Fix For: 3.0 > > > Currently IO is resubmitted to inboundExecutor for further processing (even > there are corner cases then a message handler called in IO thread) > It makes message processing essentially single threaded and introduces > additional lag due to message transition between multiple queues. > addMessageHandler should be extended with a 3-d argument: a pool for > submitting a callback for execution, or an executorSelector like in jraft. > inboundExecutor should be changed to striped, use more than one thread, and > serve messages without explicit executor. Delivery guaranties should be > preserved: if a message A is send before B, B can't be processed on a > receiver before A. A stripe is defined by sender-receiver pair (or can be > user defined - TBD) > outboundExecutor also looks like a contention point - need to be addressed as > well -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-20373) Fix IO threading model
[ https://issues.apache.org/jira/browse/IGNITE-20373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Scherbakov updated IGNITE-20373: --- Description: Currently IO is resubmitted to inboundExecutor for further processing (even there are corner cases then a message handler called in IO thread) It makes message processing essentially single threaded and introduces additional lag due to message transition to additional queue. addMessageHandler should be extended with a 3-d argument: a pool for submitting a callback for execution, or an executorSelector like in jraft. inboundExecutor should be changed to striped, use more than one thread, and serve messages without explicit executor. Delivery guaranties should be preserved: if a message A is send before B, B can't be processed on a receiver before A. A stripe is defined by sender-receiver pair (or can be user defined - TBD) outboundExecutor also looks like a contention point - need to be addressed as well was: Currently IO is resubmitted to inboundExecutor for further processing (even there are corner cases then a message handler called in IO thread) It makes message processing essentially single threaded and introduces additional lag due to message transition between multiple queues. addMessageHandler should be extended with a 3-d argument: a pool for submitting a callback for execution, or an executorSelector like in jraft. inboundExecutor should be changed to striped, use more than one thread, and serve messages without explicit executor. Delivery guaranties should be preserved: if a message A is send before B, B can't be processed on a receiver before A. A stripe is defined by sender-receiver pair (or can be user defined - TBD) outboundExecutor also looks like a contention point - need to be addressed as well > Fix IO threading model > -- > > Key: IGNITE-20373 > URL: https://issues.apache.org/jira/browse/IGNITE-20373 > Project: Ignite > Issue Type: Improvement >Affects Versions: 3.0 >Reporter: Alexey Scherbakov >Priority: Major > Labels: ignite-3, ignite3_performance > Fix For: 3.0 > > > Currently IO is resubmitted to inboundExecutor for further processing (even > there are corner cases then a message handler called in IO thread) > It makes message processing essentially single threaded and introduces > additional lag due to message transition to additional queue. > addMessageHandler should be extended with a 3-d argument: a pool for > submitting a callback for execution, or an executorSelector like in jraft. > inboundExecutor should be changed to striped, use more than one thread, and > serve messages without explicit executor. Delivery guaranties should be > preserved: if a message A is send before B, B can't be processed on a > receiver before A. A stripe is defined by sender-receiver pair (or can be > user defined - TBD) > outboundExecutor also looks like a contention point - need to be addressed as > well -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-18302) ignite-spring-sessions: IgniteSession serialization drags its parent class
[
https://issues.apache.org/jira/browse/IGNITE-18302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17762738#comment-17762738
]
Alexandr Shapkin commented on IGNITE-18302:
---
[~anovikov] LGTM in general, please fix a couple of minor things in PR before
merging.
> ignite-spring-sessions: IgniteSession serialization drags its parent class
> --
>
> Key: IGNITE-18302
> URL: https://issues.apache.org/jira/browse/IGNITE-18302
> Project: Ignite
> Issue Type: Bug
> Components: extensions
>Affects Versions: 2.15
>Reporter: Alexandr Shapkin
>Assignee: Andrey Novikov
>Priority: Major
> Fix For: 2.16
>
>
> In short, there is a bug in ignite-spring-session-ext implementation.
> We store IgniteIndexedSessionRepository${{{}IgniteSession{}}} in the cache,
> but it’s an internal non-static class, having a reference to the parent
> [\{{{}IgniteIndexedSessionRepository{}}}] indirectly.
> Hence, during the serialization Ignite also writes {{{}name=this$0,
> type=Object, fieldId=0xCBDD23AA (-874699862){}}}, which is the reference to
> {{{}IgniteIndexedSessionRepository{}}}. That leads to the following issues:
> * we are serializing and saving internal utility data, like}} Ignite
> ignite{}}}, {{private IndexResolver indexResolver}} etc
> * one of the IgniteIndexedSessionRepository’s fields is IgniteCache itself -
> {{IgniteCache sessions}} that basically keeps every
> session so far leading to a StackOverflow error after some time.
>
> {code:java}
> [2022-11-25T17:27:29,268][ERROR][sys-stripe-0-#1%USERS_IGNITE%][GridCacheIoManager]
> Failed processing message [senderId=0f0ca915-d6cd-4580-92a3-1fbc3d2a5722,
> msg=GridNearSingleGetResponse [futId=1669397231378, res=-547701325,
> topVer=null, err=null, flags=0]] 2java.lang.StackOverflowError: null 3 at
> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshallerUtils.descriptorFromCache(OptimizedMarshallerUtils.java:328)
> ~[ignite-core-8.8.22.jar:8.8.22] 4 at
> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshallerUtils.classDescriptor(OptimizedMarshallerUtils.java:273)
> ~[ignite-core-8.8.22.jar:8.8.22] 5 at
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:354)
> ~[ignite-core-8.8.22.jar:8.8.22] 6 at
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:211)
> ~[ignite-core-8.8.22.jar:8.8.22] 7 at
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:480) ~[?:?] 8 at
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:447) ~[?:?] 9 at
> org.apache.ignite.internal.processors.cache.GridCacheProxyImpl.readExternal(GridCacheProxyImpl.java:1662)
> ~[ignite-core-8.8.22.jar:8.8.22] 10 at
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:569)
> ~[ignite-core-8.8.22.jar:8.8.22] 11 at
> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:979)
> ~[ignite-core-8.8.22.jar:8.8.22] 12 at
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:359)
> ~[ignite-core-8.8.22.jar:8.8.22] 13 at
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:211)
> ~[ignite-core-8.8.22.jar:8.8.22] 14 at
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:480) ~[?:?] 15 at
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:447) ~[?:?] 16...
> 17[2022-11-25T17:27:29,276][ERROR][sys-stripe-0-#1%USERS_IGNITE%][] Critical
> system error detected. Will be handled accordingly to configured handler
> [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
> super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
> [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
> failureCtx=FailureContext [type=CRITICAL_ERROR,
> err=java.lang.StackOverflowError]]{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-20379) Concurrency issues with write intent tracking
[
https://issues.apache.org/jira/browse/IGNITE-20379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirill Sizov updated IGNITE-20379:
---
Labels: ignite-3 (was: )
> Concurrency issues with write intent tracking
> -
>
> Key: IGNITE-20379
> URL: https://issues.apache.org/jira/browse/IGNITE-20379
> Project: Ignite
> Issue Type: Bug
>Reporter: Kirill Sizov
>Assignee: Kirill Sizov
>Priority: Major
> Labels: ignite-3
>
> There are a few concurrency issues with the write intent tracking code
> 1. See {{PendingRows}} class. The value is a TreeSet, which is not a
> concurrent collection. Thus it's not safe to do the read this way:
> {code}
> public Set getPendingRowIds(UUID txId) {
> return txsPendingRowIds.getOrDefault(txId, EMPTY_SET);
> }
> {code}
> 2. See {{StorageUpdateHandler.handleTransactionCleanup}}. We first get a
> collection of rows for a specific transaction from the map, then process the
> rows in the storage, then remove the whole transaction entry from the map:
> {code}
> public void handleTransactionCleanup(UUID txId, boolean commit, @Nullable
> HybridTimestamp commitTimestamp) {
> Set pendingRowIds = pendingRows.getPendingRowIds(txId);
> handleTransactionCleanup(pendingRowIds, commit, commitTimestamp, ()
> -> pendingRows.removePendingRowIds(txId));
> }
> {code}
> If a parallel thread adds more write intents to the map, we'll silently lose
> them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (IGNITE-20379) Concurrency issues with write intent tracking
[
https://issues.apache.org/jira/browse/IGNITE-20379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirill Sizov reassigned IGNITE-20379:
--
Assignee: Kirill Sizov
> Concurrency issues with write intent tracking
> -
>
> Key: IGNITE-20379
> URL: https://issues.apache.org/jira/browse/IGNITE-20379
> Project: Ignite
> Issue Type: Bug
>Reporter: Kirill Sizov
>Assignee: Kirill Sizov
>Priority: Major
>
> There are a few concurrency issues with the write intent tracking code
> 1. See {{PendingRows}} class. The value is a TreeSet, which is not a
> concurrent collection. Thus it's not safe to do the read this way:
> {code}
> public Set getPendingRowIds(UUID txId) {
> return txsPendingRowIds.getOrDefault(txId, EMPTY_SET);
> }
> {code}
> 2. See {{StorageUpdateHandler.handleTransactionCleanup}}. We first get a
> collection of rows for a specific transaction from the map, then process the
> rows in the storage, then remove the whole transaction entry from the map:
> {code}
> public void handleTransactionCleanup(UUID txId, boolean commit, @Nullable
> HybridTimestamp commitTimestamp) {
> Set pendingRowIds = pendingRows.getPendingRowIds(txId);
> handleTransactionCleanup(pendingRowIds, commit, commitTimestamp, ()
> -> pendingRows.removePendingRowIds(txId));
> }
> {code}
> If a parallel thread adds more write intents to the map, we'll silently lose
> them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IGNITE-20379) Concurrency issues with write intent tracking
Kirill Sizov created IGNITE-20379:
--
Summary: Concurrency issues with write intent tracking
Key: IGNITE-20379
URL: https://issues.apache.org/jira/browse/IGNITE-20379
Project: Ignite
Issue Type: Bug
Reporter: Kirill Sizov
There are a few concurrency issues with the write intent tracking code
1. See {{PendingRows}} class. The value is a TreeSet, which is not a concurrent
collection. Thus it's not safe to do the read this way:
{code}
public Set getPendingRowIds(UUID txId) {
return txsPendingRowIds.getOrDefault(txId, EMPTY_SET);
}
{code}
2. See {{StorageUpdateHandler.handleTransactionCleanup}}. We first get a
collection of rows for a specific transaction from the map, then process the
rows in the storage, then remove the whole transaction entry from the map:
{code}
public void handleTransactionCleanup(UUID txId, boolean commit, @Nullable
HybridTimestamp commitTimestamp) {
Set pendingRowIds = pendingRows.getPendingRowIds(txId);
handleTransactionCleanup(pendingRowIds, commit, commitTimestamp, () ->
pendingRows.removePendingRowIds(txId));
}
{code}
If a parallel thread adds more write intents to the map, we'll silently lose
them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-17578) Transactions: async cleanup processing on tx commit
[
https://issues.apache.org/jira/browse/IGNITE-17578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17762769#comment-17762769
]
Alexander Lapin commented on IGNITE-17578:
--
[~Denis Chudov] LGTM
> Transactions: async cleanup processing on tx commit
> ---
>
> Key: IGNITE-17578
> URL: https://issues.apache.org/jira/browse/IGNITE-17578
> Project: Ignite
> Issue Type: Improvement
>Reporter: Alexander Lapin
>Assignee: Denis Chudov
>Priority: Major
> Labels: ignite-3, transaction, transaction3_recovery
>
> h3. Motivation
> Within RW transaction commit process, according to the tx design it's
> required to return the control to the outer logic right after
> # COMMITED/ABORTED txn state replication
> # Locks release.
> Follow-up cleanup process, that will apply or remove write intents, should be
> asynchronous. Currently, it's not true. It also worth to mention that
> currently, locks are released after write intent application. That should be
> inverted. Such enhancements mean that not only RO but also RW transactions
> may retrieve writeIntent and thus perform writeInentResolution - it's covered
> with separate ticket https://issues.apache.org/jira/browse/IGNITE-19570 that
> should be implemented prior to this one.
> h3. Definition of Done
> * Write intent application or removal should be implemented in an async
> format.
> * Write intent application and locks release process should be inverted.
> h3. Implementation Notes
> Generally it's only required to change some code in
> org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener#processTxCleanupAction
> {code:java}
> return allOffFuturesExceptionIgnored(txUpdateFutures,
> request).thenCompose(v -> {
> TxCleanupCommand txCleanupCmd = MSG_FACTORY.txCleanupCommand()
> .txId(request.txId())
> .commit(request.commit())
> .commitTimestampLong(request.commitTimestampLong())
> .safeTimeLong(hybridClock.nowLong())
> .build();
> return raftClient
> .run(txCleanupCmd)
> .thenCompose(ignored ->
> allOffFuturesExceptionIgnored(txReadFutures, request)
> .thenRun(() -> releaseTxLocks(request.txId(;
> });
> } {code}
> * releaseTxLocks priour to sending txCleanupCmd.
> * send txCleanupCmd in async manner, meaning that processTxCleanupAction
> should return the result after locks are released.
> It seems that durable writeIntentApplication, the process that is triggered
> by sending txCleanupCmd will be durable because of raftClient inner
> implementation, apparently new topologyAwareRaftClient and special recovery
> procedures on primary re-election that will be covered by separate ticket, so
> nothing to do here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IGNITE-20380) Try to deal with TableManager#assignmentsChangeListeners
Kirill Tkalenko created IGNITE-20380: Summary: Try to deal with TableManager#assignmentsChangeListeners Key: IGNITE-20380 URL: https://issues.apache.org/jira/browse/IGNITE-20380 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko *org.apache.ignite.internal.table.distributed.TableManager#assignmentsChangeListeners* that are used in *org.apache.ignite.client.handler.ClientInboundMessageHandler* have been discovered and I would like to see if it would be possible to change these listeners to catalog listeners for example or something like that. And also, most likely, the call of listeners should be made after writing to the metastore. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (IGNITE-20333) Sql. Backport processing DEFAULT constraints into CALCITE code.
[ https://issues.apache.org/jira/browse/IGNITE-20333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Evgeny Stanilovsky resolved IGNITE-20333. - Resolution: Fixed > Sql. Backport processing DEFAULT constraints into CALCITE code. > --- > > Key: IGNITE-20333 > URL: https://issues.apache.org/jira/browse/IGNITE-20333 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 3.0.0-beta1 >Reporter: Evgeny Stanilovsky >Assignee: Evgeny Stanilovsky >Priority: Major > Labels: ignite-3 > > It would be helpful to backport _default_ constraints processing into calcite > code base. > Appropriate issue [1] > [1] CALCITE-5950 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-20381) Add busyLock usage in CatalogManagerImpl
Kirill Tkalenko created IGNITE-20381: Summary: Add busyLock usage in CatalogManagerImpl Key: IGNITE-20381 URL: https://issues.apache.org/jira/browse/IGNITE-20381 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko It was found that the *IgniteSpinBusyLock* is not used in *org.apache.ignite.internal.catalog.CatalogManagerImpl*, this needs to be fixed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-20382) Calcite engine. Add metrics for CalciteQueryExecutor thread pool
Aleksey Plekhanov created IGNITE-20382: -- Summary: Calcite engine. Add metrics for CalciteQueryExecutor thread pool Key: IGNITE-20382 URL: https://issues.apache.org/jira/browse/IGNITE-20382 Project: Ignite Issue Type: Improvement Reporter: Aleksey Plekhanov Assignee: Aleksey Plekhanov Currently, all thread-pools except CalciteQueryExecutor can be monitored via metrics, we should add this ability for CalciteQueryExecutor too. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-19938) Add validation for columns parameters in Catalog.
[
https://issues.apache.org/jira/browse/IGNITE-19938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Evgeny Stanilovsky reassigned IGNITE-19938:
---
Assignee: Evgeny Stanilovsky
> Add validation for columns parameters in Catalog.
> -
>
> Key: IGNITE-19938
> URL: https://issues.apache.org/jira/browse/IGNITE-19938
> Project: Ignite
> Issue Type: Improvement
> Components: sql
>Reporter: Andrey Mashenkov
>Assignee: Evgeny Stanilovsky
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> ColumnParams builder is not restrictive at all, and allows to create columns
> with invalid parameters.
> E.g. some types requires precision and/or scale, while this make no sense for
> the others.
> Some of our tests allows "undefined" values for precision/scale/length for
> any column type, which passes to the column descriptors as special value
> (zero) with no reason. They should be fixed as well.
> Also, we have multiple places where default and max prevision and scale are
> defined: in catalog, in schema (see NativeType) and in sql-engine modules.
> Let's make a single point of truth.
> Let's add parameters validation to builder and check precision/scale/length
> are valid values, regarding the actual column type, SQL standard document and
> that we may need different validation strategy for CREATE and ALTER commands.
> Startpoint is
> {code:java}
> CatalogUtils.fromParams(ColumnParams params)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (IGNITE-20226) Sql. Some tests are muted after casting char -> varbinary is disallowed
[
https://issues.apache.org/jira/browse/IGNITE-20226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Evgeny Stanilovsky reassigned IGNITE-20226:
---
Assignee: Evgeny Stanilovsky
> Sql. Some tests are muted after casting char -> varbinary is disallowed
> ---
>
> Key: IGNITE-20226
> URL: https://issues.apache.org/jira/browse/IGNITE-20226
> Project: Ignite
> Issue Type: Improvement
> Components: sql
>Affects Versions: 3.0.0-beta1
>Reporter: Evgeny Stanilovsky
>Assignee: Evgeny Stanilovsky
>Priority: Major
> Labels: ignite-3
>
> Some tests was muted under this issue, some refactoring is helpful.
> Because BINARY/VARBINARY is not convertible from any other types, the
> following test cases should be disabled or moved to appropriate subclasses
> {code}
> testCoalesceMissingTypesIsIllegal
> testEqConditionWithDynamicParameters
> testInsertFromDynamicParameterFromConvertible
> testDisallowMismatchTypesOnInsertDynamicParam
> testDisallowMismatchTypesOnInsert
> testUpdateFromDynamicParameterFromConvertible
> {code}
> `testCastFromString` can simply be moved to `ItUuidExpressionTest`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-20335) Move IgniteRunner to the internal package
[ https://issues.apache.org/jira/browse/IGNITE-20335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vyacheslav Koptilin updated IGNITE-20335: - Description: It is not assumed that the -IgniteRunner- can be used directly, but this class is placed into `org.apache.ignite` package. So, it totally makes sense to move it to the internal one. Definition of Done: - -IgniteRunner- moved to `org.apache.ignite.internal.app` package - build and packaging scripts are updated in accordance with #1 - removed outdated TODO in the -IgniteRunnerTest- : /** TODO: Replace this test by full integration test on the cli side IGNITE-15097. */ was: It is not assumed that the -IgniteRunner- can be used directly, but this class is placed into `org.apache.ignite` package. So, it totally makes sense to move it to the internal one. Definition of Done: - -IgniteRunner- moved to -org.apache.ignite.internal.app` package - build and packaging scripts are updated in accordance with #1 - removed outdated TODO in the -IgniteRunnerTest- : /** TODO: Replace this test by full integration test on the cli side IGNITE-15097. */ > Move IgniteRunner to the internal package > - > > Key: IGNITE-20335 > URL: https://issues.apache.org/jira/browse/IGNITE-20335 > Project: Ignite > Issue Type: Bug >Reporter: Vyacheslav Koptilin >Assignee: Vyacheslav Koptilin >Priority: Major > Labels: ignite-3 > > It is not assumed that the -IgniteRunner- can be used directly, but this > class is placed into `org.apache.ignite` package. So, it totally makes sense > to move it to the internal one. > Definition of Done: > - -IgniteRunner- moved to `org.apache.ignite.internal.app` package > - build and packaging scripts are updated in accordance with #1 > - removed outdated TODO in the -IgniteRunnerTest- : /** TODO: Replace this > test by full integration test on the cli side IGNITE-15097. */ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-20335) Move IgniteRunner to the internal package
[ https://issues.apache.org/jira/browse/IGNITE-20335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vyacheslav Koptilin updated IGNITE-20335: - Description: It is not assumed that the _IgniteRunner_ can be used directly, but this class is placed into `org.apache.ignite` package. So, it totally makes sense to move it to the internal one. Definition of Done: - _IgniteRunner_ moved to `org.apache.ignite.internal.app` package - build and packaging scripts are updated in accordance with #1 - removed outdated TODO in the _IgniteRunnerTest_ : /** TODO: Replace this test by full integration test on the cli side IGNITE-15097. */ was: It is not assumed that the -IgniteRunner- can be used directly, but this class is placed into `org.apache.ignite` package. So, it totally makes sense to move it to the internal one. Definition of Done: - -IgniteRunner- moved to `org.apache.ignite.internal.app` package - build and packaging scripts are updated in accordance with #1 - removed outdated TODO in the -IgniteRunnerTest- : /** TODO: Replace this test by full integration test on the cli side IGNITE-15097. */ > Move IgniteRunner to the internal package > - > > Key: IGNITE-20335 > URL: https://issues.apache.org/jira/browse/IGNITE-20335 > Project: Ignite > Issue Type: Bug >Reporter: Vyacheslav Koptilin >Assignee: Vyacheslav Koptilin >Priority: Major > Labels: ignite-3 > > It is not assumed that the _IgniteRunner_ can be used directly, but this > class is placed into `org.apache.ignite` package. So, it totally makes sense > to move it to the internal one. > Definition of Done: > - _IgniteRunner_ moved to `org.apache.ignite.internal.app` package > - build and packaging scripts are updated in accordance with #1 > - removed outdated TODO in the _IgniteRunnerTest_ : /** TODO: Replace this > test by full integration test on the cli side IGNITE-15097. */ -- This message was sent by Atlassian Jira (v8.20.10#820010)
