[jira] [Resolved] (IGNITE-21739) JDBC connection to a multi-node cluster doesn't take into account clientConnector.port from each node
[
https://issues.apache.org/jira/browse/IGNITE-21739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikita Sivkov resolved IGNITE-21739.
Resolution: Invalid
Actually, the connection string was like
{{jdbc:ignite:thin://172.24.1.2,172.24.1.3,172.24.1.4 and the actual issue is
related to [IGNITE-21577|https://issues.apache.org/jira/browse/IGNITE-21577]}}
> JDBC connection to a multi-node cluster doesn't take into account
> clientConnector.port from each node
> -
>
> Key: IGNITE-21739
> URL: https://issues.apache.org/jira/browse/IGNITE-21739
> Project: Ignite
> Issue Type: Bug
> Components: jdbc
>Affects Versions: 3.0.0-beta2
> Environment: * multi-node cluster
> * different `{{{}clientConnector.port{}}}` on each cluster node
>Reporter: Nikita Sivkov
>Priority: Major
> Labels: ignite-3
> Attachments: exception.log
>
>
> *WHEN* you create a multi-node cluster
> *AND* specify different {color:#de350b}{{clientConnector.port}}{color} on
> each cluster node
> (for example,
> node1 (172.24.1.2) - {color:#de350b}{{clientConnector.port=10800}}{color}
> node2 (172.24.1.3) - {color:#de350b}{{clientConnector.port=10801}}{color}
> node3 (172.24.1.4) - {color:#de350b}{{clientConnector.port=10802}}{color})
> *AND* connect to cluster like
> {color:#de350b}{{{}jdbc:ignite:thin://{node1address{{color} (for example,
> {{{color:#de350b}jdbc:ignite:thin://172.24.1.2{color})}}
> *AND* try to insert a couple of records
> *THEN* you will get an error like
> {code:java}
> Mar 12, 2024 7:37:21 PM org.apache.ignite.internal.logger.IgniteLogger
> warnWARNING: Failed to establish connection to 172.24.1.3:10800:
> org.apache.ignite.client.IgniteClientConnectionException: IGN-CLIENT-1
> TraceId:df21d718-d40c-4506-84e7-6ec141de9ab5 Client failed to connect:
> Connection refused: no further information:
> /172.24.1.3:10800java.util.concurrent.CompletionException:
> org.apache.ignite.client.IgniteClientConnectionException: IGN-CLIENT-1
> TraceId:df21d718-d40c-4506-84e7-6ec141de9ab5 Client failed to connect:
> Connection refused: no further information: /172.24.1.3:10800 at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
>at
> java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1063)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
> at
> org.apache.ignite.internal.client.io.netty.NettyClientConnectionMultiplexer.lambda$openAsync$1(NettyClientConnectionMultiplexer.java:197)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
> at
> io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
> at
> io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629)
> at
> io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118)
> at
> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:322)
> at
> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:338)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
> at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)Caused by:
> org.apache.ignite.client.IgniteClientConnectionException: IGN-CLIENT-1
> TraceId:df21d718-d40c-4506-84e7-6ec141de9ab5 Client failed to connect:
> Connection refused: no further information: /172.24.1.3:10800at
> org.apache.ignite.internal.client.io.netty.NettyClientConnectionMultiplexer.lambda$openAsync$1(NettyClientConnectionMultiplexer.java:194)
> ... 17 m
[jira] [Updated] (IGNITE-21749) Add assertions for destroyed/closed storages behavior
[ https://issues.apache.org/jira/browse/IGNITE-21749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirill Tkalenko updated IGNITE-21749: - Epic Link: IGNITE-20782 > Add assertions for destroyed/closed storages behavior > - > > Key: IGNITE-21749 > URL: https://issues.apache.org/jira/browse/IGNITE-21749 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Assignee: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Also, the number of warnings in AbstractMvTableStorageTest should be reduced. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-19976) Sql: ClassCastException when trying to select from indexed smallint column
[
https://issues.apache.org/jira/browse/IGNITE-19976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Evgeny Stanilovsky updated IGNITE-19976:
Ignite Flags: (was: Docs Required,Release Notes Required)
> Sql: ClassCastException when trying to select from indexed smallint column
> --
>
> Key: IGNITE-19976
> URL: https://issues.apache.org/jira/browse/IGNITE-19976
> Project: Ignite
> Issue Type: Bug
> Components: sql
>Affects Versions: 3.0.0-beta1, 3.0.0-beta2
>Reporter: Andrey Khitrin
>Assignee: Maksim Zhuravkov
>Priority: Major
> Labels: ignite-3, sql
> Fix For: 3.0.0-beta2
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Steps to reproduce:
> {code:sql}
> sql-cli> create table tt(id INTEGER not null, field_1 TINYINT, field_2
> SMALLINT, primary key (id));
> Updated 0 rows.
> sql-cli> SELECT * FROM tt WHERE field_2 = 50;
> ╔╤═╤═╗
> ║ ID │ FIELD_1 │ FIELD_2 ║
> ╠╧═╧═╣
> ║ (empty)║
> ╚╝
> -- select over field_2 works fine until we create an index on it!
> sql-cli> CREATE INDEX tt_idx ON tt (field_2);
> Updated 0 rows.
> sql-cli> SELECT * FROM tt WHERE field_2 = 50;
> SQL query execution error
> Exception while executing query [query=SELECT * FROM tt WHERE field_2 = 50;].
> Error message:Query remote fragment execution failed: nodeName=defaultNode,
> queryId=15a68600-41bc-46c5-b20d-7a96aad15477, fragmentId=4,
> originalMessage=class java.lang.Integer cannot be cast to class
> java.lang.Short (java.lang.Integer and java.lang.Short are in module
> java.base of loader 'bootstrap')
> {code}
> This doesn't look good. A created index must not cause casting exception.
> Reproduced on commit `fef7a24c2a029cac720d2fea3815c2a70a86b72f` (2023-07-12)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-21749) Add assertions for destroyed/closed storages behavior
[ https://issues.apache.org/jira/browse/IGNITE-21749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17826982#comment-17826982 ] Roman Puchkovskiy commented on IGNITE-21749: Thanks! > Add assertions for destroyed/closed storages behavior > - > > Key: IGNITE-21749 > URL: https://issues.apache.org/jira/browse/IGNITE-21749 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Assignee: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Also, the number of warnings in AbstractMvTableStorageTest should be reduced. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-21753) Implement a hacky self-compaction for selected Metastorage keys
Roman Puchkovskiy created IGNITE-21753: -- Summary: Implement a hacky self-compaction for selected Metastorage keys Key: IGNITE-21753 URL: https://issues.apache.org/jira/browse/IGNITE-21753 Project: Ignite Issue Type: Improvement Reporter: Roman Puchkovskiy -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21753) Implement a hacky self-compaction for selected Metastorage keys
[ https://issues.apache.org/jira/browse/IGNITE-21753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Puchkovskiy updated IGNITE-21753: --- Description: To enable Metastorage compaction (that is, removal of overwritten revisions that are not needed to anyone any more), there is IGNITE-19417, but it might take some time to define the corresponding LWM's rules. As a crutch, we might implement a simplified procedure for some keys. For instance, a big percentage of garbage is created by a constantly overwritten key related to lease tracking. If a key is never scanned (either distributively, via range()/prefix(), or locally, via xxxLocally() returning a Cursor) and only its latest version is always accessed (this seems to be true for the above mentioned lease-related key), we can just do compactKey() for such a key after any update involving it. We might also need to reimplement getAndPut() to read before doing a put as the current implementation requires 2 latest versions to be stored (because it makes a put first and only then reads using the 'before the put' revision). > Implement a hacky self-compaction for selected Metastorage keys > --- > > Key: IGNITE-21753 > URL: https://issues.apache.org/jira/browse/IGNITE-21753 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > > To enable Metastorage compaction (that is, removal of overwritten revisions > that are not needed to anyone any more), there is IGNITE-19417, but it might > take some time to define the corresponding LWM's rules. > As a crutch, we might implement a simplified procedure for some keys. For > instance, a big percentage of garbage is created by a constantly overwritten > key related to lease tracking. > If a key is never scanned (either distributively, via range()/prefix(), or > locally, via xxxLocally() returning a Cursor) and only its latest version is > always accessed (this seems to be true for the above mentioned lease-related > key), we can just do compactKey() for such a key after any update involving > it. > We might also need to reimplement getAndPut() to read before doing a put as > the current implementation requires 2 latest versions to be stored (because > it makes a put first and only then reads using the 'before the put' revision). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21753) Implement a hacky self-compaction for selected Metastorage keys
[ https://issues.apache.org/jira/browse/IGNITE-21753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Puchkovskiy updated IGNITE-21753: --- Description: To enable Metastorage compaction (that is, removal of overwritten revisions that are not needed to anyone any more), there is IGNITE-19417, but it might take some time to define the corresponding LWM's rules. As a crutch, we might implement a simplified procedure for some keys. For instance, a big percentage of garbage is created by a constantly overwritten key related to lease tracking. If a key is never scanned (either distributively, via range()/prefix(), or locally, via xxxLocally() returning a Cursor) and only its latest version is always accessed (this seems to be true for the above mentioned lease-related key), we can just do compactKey() for such a key after any update involving it. We might also need to reimplement getAndPut() to read before doing a put as the current implementation requires 2 latest versions to be stored (because it makes a put first and only then reads using the 'before the put' revision). The crutch might be implemented by whitelisting all such 'special' keys in the KV storage backing the Metastorage to make it opaque for the API, or an API might be temporarily changed. was: To enable Metastorage compaction (that is, removal of overwritten revisions that are not needed to anyone any more), there is IGNITE-19417, but it might take some time to define the corresponding LWM's rules. As a crutch, we might implement a simplified procedure for some keys. For instance, a big percentage of garbage is created by a constantly overwritten key related to lease tracking. If a key is never scanned (either distributively, via range()/prefix(), or locally, via xxxLocally() returning a Cursor) and only its latest version is always accessed (this seems to be true for the above mentioned lease-related key), we can just do compactKey() for such a key after any update involving it. We might also need to reimplement getAndPut() to read before doing a put as the current implementation requires 2 latest versions to be stored (because it makes a put first and only then reads using the 'before the put' revision). > Implement a hacky self-compaction for selected Metastorage keys > --- > > Key: IGNITE-21753 > URL: https://issues.apache.org/jira/browse/IGNITE-21753 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > > To enable Metastorage compaction (that is, removal of overwritten revisions > that are not needed to anyone any more), there is IGNITE-19417, but it might > take some time to define the corresponding LWM's rules. > As a crutch, we might implement a simplified procedure for some keys. For > instance, a big percentage of garbage is created by a constantly overwritten > key related to lease tracking. > If a key is never scanned (either distributively, via range()/prefix(), or > locally, via xxxLocally() returning a Cursor) and only its latest version is > always accessed (this seems to be true for the above mentioned lease-related > key), we can just do compactKey() for such a key after any update involving > it. > We might also need to reimplement getAndPut() to read before doing a put as > the current implementation requires 2 latest versions to be stored (because > it makes a put first and only then reads using the 'before the put' revision). > The crutch might be implemented by whitelisting all such 'special' keys in > the KV storage backing the Metastorage to make it opaque for the API, or an > API might be temporarily changed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21753) Implement a hacky self-compaction for selected Metastorage keys
[ https://issues.apache.org/jira/browse/IGNITE-21753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Puchkovskiy updated IGNITE-21753: --- Description: To enable Metastorage compaction (that is, removal of overwritten revisions that are not needed to anyone any more), there is IGNITE-19417, but it might take some time to define the corresponding LWM's rules. As a crutch, we might implement a simplified procedure for some keys. For instance, a big percentage of garbage is created by a constantly overwritten key related to lease tracking. If a key is never scanned (either distributively, via range()/prefix(), or locally, via xxxLocally() returning a Cursor) and only its latest version is always accessed (this seems to be true for the above mentioned lease-related key), we can just do compactKey() for such a key after any update involving it. We might also need to reimplement getAndPut() to read before doing a put as the current implementation requires 2 latest versions to be stored (because it makes a put first and only then reads using the 'before the put' revision). The crutch might be implemented by whitelisting all such 'special' keys in the KV storage backing the Metastorage to make it opaque for the API, or an API might be temporarily changed. Anyway, a detailed design is needed. was: To enable Metastorage compaction (that is, removal of overwritten revisions that are not needed to anyone any more), there is IGNITE-19417, but it might take some time to define the corresponding LWM's rules. As a crutch, we might implement a simplified procedure for some keys. For instance, a big percentage of garbage is created by a constantly overwritten key related to lease tracking. If a key is never scanned (either distributively, via range()/prefix(), or locally, via xxxLocally() returning a Cursor) and only its latest version is always accessed (this seems to be true for the above mentioned lease-related key), we can just do compactKey() for such a key after any update involving it. We might also need to reimplement getAndPut() to read before doing a put as the current implementation requires 2 latest versions to be stored (because it makes a put first and only then reads using the 'before the put' revision). The crutch might be implemented by whitelisting all such 'special' keys in the KV storage backing the Metastorage to make it opaque for the API, or an API might be temporarily changed. > Implement a hacky self-compaction for selected Metastorage keys > --- > > Key: IGNITE-21753 > URL: https://issues.apache.org/jira/browse/IGNITE-21753 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > > To enable Metastorage compaction (that is, removal of overwritten revisions > that are not needed to anyone any more), there is IGNITE-19417, but it might > take some time to define the corresponding LWM's rules. > As a crutch, we might implement a simplified procedure for some keys. For > instance, a big percentage of garbage is created by a constantly overwritten > key related to lease tracking. > If a key is never scanned (either distributively, via range()/prefix(), or > locally, via xxxLocally() returning a Cursor) and only its latest version is > always accessed (this seems to be true for the above mentioned lease-related > key), we can just do compactKey() for such a key after any update involving > it. > We might also need to reimplement getAndPut() to read before doing a put as > the current implementation requires 2 latest versions to be stored (because > it makes a put first and only then reads using the 'before the put' revision). > The crutch might be implemented by whitelisting all such 'special' keys in > the KV storage backing the Metastorage to make it opaque for the API, or an > API might be temporarily changed. > Anyway, a detailed design is needed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-21678) IgniteToIgniteCdcStreamer fails if BinaryConfiguration is specified in config
[
https://issues.apache.org/jira/browse/IGNITE-21678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17826989#comment-17826989
]
Ignite TC Bot commented on IGNITE-21678:
{panel:title=Branch: [pull/11275/head] Base: [master] : No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
{panel:title=Branch: [pull/11275/head] Base: [master] : New Tests
(2)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}
{color:#8b}PDS 2{color} [[tests
1|https://ci2.ignite.apache.org/viewLog.html?buildId=7783486]]
* {color:#013220}IgnitePdsTestSuite2:
CdcPushMetricsExporterTest.testIgniteToIgniteConsumer - PASSED{color}
{color:#8b}Disk Page Compressions 2{color} [[tests
1|https://ci2.ignite.apache.org/viewLog.html?buildId=7783530]]
* {color:#013220}IgnitePdsCompressionTestSuite2:
CdcPushMetricsExporterTest.testIgniteToIgniteConsumer - PASSED{color}
{panel}
[TeamCity *--> Run :: All*
Results|https://ci2.ignite.apache.org/viewLog.html?buildId=7783534&buildTypeId=IgniteTests24Java8_RunAll]
> IgniteToIgniteCdcStreamer fails if BinaryConfiguration is specified in config
> -
>
> Key: IGNITE-21678
> URL: https://issues.apache.org/jira/browse/IGNITE-21678
> Project: Ignite
> Issue Type: Bug
>Reporter: Sergey Korotkov
>Assignee: Sergey Korotkov
>Priority: Major
> Labels: ise
> Fix For: 2.17
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The below exception occurs if destination cluster has the explicit
> BinaryConfiguration instance included into the IgniteConfiguration:
> {noformat}
> 2024-03-04T12:14:31,777][INFO ][Thread-0][] Ignite Change Data Capture
> Application stopped.
> [2024-03-04T12:14:31,782][ERROR][Thread-0][] Cdc error
> org.apache.ignite.IgniteException: Failed to start manager:
> GridManagerAdapter [enabled=true,
> name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
> at
> org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1163)
> ~[classes/:?]
> at org.apache.ignite.Ignition.start(Ignition.java:303) ~[classes/:?]
> at
> org.apache.ignite.cdc.IgniteToIgniteCdcStreamer.start(IgniteToIgniteCdcStreamer.java:86)
> ~[ignite-cdc-ext-1.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.cdc.WalRecordsConsumer.start(WalRecordsConsumer.java:192)
> ~[classes/:?]
> at org.apache.ignite.internal.cdc.CdcMain.runX(CdcMain.java:360)
> ~[classes/:?]
> at org.apache.ignite.internal.cdc.CdcMain.run(CdcMain.java:306)
> [classes/:?]
> at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
> Caused by: org.apache.ignite.IgniteCheckedException: Failed to start manager:
> GridManagerAdapter [enabled=true,
> name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
> at
> org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1777)
> ~[classes/:?]
> at
> org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1159)
> ~[classes/:?]
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1725)
> ~[classes/:?]
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1647)
> ~[classes/:?]
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1089)
> ~[classes/:?]
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:625)
> ~[classes/:?]
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:547)
> ~[classes/:?]
> at org.apache.ignite.Ignition.start(Ignition.java:300) ~[classes/:?]
> ... 5 more
> Caused by: org.apache.ignite.IgniteCheckedException: Failed to start SPI:
> TcpDiscoverySpi [addrRslvr=null, addressFilter=null, sockTimeout=5000,
> ackTimeout=5000, marsh=JdkMarshaller
> [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@5f243268],
> reconCnt=10, reconDelay=2000, maxAckTimeout=60, soLinger=0,
> forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null,
> skipAddrsRandomization=false]
> at
> org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:280)
> ~[classes/:?]
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:1072)
> ~[classes/:?]
> at
> org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1772)
> ~[classes/:?]
> at
> org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1159)
> ~[classes/:?]
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1725)
> ~[classes/:?]
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstan
[jira] [Created] (IGNITE-21754) Remove destroyed tables on recovery
Kirill Tkalenko created IGNITE-21754: Summary: Remove destroyed tables on recovery Key: IGNITE-21754 URL: https://issues.apache.org/jira/browse/IGNITE-21754 Project: Ignite Issue Type: Improvement Reporter: Kirill Tkalenko Fix For: 3.0.0-beta2 We need to destroy tables on node recovery that fall under the low watermark for all storage engines. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-20009) Sql. Rework 2-phase aggregates part 2. AVG as SUM / COUNT.
[ https://issues.apache.org/jira/browse/IGNITE-20009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maksim Zhuravkov reassigned IGNITE-20009: - Assignee: Maksim Zhuravkov > Sql. Rework 2-phase aggregates part 2. AVG as SUM / COUNT. > -- > > Key: IGNITE-20009 > URL: https://issues.apache.org/jira/browse/IGNITE-20009 > Project: Ignite > Issue Type: Improvement > Components: sql >Affects Versions: 3.0.0-beta1 >Reporter: Maksim Zhuravkov >Assignee: Maksim Zhuravkov >Priority: Minor > Labels: ignite-3 > Time Spent: 10m > Remaining Estimate: 0h > > Map/reduce implementation of some aggregates requires replacing an aggregate > with a combination of aggregates. For example AVG(col) should be implemented > as > MAP: SUM(col) sum_col, COUNT (col) as count_col > REDUCE: SUM(sum_col)/SUM(COUNT(count_col)) > To implement AVG for two phase aggregation we need the following changes: > - Replace AVG on MAP phase with SUM and COUNT. > - Change rowType produced by MAP phase/accepted by REDUCE phase. > - Add a projection after a reduce aggregate that includes division and > produces final result (it obviously should include other results of other > aggregates in the correct order). Divisor should be checked for 0 in order > not to trigger division by zero error. > Calcite has `AggregateReduceFunctionsRule` that rewrites AVG aggregate (see > reduceAvg method) that code can be used a reference. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-21087) VacuumTask removal
[ https://issues.apache.org/jira/browse/IGNITE-21087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827025#comment-17827025 ] Julia Bakulina commented on IGNITE-21087: - TCbot2 green visa: https://tcbot2.sbt-ignite-dev.ru/pr.html?serverId=apache&suiteId=IgniteTests24Java8_RunAll&branchForTc=pull/11273/head&action=Latest > VacuumTask removal > -- > > Key: IGNITE-21087 > URL: https://issues.apache.org/jira/browse/IGNITE-21087 > Project: Ignite > Issue Type: Sub-task >Reporter: Julia Bakulina >Assignee: Julia Bakulina >Priority: Minor > Labels: ise > Time Spent: 50m > Remaining Estimate: 0h > > Remove VacuumTask and MvccTxEntry. > Deleted classes: > * VacuumTask > * VacuumMetricsReducer > * VacuumMetrics > * PreviousQueries > * MvccTxEntry -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-21087) VacuumTask removal
[
https://issues.apache.org/jira/browse/IGNITE-21087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827026#comment-17827026
]
Ignite TC Bot commented on IGNITE-21087:
{panel:title=Branch: [pull/11273/head] Base: [master] : No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
{panel:title=Branch: [pull/11273/head] Base: [master] : No new tests
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}{panel}
[TeamCity *--> Run :: All*
Results|https://ci2.ignite.apache.org/viewLog.html?buildId=7783204&buildTypeId=IgniteTests24Java8_RunAll]
> VacuumTask removal
> --
>
> Key: IGNITE-21087
> URL: https://issues.apache.org/jira/browse/IGNITE-21087
> Project: Ignite
> Issue Type: Sub-task
>Reporter: Julia Bakulina
>Assignee: Julia Bakulina
>Priority: Minor
> Labels: ise
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Remove VacuumTask and MvccTxEntry.
> Deleted classes:
> * VacuumTask
> * VacuumMetricsReducer
> * VacuumMetrics
> * PreviousQueries
> * MvccTxEntry
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] (IGNITE-21087) VacuumTask removal
[ https://issues.apache.org/jira/browse/IGNITE-21087 ] Julia Bakulina deleted comment on IGNITE-21087: - was (Author: JIRAUSER294860): TCbot2 green visa: https://tcbot2.sbt-ignite-dev.ru/pr.html?serverId=apache&suiteId=IgniteTests24Java8_RunAll&branchForTc=pull/11273/head&action=Latest > VacuumTask removal > -- > > Key: IGNITE-21087 > URL: https://issues.apache.org/jira/browse/IGNITE-21087 > Project: Ignite > Issue Type: Sub-task >Reporter: Julia Bakulina >Assignee: Julia Bakulina >Priority: Minor > Labels: ise > Time Spent: 50m > Remaining Estimate: 0h > > Remove VacuumTask and MvccTxEntry. > Deleted classes: > * VacuumTask > * VacuumMetricsReducer > * VacuumMetrics > * PreviousQueries > * MvccTxEntry -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21087) VacuumTask removal
[ https://issues.apache.org/jira/browse/IGNITE-21087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julia Bakulina updated IGNITE-21087: Description: Deleted classes: * VacuumTask * VacuumMetricsReducer * VacuumMetrics * PreviousQueries * MvccEmptyLongList * MvccLongList * MvccCoordinatorChangeAware * MvccQueryTracker * MvccQueryTrackerImpl * StaticMvccQueryTracker was: Remove VacuumTask and MvccTxEntry. Deleted classes: * VacuumTask * VacuumMetricsReducer * VacuumMetrics * PreviousQueries * MvccTxEntry > VacuumTask removal > -- > > Key: IGNITE-21087 > URL: https://issues.apache.org/jira/browse/IGNITE-21087 > Project: Ignite > Issue Type: Sub-task >Reporter: Julia Bakulina >Assignee: Julia Bakulina >Priority: Minor > Labels: ise > Time Spent: 50m > Remaining Estimate: 0h > > Deleted classes: > * VacuumTask > * VacuumMetricsReducer > * VacuumMetrics > * PreviousQueries > * MvccEmptyLongList > * MvccLongList > * MvccCoordinatorChangeAware > * MvccQueryTracker > * MvccQueryTrackerImpl > * StaticMvccQueryTracker -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21087) Remove VacuumTask, MvccQueryTracker and MvccLongList
[ https://issues.apache.org/jira/browse/IGNITE-21087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julia Bakulina updated IGNITE-21087: Summary: Remove VacuumTask, MvccQueryTracker and MvccLongList (was: VacuumTask removal) > Remove VacuumTask, MvccQueryTracker and MvccLongList > > > Key: IGNITE-21087 > URL: https://issues.apache.org/jira/browse/IGNITE-21087 > Project: Ignite > Issue Type: Sub-task >Reporter: Julia Bakulina >Assignee: Julia Bakulina >Priority: Minor > Labels: ise > Time Spent: 50m > Remaining Estimate: 0h > > Deleted classes: > * VacuumTask > * VacuumMetricsReducer > * VacuumMetrics > * PreviousQueries > * MvccEmptyLongList > * MvccLongList > * MvccCoordinatorChangeAware > * MvccQueryTracker > * MvccQueryTrackerImpl > * StaticMvccQueryTracker -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-21755) Index destruction optimization
Andrey Mashenkov created IGNITE-21755: - Summary: Index destruction optimization Key: IGNITE-21755 URL: https://issues.apache.org/jira/browse/IGNITE-21755 Project: Ignite Issue Type: Improvement Reporter: Andrey Mashenkov LWM based on local clock and not synced with cluster time and metastorage events. So, it is possible LWM will be updated concurrently with index drop event. This may lead to a situation, when we enqueued index destruction event behind the LWM, but could destroy index instantly instead of waiting for the next onLwmUpdate triggering. E.g. this can be addressed via adding a continuation to the current Catalog ready future, where do LWM re-check and call onLwmChange if needed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-21671) Remove destroyed PageMemory indexes on recovery
[
https://issues.apache.org/jira/browse/IGNITE-21671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827032#comment-17827032
]
Kirill Tkalenko commented on IGNITE-21671:
--
Looks pretty good.
> Remove destroyed PageMemory indexes on recovery
> ---
>
> Key: IGNITE-21671
> URL: https://issues.apache.org/jira/browse/IGNITE-21671
> Project: Ignite
> Issue Type: Improvement
>Reporter: Aleksandr Polovtcev
>Assignee: Aleksandr Polovtcev
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> When {{AbstractPageMemoryMvPartitionStorage}} is started, we need to perform
> the following recovery actions:
> # Scan the index meta tree;
> # Check the observable Catalog history (i.e. up to the LWM value);
> # If an index is present in the meta tree, but not in the Catalog - destroy
> it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-21750) Throw a special exception when trying to read from/write to a destroyed IndexStorage
[ https://issues.apache.org/jira/browse/IGNITE-21750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Puchkovskiy updated IGNITE-21750: --- Description: Also: # DESTROYING state is to be renamed to DESTROYED # DESTROYED also becomes a terminal state (like CLOSED), so after a destruction completes the state does not move to CLOSED # RocksDB-based index storages need to be switched to the DESTROYED state when destroyIndex(int) is called on MvTableStorage > Throw a special exception when trying to read from/write to a destroyed > IndexStorage > > > Key: IGNITE-21750 > URL: https://issues.apache.org/jira/browse/IGNITE-21750 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Assignee: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > Also: > # DESTROYING state is to be renamed to DESTROYED > # DESTROYED also becomes a terminal state (like CLOSED), so after a > destruction completes the state does not move to CLOSED > # RocksDB-based index storages need to be switched to the DESTROYED state > when destroyIndex(int) is called on MvTableStorage -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21744) Use common thread pool for LWM updates.
[ https://issues.apache.org/jira/browse/IGNITE-21744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Mashenkov updated IGNITE-21744: -- Ignite Flags: (was: Docs Required,Release Notes Required) > Use common thread pool for LWM updates. > --- > > Key: IGNITE-21744 > URL: https://issues.apache.org/jira/browse/IGNITE-21744 > Project: Ignite > Issue Type: Improvement >Reporter: Andrey Mashenkov >Priority: Minor > Labels: ignite-3 > > LowWatermarkImpl creates a thread pool for scheduled task execution. > Let's use a common pool for that. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21711) ClockWaiter refactoring
[ https://issues.apache.org/jira/browse/IGNITE-21711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Mashenkov updated IGNITE-21711: -- Summary: ClockWaiter refactoring (was: Allow get current time from ClockWaiter.) > ClockWaiter refactoring > --- > > Key: IGNITE-21711 > URL: https://issues.apache.org/jira/browse/IGNITE-21711 > Project: Ignite > Issue Type: Improvement >Reporter: Andrey Mashenkov >Priority: Minor > Labels: ignite-3 > > ClockWaiter allows to get notification at certain time, but there is no way > to know the current time. > Because of this fact, we have to pass both object (clockWaiter and clock) to > dependant objects. > Let's > 1. Add a shortcut for `clock.now()` in ClockWater. > 2. Rename ClockWaiter -> ClockService > 3. Use ClockService instead of HybridClock in components, which don't use the > clock directly via `clock.update` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-21756) Use common thread pool in ClockWaiter
Andrey Mashenkov created IGNITE-21756: - Summary: Use common thread pool in ClockWaiter Key: IGNITE-21756 URL: https://issues.apache.org/jira/browse/IGNITE-21756 Project: Ignite Issue Type: Improvement Reporter: Andrey Mashenkov ClockWaiter creates a thread pool for tasks execution. Let's node pass a common thread pool via constructor. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21711) ClockWaiter refactoring
[ https://issues.apache.org/jira/browse/IGNITE-21711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Mashenkov updated IGNITE-21711: -- Description: ClockWaiter allows to get notification at certain time, but there is no way to know the current time. Because of this fact, we have to pass both object (clockWaiter and clock) to dependant objects. Let's 1. Add a shortcut for `clock.now()` in ClockWater. 2. Rename ClockWaiter -> ClockService 3. Use ClockService instead of HybridClock in components, which now required both Clock and ClockWaiter. Or maybe all components, which are not intended to update clock via `clock.update(timestamp)` was: ClockWaiter allows to get notification at certain time, but there is no way to know the current time. Because of this fact, we have to pass both object (clockWaiter and clock) to dependant objects. Let's 1. Add a shortcut for `clock.now()` in ClockWater. 2. Rename ClockWaiter -> ClockService 3. Use ClockService instead of HybridClock in components, which don't use the clock directly via `clock.update` > ClockWaiter refactoring > --- > > Key: IGNITE-21711 > URL: https://issues.apache.org/jira/browse/IGNITE-21711 > Project: Ignite > Issue Type: Improvement >Reporter: Andrey Mashenkov >Priority: Minor > Labels: ignite-3 > > ClockWaiter allows to get notification at certain time, but there is no way > to know the current time. > Because of this fact, we have to pass both object (clockWaiter and clock) to > dependant objects. > Let's > 1. Add a shortcut for `clock.now()` in ClockWater. > 2. Rename ClockWaiter -> ClockService > 3. Use ClockService instead of HybridClock in components, which now required > both Clock and ClockWaiter. Or maybe all components, which are not intended > to update clock via `clock.update(timestamp)` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21744) Use common thread pool for LWM updates.
[ https://issues.apache.org/jira/browse/IGNITE-21744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Mashenkov updated IGNITE-21744: -- Labels: ignite-3 tech-debt (was: ignite-3) > Use common thread pool for LWM updates. > --- > > Key: IGNITE-21744 > URL: https://issues.apache.org/jira/browse/IGNITE-21744 > Project: Ignite > Issue Type: Improvement >Reporter: Andrey Mashenkov >Priority: Minor > Labels: ignite-3, tech-debt > > LowWatermarkImpl creates a thread pool for scheduled task execution. > Let's use a common pool for that. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-21757) Flaky ItComputeTestStandalone
Vadim Pakhnushev created IGNITE-21757:
-
Summary: Flaky ItComputeTestStandalone
Key: IGNITE-21757
URL: https://issues.apache.org/jira/browse/IGNITE-21757
Project: Ignite
Issue Type: Bug
Components: compute
Reporter: Vadim Pakhnushev
Assignee: Vadim Pakhnushev
org.awaitility.core.ConditionTimeoutException: Lambda expression in
org.apache.ignite.internal.compute.ItComputeTestStandalone that uses
org.apache.ignite.internal.app.IgniteImpl,
org.apache.ignite.internal.app.IgniteImpljava.lang.String,
java.lang.Stringorg.apache.ignite.compute.version.Version: expected
but was null within 10 seconds.
10:40:04 org.awaitility.core.ConditionTimeoutException: Lambda expression
in org.apache.ignite.internal.compute.ItComputeTestStandalone that uses
org.apache.ignite.internal.app.IgniteImpl,
org.apache.ignite.internal.app.IgniteImpljava.lang.String,
java.lang.Stringorg.apache.ignite.compute.version.Version: expected
but was null within 10 seconds.
at
app//org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
at
app//org.awaitility.core.AbstractHamcrestCondition.await(AbstractHamcrestCondition.java:86)
at
app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:985)
at
app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:691)
at
app//org.apache.ignite.internal.compute.ItComputeTestStandalone.deployJar(ItComputeTestStandalone.java:202)
at
app//org.apache.ignite.internal.compute.ItComputeTestStandalone.setUp(ItComputeTestStandalone.java:67)
at java.base@11.0.17/java.lang.reflect.Method.invoke(Method.java:566)
at java.base@11.0.17/java.util.ArrayList.forEach(ArrayList.java:1541)
at java.base@11.0.17/java.util.ArrayList.forEach(ArrayList.java:1541)
10:40:04
org.apache.ignite.internal.compute.ItComputeTestStandalone.executesJobLocallyAsync()
10:40:04 [2024-03-14T07:40:04,126][INFO ][Test
worker][ItComputeTestStandalone] >>> Starting test:
ItComputeTestStandalone#executesJobLocallyAsync, displayName:
executesJobLocallyAsync()
[2024-03-14T07:40:04,127][INFO ][Test worker][DeploymentManagerImpl]
Undeploying jobs:1.0.0
[2024-03-14T07:40:04,127][INFO ][Test worker][DeployMessagingService]
Stop in progress deploy for jobs:1.0.0
[2024-03-14T07:40:04,240][INFO ][Test worker][DeploymentManagerImpl]
Deploying jobs:1.0.0 on NodesToDeploy\{nodesList=null, deployMode=MAJORITY}
[2024-03-14T07:40:04,244][INFO ][Test worker][ItComputeTestStandalone]
>>> Stopping test: ItComputeTestStandalone#executesJobLocallyAsync,
displayName: executesJobLocallyAsync(), cost: 117ms.
10:40:04 java.lang.AssertionError:
Expected: is
but: was
10:40:04 java.lang.AssertionError:
Expected: is
but: was
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
at
org.apache.ignite.internal.compute.ItComputeTestStandalone.deployJar(ItComputeTestStandalone.java:201)
at
org.apache.ignite.internal.compute.ItComputeTestStandalone.setUp(ItComputeTestStandalone.java:67)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-21757) Flaky ItComputeTestStandalone
[
https://issues.apache.org/jira/browse/IGNITE-21757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vadim Pakhnushev updated IGNITE-21757:
--
Description:
The test fails sometimes with this error:
{code:java}
[2024-03-14T07:39:53,879][INFO ][Test worker][ItComputeTestStandalone] >>>
Starting test: ItComputeTestStandalone#executesColocatedWithTupleKeyAsync,
displayName: executesColocatedWithTupleKeyAsync()
[2024-03-14T07:39:53,880][INFO ][Test worker][DeploymentManagerImpl]
Undeploying jobs:1.0.0
[2024-03-14T07:39:53,880][INFO ][Test worker][DeployMessagingService] Stop in
progress deploy for jobs:1.0.0
[2024-03-14T07:39:54,022][INFO ][Test worker][DeploymentManagerImpl] Deploying
jobs:1.0.0 on NodesToDeploy{nodesList=null, deployMode=MAJORITY}
[2024-03-14T07:39:56,782][INFO ][%icts_n_0%lease-updater][LeaseUpdater] Leases
updated (printed once per 10 iteration(s)): [inCurrentIteration=LeaseStats
[leasesCreated=0, leasesPublished=0, leasesProlonged=0,
leasesWithoutCandidates=0], active=25, currentAssignmentsSize=25].
[2024-03-14T07:40:02,289][INFO ][%icts_n_0%lease-updater][LeaseUpdater] Leases
updated (printed once per 10 iteration(s)): [inCurrentIteration=LeaseStats
[leasesCreated=0, leasesPublished=0, leasesProlonged=0,
leasesWithoutCandidates=0], active=25, currentAssignmentsSize=25].
[2024-03-14T07:40:04,121][INFO ][Test worker][ItComputeTestStandalone] >>>
Stopping test: ItComputeTestStandalone#executesColocatedWithTupleKeyAsync,
displayName: executesColocatedWithTupleKeyAsync(), cost: 10241ms.
[10:40:04]F:
[org.apache.ignite.internal.compute.ItComputeTestStandalone.executesColocatedWithTupleKeyAsync()]
org.awaitility.core.ConditionTimeoutException: Lambda expression in
org.apache.ignite.internal.compute.ItComputeTestStandalone that uses
org.apache.ignite.internal.app.IgniteImpl,
org.apache.ignite.internal.app.IgniteImpljava.lang.String,
java.lang.Stringorg.apache.ignite.compute.version.Version: expected
but was null within 10 seconds. at
app//org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
at
app//org.awaitility.core.AbstractHamcrestCondition.await(AbstractHamcrestCondition.java:86)
at
app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:985)
at
app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:691)
at
app//org.apache.ignite.internal.compute.ItComputeTestStandalone.deployJar(ItComputeTestStandalone.java:202)
at
app//org.apache.ignite.internal.compute.ItComputeTestStandalone.setUp(ItComputeTestStandalone.java:67){code}
was:
org.awaitility.core.ConditionTimeoutException: Lambda expression in
org.apache.ignite.internal.compute.ItComputeTestStandalone that uses
org.apache.ignite.internal.app.IgniteImpl,
org.apache.ignite.internal.app.IgniteImpljava.lang.String,
java.lang.Stringorg.apache.ignite.compute.version.Version: expected
but was null within 10 seconds.
10:40:04 org.awaitility.core.ConditionTimeoutException: Lambda expression
in org.apache.ignite.internal.compute.ItComputeTestStandalone that uses
org.apache.ignite.internal.app.IgniteImpl,
org.apache.ignite.internal.app.IgniteImpljava.lang.String,
java.lang.Stringorg.apache.ignite.compute.version.Version: expected
but was null within 10 seconds.
at
app//org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
at
app//org.awaitility.core.AbstractHamcrestCondition.await(AbstractHamcrestCondition.java:86)
at
app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:985)
at
app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:691)
at
app//org.apache.ignite.internal.compute.ItComputeTestStandalone.deployJar(ItComputeTestStandalone.java:202)
at
app//org.apache.ignite.internal.compute.ItComputeTestStandalone.setUp(ItComputeTestStandalone.java:67)
at java.base@11.0.17/java.lang.reflect.Method.invoke(Method.java:566)
at java.base@11.0.17/java.util.ArrayList.forEach(ArrayList.java:1541)
at java.base@11.0.17/java.util.ArrayList.forEach(ArrayList.java:1541)
10:40:04
org.apache.ignite.internal.compute.ItComputeTestStandalone.executesJobLocallyAsync()
10:40:04 [2024-03-14T07:40:04,126][INFO ][Test
worker][ItComputeTestStandalone] >>> Starting test:
ItComputeTestStandalone#executesJobLocallyAsync, displayName:
executesJobLocallyAsync()
[2024-03-14T07:40:04,127][INFO ][Test worker][DeploymentManagerImpl]
Undeploying jobs:1.0.0
[2024-03-14T07:40:04,127][INFO ][Test worker][DeployMessagingService]
Stop in progress deploy for jobs:1.0.0
[2024-03-14T07:40:04,240][INFO ][Test worker][DeploymentManagerImpl]
Deploying jobs:1.0.0 on NodesToDeploy\{nodesList=null, deployMode=MAJORITY}
[2024-03-14T07:40:04,244][INFO ][Test worker][ItComputeTestStandalone]
>>> Stopping test: ItComputeTestStandalone#exe
[jira] [Updated] (IGNITE-21757) Flaky ItComputeTestStandalone
[
https://issues.apache.org/jira/browse/IGNITE-21757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vadim Pakhnushev updated IGNITE-21757:
--
Labels: ignite-3 (was: )
> Flaky ItComputeTestStandalone
> -
>
> Key: IGNITE-21757
> URL: https://issues.apache.org/jira/browse/IGNITE-21757
> Project: Ignite
> Issue Type: Bug
> Components: compute
>Reporter: Vadim Pakhnushev
>Assignee: Vadim Pakhnushev
>Priority: Major
> Labels: ignite-3
>
> The test fails sometimes with this error:
>
> {code:java}
> [2024-03-14T07:39:53,879][INFO ][Test worker][ItComputeTestStandalone] >>>
> Starting test: ItComputeTestStandalone#executesColocatedWithTupleKeyAsync,
> displayName: executesColocatedWithTupleKeyAsync()
> [2024-03-14T07:39:53,880][INFO ][Test worker][DeploymentManagerImpl]
> Undeploying jobs:1.0.0
> [2024-03-14T07:39:53,880][INFO ][Test worker][DeployMessagingService] Stop in
> progress deploy for jobs:1.0.0
> [2024-03-14T07:39:54,022][INFO ][Test worker][DeploymentManagerImpl]
> Deploying jobs:1.0.0 on NodesToDeploy{nodesList=null, deployMode=MAJORITY}
> [2024-03-14T07:39:56,782][INFO ][%icts_n_0%lease-updater][LeaseUpdater]
> Leases updated (printed once per 10 iteration(s)):
> [inCurrentIteration=LeaseStats [leasesCreated=0, leasesPublished=0,
> leasesProlonged=0, leasesWithoutCandidates=0], active=25,
> currentAssignmentsSize=25].
> [2024-03-14T07:40:02,289][INFO ][%icts_n_0%lease-updater][LeaseUpdater]
> Leases updated (printed once per 10 iteration(s)):
> [inCurrentIteration=LeaseStats [leasesCreated=0, leasesPublished=0,
> leasesProlonged=0, leasesWithoutCandidates=0], active=25,
> currentAssignmentsSize=25].
> [2024-03-14T07:40:04,121][INFO ][Test worker][ItComputeTestStandalone] >>>
> Stopping test: ItComputeTestStandalone#executesColocatedWithTupleKeyAsync,
> displayName: executesColocatedWithTupleKeyAsync(), cost: 10241ms.
> [10:40:04]F:
> [org.apache.ignite.internal.compute.ItComputeTestStandalone.executesColocatedWithTupleKeyAsync()]
> org.awaitility.core.ConditionTimeoutException: Lambda expression in
> org.apache.ignite.internal.compute.ItComputeTestStandalone that uses
> org.apache.ignite.internal.app.IgniteImpl,
> org.apache.ignite.internal.app.IgniteImpljava.lang.String,
> java.lang.Stringorg.apache.ignite.compute.version.Version: expected
> but was null within 10 seconds. at
> app//org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
> at
> app//org.awaitility.core.AbstractHamcrestCondition.await(AbstractHamcrestCondition.java:86)
> at
> app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:985)
> at
> app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:691)
> at
> app//org.apache.ignite.internal.compute.ItComputeTestStandalone.deployJar(ItComputeTestStandalone.java:202)
> at
> app//org.apache.ignite.internal.compute.ItComputeTestStandalone.setUp(ItComputeTestStandalone.java:67){code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-21578) ItDurableFinishTest#testWaitForCleanup failed with NPE
[
https://issues.apache.org/jira/browse/IGNITE-21578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Lapin updated IGNITE-21578:
-
Reviewer: Alexander Lapin (was: Vladislav Pyatkov)
> ItDurableFinishTest#testWaitForCleanup failed with NPE
> ---
>
> Key: IGNITE-21578
> URL: https://issues.apache.org/jira/browse/IGNITE-21578
> Project: Ignite
> Issue Type: Bug
>Reporter: Alexander Lapin
>Assignee: Kirill Sizov
>Priority: Major
> Labels: ignite-3
> Time Spent: 20m
> Remaining Estimate: 0h
>
> [https://ci.ignite.apache.org/buildConfiguration/ApacheIgnite3xGradle_Test_RunAllTests/7870395?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandBuildProblemsSection=true&expandCode+Inspection=true&expandBuildChangesSection=true]
> {code:java}
> Caused by: java.lang.NullPointerException
> at
> org.apache.ignite.internal.tx.impl.TxManagerImpl.lambda$finishFull$3(TxManagerImpl.java:472)
> ~[ignite-transactions-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.tx.impl.VolatileTxStateMetaStorage.lambda$updateMeta$0(VolatileTxStateMetaStorage.java:73)
> ~[ignite-transactions-3.0.0-SNAPSHOT.jar:?]
> at
> java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1908)
> ~[?:?]
> at
> org.apache.ignite.internal.tx.impl.VolatileTxStateMetaStorage.updateMeta(VolatileTxStateMetaStorage.java:72)
> ~[ignite-transactions-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.tx.impl.TxManagerImpl.updateTxMeta(TxManagerImpl.java:455)
> ~[ignite-transactions-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.tx.impl.TxManagerImpl.finishFull(TxManagerImpl.java:472)
> ~[ignite-transactions-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.table.distributed.storage.InternalTableImpl.lambda$postEnlist$13(InternalTableImpl.java:593)
> ~[ignite-table-3.0.0-SNAPSHOT.jar:?] {code}
> Seems that the reason is that old meta may be null in case of exception
> {code:java}
> public void finishFull(HybridTimestampTracker timestampTracker, UUID
> txId, boolean commit) {
> ...
> updateTxMeta(txId, old -> new TxStateMeta(finalState,
> old.txCoordinatorId(), old.commitPartitionId(), old.commitTimestamp()));
> ...
> }
> {code}
> {code:java}
> return fut.handle((BiFunction>)
> (r, e) -> {
> if (full) { // Full txn is already finished remotely. Just update
> local state.
> txManager.finishFull(observableTimestampTracker, tx0.id(), e
> == null);{code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-21578) ItDurableFinishTest#testWaitForCleanup failed with NPE
[
https://issues.apache.org/jira/browse/IGNITE-21578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827052#comment-17827052
]
Alexander Lapin commented on IGNITE-21578:
--
[~ksizov] LGTM.
> ItDurableFinishTest#testWaitForCleanup failed with NPE
> ---
>
> Key: IGNITE-21578
> URL: https://issues.apache.org/jira/browse/IGNITE-21578
> Project: Ignite
> Issue Type: Bug
>Reporter: Alexander Lapin
>Assignee: Kirill Sizov
>Priority: Major
> Labels: ignite-3
> Time Spent: 20m
> Remaining Estimate: 0h
>
> [https://ci.ignite.apache.org/buildConfiguration/ApacheIgnite3xGradle_Test_RunAllTests/7870395?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandBuildProblemsSection=true&expandCode+Inspection=true&expandBuildChangesSection=true]
> {code:java}
> Caused by: java.lang.NullPointerException
> at
> org.apache.ignite.internal.tx.impl.TxManagerImpl.lambda$finishFull$3(TxManagerImpl.java:472)
> ~[ignite-transactions-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.tx.impl.VolatileTxStateMetaStorage.lambda$updateMeta$0(VolatileTxStateMetaStorage.java:73)
> ~[ignite-transactions-3.0.0-SNAPSHOT.jar:?]
> at
> java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1908)
> ~[?:?]
> at
> org.apache.ignite.internal.tx.impl.VolatileTxStateMetaStorage.updateMeta(VolatileTxStateMetaStorage.java:72)
> ~[ignite-transactions-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.tx.impl.TxManagerImpl.updateTxMeta(TxManagerImpl.java:455)
> ~[ignite-transactions-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.tx.impl.TxManagerImpl.finishFull(TxManagerImpl.java:472)
> ~[ignite-transactions-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.table.distributed.storage.InternalTableImpl.lambda$postEnlist$13(InternalTableImpl.java:593)
> ~[ignite-table-3.0.0-SNAPSHOT.jar:?] {code}
> Seems that the reason is that old meta may be null in case of exception
> {code:java}
> public void finishFull(HybridTimestampTracker timestampTracker, UUID
> txId, boolean commit) {
> ...
> updateTxMeta(txId, old -> new TxStateMeta(finalState,
> old.txCoordinatorId(), old.commitPartitionId(), old.commitTimestamp()));
> ...
> }
> {code}
> {code:java}
> return fut.handle((BiFunction>)
> (r, e) -> {
> if (full) { // Full txn is already finished remotely. Just update
> local state.
> txManager.finishFull(observableTimestampTracker, tx0.id(), e
> == null);{code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-17666) Scan subscription cancel does not close a server side cursor
[ https://issues.apache.org/jira/browse/IGNITE-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-17666: - Fix Version/s: 3.0.0-beta2 > Scan subscription cancel does not close a server side cursor > > > Key: IGNITE-17666 > URL: https://issues.apache.org/jira/browse/IGNITE-17666 > Project: Ignite > Issue Type: Improvement >Reporter: Vladislav Pyatkov >Assignee: Vladislav Pyatkov >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 10m > Remaining Estimate: 0h > > h3. Motivation > Read-only transactions start a scan operation and save transaction cursors on > the replica side. The cursors stay on the server until they are closed. > Read-only transactions finish locally and do not notify all replicas of the > transaction. > h3. Definition of done > # Enlist node to RO transaction. > # Prohibit adding operations to the read-only transaction after the > transaction is finished. > # Send the finish transaction request to all enlisted nodes to close cursors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21221) Txn resource cleanup
[ https://issues.apache.org/jira/browse/IGNITE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-21221: - Epic Name: Txn cursors cleanup (was: Txn resource cleanup) > Txn resource cleanup > > > Key: IGNITE-21221 > URL: https://issues.apache.org/jira/browse/IGNITE-21221 > Project: Ignite > Issue Type: Epic >Reporter: Alexander Lapin >Priority: Major > Labels: ignite-3 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (IGNITE-17666) Scan subscription cancel does not close a server side cursor
[ https://issues.apache.org/jira/browse/IGNITE-17666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin resolved IGNITE-17666. -- Resolution: Fixed > Scan subscription cancel does not close a server side cursor > > > Key: IGNITE-17666 > URL: https://issues.apache.org/jira/browse/IGNITE-17666 > Project: Ignite > Issue Type: Improvement >Reporter: Vladislav Pyatkov >Assignee: Vladislav Pyatkov >Priority: Major > Labels: ignite-3 > Time Spent: 10m > Remaining Estimate: 0h > > h3. Motivation > Read-only transactions start a scan operation and save transaction cursors on > the replica side. The cursors stay on the server until they are closed. > Read-only transactions finish locally and do not notify all replicas of the > transaction. > h3. Definition of done > # Enlist node to RO transaction. > # Prohibit adding operations to the read-only transaction after the > transaction is finished. > # Send the finish transaction request to all enlisted nodes to close cursors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21758) Txn resource cleanup
[ https://issues.apache.org/jira/browse/IGNITE-21758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-21758: - Ignite Flags: (was: Docs Required,Release Notes Required) > Txn resource cleanup > > > Key: IGNITE-21758 > URL: https://issues.apache.org/jira/browse/IGNITE-21758 > Project: Ignite > Issue Type: Epic >Reporter: Alexander Lapin >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-21758) Txn resource cleanup
Alexander Lapin created IGNITE-21758: Summary: Txn resource cleanup Key: IGNITE-21758 URL: https://issues.apache.org/jira/browse/IGNITE-21758 Project: Ignite Issue Type: Epic Reporter: Alexander Lapin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18617) Clear the map of tx cleanup ready futures on tx state vacuum
[ https://issues.apache.org/jira/browse/IGNITE-18617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-18617: - Epic Link: IGNITE-21758 (was: IGNITE-21221) > Clear the map of tx cleanup ready futures on tx state vacuum > > > Key: IGNITE-18617 > URL: https://issues.apache.org/jira/browse/IGNITE-18617 > Project: Ignite > Issue Type: Improvement >Reporter: Denis Chudov >Priority: Major > Labels: ignite-3 > > The map PartitionReplicaListener#txCleanupReadyFutures is not cleared > anywhere. It should be cleared in the same way as tx states storage (which is > not defined yet). > TBD. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21221) Txn cursors cleanup
[ https://issues.apache.org/jira/browse/IGNITE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-21221: - Summary: Txn cursors cleanup (was: Txn resource cleanup) > Txn cursors cleanup > --- > > Key: IGNITE-21221 > URL: https://issues.apache.org/jira/browse/IGNITE-21221 > Project: Ignite > Issue Type: Epic >Reporter: Alexander Lapin >Priority: Major > Labels: ignite-3 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21759) Prepare general txn cleanup logic: txnCleanupTask along with txnCleanupContext
[ https://issues.apache.org/jira/browse/IGNITE-21759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-21759: - Ignite Flags: (was: Docs Required,Release Notes Required) > Prepare general txn cleanup logic: txnCleanupTask along with txnCleanupContext > -- > > Key: IGNITE-21759 > URL: https://issues.apache.org/jira/browse/IGNITE-21759 > Project: Ignite > Issue Type: Improvement >Reporter: Alexander Lapin >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-21759) Prepare general txn cleanup logic: txnCleanupTask along with txnCleanupContext
Alexander Lapin created IGNITE-21759: Summary: Prepare general txn cleanup logic: txnCleanupTask along with txnCleanupContext Key: IGNITE-21759 URL: https://issues.apache.org/jira/browse/IGNITE-21759 Project: Ignite Issue Type: Improvement Reporter: Alexander Lapin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21754) Remove destroyed tables on recovery - Page Memory
[ https://issues.apache.org/jira/browse/IGNITE-21754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Pligin updated IGNITE-21754: - Summary: Remove destroyed tables on recovery - Page Memory (was: Remove destroyed tables on recovery) > Remove destroyed tables on recovery - Page Memory > - > > Key: IGNITE-21754 > URL: https://issues.apache.org/jira/browse/IGNITE-21754 > Project: Ignite > Issue Type: Improvement >Reporter: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > We need to destroy tables on node recovery that fall under the low watermark > for all storage engines. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-21760) Remove destroyed tables on recovery - RocksDB
Vladimir Pligin created IGNITE-21760: Summary: Remove destroyed tables on recovery - RocksDB Key: IGNITE-21760 URL: https://issues.apache.org/jira/browse/IGNITE-21760 Project: Ignite Issue Type: Improvement Reporter: Vladimir Pligin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21754) Remove destroyed tables on recovery - Page Memory
[ https://issues.apache.org/jira/browse/IGNITE-21754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Pligin updated IGNITE-21754: - Description: We need to destroy tables on node recovery that fall under the low watermark for the page memory storage engine. (was: We need to destroy tables on node recovery that fall under the low watermark for all storage engines.) > Remove destroyed tables on recovery - Page Memory > - > > Key: IGNITE-21754 > URL: https://issues.apache.org/jira/browse/IGNITE-21754 > Project: Ignite > Issue Type: Improvement >Reporter: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > We need to destroy tables on node recovery that fall under the low watermark > for the page memory storage engine. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21760) Remove destroyed tables on recovery - RocksDB
[ https://issues.apache.org/jira/browse/IGNITE-21760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Pligin updated IGNITE-21760: - Description: We need to destroy tables on node recovery that fall under the low watermark for the rocks db storage engine. > Remove destroyed tables on recovery - RocksDB > - > > Key: IGNITE-21760 > URL: https://issues.apache.org/jira/browse/IGNITE-21760 > Project: Ignite > Issue Type: Improvement >Reporter: Vladimir Pligin >Priority: Major > > We need to destroy tables on node recovery that fall under the low watermark > for the rocks db storage engine. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21760) Remove destroyed tables on recovery - RocksDB
[ https://issues.apache.org/jira/browse/IGNITE-21760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Pligin updated IGNITE-21760: - Labels: ignite-3 (was: ) > Remove destroyed tables on recovery - RocksDB > - > > Key: IGNITE-21760 > URL: https://issues.apache.org/jira/browse/IGNITE-21760 > Project: Ignite > Issue Type: Improvement >Reporter: Vladimir Pligin >Priority: Major > Labels: ignite-3 > > We need to destroy tables on node recovery that fall under the low watermark > for the rocks db storage engine. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21760) Remove destroyed tables on recovery - RocksDB
[ https://issues.apache.org/jira/browse/IGNITE-21760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Pligin updated IGNITE-21760: - Epic Link: IGNITE-20782 > Remove destroyed tables on recovery - RocksDB > - > > Key: IGNITE-21760 > URL: https://issues.apache.org/jira/browse/IGNITE-21760 > Project: Ignite > Issue Type: Improvement >Reporter: Vladimir Pligin >Priority: Major > > We need to destroy tables on node recovery that fall under the low watermark > for the rocks db storage engine. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21738) Resolve potential races when table is destroying
[ https://issues.apache.org/jira/browse/IGNITE-21738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Mashenkov updated IGNITE-21738: -- Priority: Major (was: Blocker) > Resolve potential races when table is destroying > > > Key: IGNITE-21738 > URL: https://issues.apache.org/jira/browse/IGNITE-21738 > Project: Ignite > Issue Type: Improvement >Reporter: Andrey Mashenkov >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > After IGNITE-21585 there are potential races: > 1. ClientPrimaryReplicaTracker may update `primaryReplicas` after table > destroyed concurrently. > 2. TableManager may register index (see `registerIndexesToTable`) in a table, > while LWM rising up concurrently. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21738) Proper fix to linearize LWM and Metastorage event.
[ https://issues.apache.org/jira/browse/IGNITE-21738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Mashenkov updated IGNITE-21738: -- Summary: Proper fix to linearize LWM and Metastorage event. (was: Resolve potential races when table is destroying) > Proper fix to linearize LWM and Metastorage event. > -- > > Key: IGNITE-21738 > URL: https://issues.apache.org/jira/browse/IGNITE-21738 > Project: Ignite > Issue Type: Improvement >Reporter: Andrey Mashenkov >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > After IGNITE-21585 there are potential races: > 1. ClientPrimaryReplicaTracker may update `primaryReplicas` after table > destroyed concurrently. > 2. TableManager may register index (see `registerIndexesToTable`) in a table, > while LWM rising up concurrently. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21738) Proper fix to linearize LWM and Metastorage event.
[ https://issues.apache.org/jira/browse/IGNITE-21738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Mashenkov updated IGNITE-21738: -- Description: In IGNITE-21585 there were fixed potential races: 1. ClientPrimaryReplicaTracker might update `primaryReplicas` after table destroyed concurrently. The fix idea was to forbid caching data for destroyed tables. However, cache contains placeholders for all tables/partitions even if ther never be requested. 2. TableManager might register index (see `registerIndexesToTable`) in a table, while LWM rising up concurrently. The `registerIndexesToTable` execution was delegated into LowWatermark, to make sure it runs under an lwm lock. was: After IGNITE-21585 there are potential races: 1. ClientPrimaryReplicaTracker may update `primaryReplicas` after table destroyed concurrently. 2. TableManager may register index (see `registerIndexesToTable`) in a table, while LWM rising up concurrently. > Proper fix to linearize LWM and Metastorage event. > -- > > Key: IGNITE-21738 > URL: https://issues.apache.org/jira/browse/IGNITE-21738 > Project: Ignite > Issue Type: Improvement >Reporter: Andrey Mashenkov >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > In IGNITE-21585 there were fixed potential races: > 1. ClientPrimaryReplicaTracker might update `primaryReplicas` after table > destroyed concurrently. > The fix idea was to forbid caching data for destroyed tables. However, cache > contains placeholders for all tables/partitions even if ther never be > requested. > 2. TableManager might register index (see `registerIndexesToTable`) in a > table, while LWM rising up concurrently. > The `registerIndexesToTable` execution was delegated into LowWatermark, to > make sure it runs under an lwm lock. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21759) Prepare general txn cleanup logic: txnCleanupTask along with txnCleanupContext
[ https://issues.apache.org/jira/browse/IGNITE-21759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-21759: - Description: h3. Motivation Within one of Txn cursros cleanup sub tickets cursor cleanup scheduler was introduced. Now it's turn to generalize it to handle txn resource cleanup task. Thus it's expected that two new entities will be introduced: TxnResourceCleanupTask and TxnResourceCleanupContext, the latter is expected to be node-local, volatile and be, probably, a part of a TxManager. Few words about the logic inside TxnResourceCleanupTask that is expected to be implemented within given ticket (it'll be adjusted later on in other jiras): # CleanupTask evaluates txnStateCleanupWatermark as System.currentTimeMillis. There's no sense in using common clock and thus we may reduce the contention. # CleanupTask iterates over txnResourceCleanupContext.txnStatesToCleanup and if txnState.markForRemovalTimestamp + txnStateTTL (more details a bit later) >= txnStateCleanupWatermark: ## Removes given txnId from txnStateVolatileMap. ## Removes same txnId from txnResourceCleanupContext.txnStatesToCleanup. # CleanupTask scans txnStateVolatileMap and filters all finished (COMMITED or ABORTED) transactions. # Depending on txnStateTTL cleanupTask either removes the finished entry from the txnStateVolatileMap (if txnStateTTL == 0) or populates the txnResourceCleanupContext.transactionsToCleanup map with txId -> txnStateCleanupWatermark semantically txnToCleanupDetectionTime. Such bypassing - immediate removal instead of common "mark -> nextTimerIteration -> remove" is useful for testing purposes. txnStateTTL is a new cluster configuration property that defines the *minimum* lifetime of a transaction state in milliseconds. Real txnState lifetime will also depend on cleanupTimer intervals which of course not suitable for test purposes thus we should introduce manual ability to run TxnResourceCleanupTask immediately. At least we will use it within tests. h3. Definition of Done * New configuration property txnStateTTL is introduced with default of 30_000 milliseconds. * At least two new classes are introduced: TxnResourceCleanupTask and TxnResourceCleanupContext. * Cleanup scheduler is generalized and adjusted to run TxnResourceCleanupTask on every iteration. * TxnResourceCleanupTask is thread safe, and implemented according to the description in motivation section. * Special trigger txnResourceCleanup is introduced within TxManager. > Prepare general txn cleanup logic: txnCleanupTask along with txnCleanupContext > -- > > Key: IGNITE-21759 > URL: https://issues.apache.org/jira/browse/IGNITE-21759 > Project: Ignite > Issue Type: Improvement >Reporter: Alexander Lapin >Priority: Major > > h3. Motivation > Within one of Txn cursros cleanup sub tickets cursor cleanup scheduler was > introduced. Now it's turn to generalize it to handle txn resource cleanup > task. Thus it's expected that two new entities will be introduced: > TxnResourceCleanupTask and TxnResourceCleanupContext, the latter is expected > to be node-local, volatile and be, probably, a part of a TxManager. > Few words about the logic inside TxnResourceCleanupTask that is expected to > be implemented within given ticket (it'll be adjusted later on in other > jiras): > # CleanupTask evaluates txnStateCleanupWatermark as > System.currentTimeMillis. There's no sense in using common clock and thus we > may reduce the contention. > # CleanupTask iterates over txnResourceCleanupContext.txnStatesToCleanup and > if txnState.markForRemovalTimestamp + txnStateTTL (more details a bit later) > >= txnStateCleanupWatermark: > ## Removes given txnId from txnStateVolatileMap. > ## Removes same txnId from txnResourceCleanupContext.txnStatesToCleanup. > # CleanupTask scans txnStateVolatileMap and filters all finished (COMMITED > or ABORTED) transactions. > # Depending on txnStateTTL cleanupTask either removes the finished entry > from the txnStateVolatileMap (if txnStateTTL == 0) or populates the > txnResourceCleanupContext.transactionsToCleanup map with txId -> > txnStateCleanupWatermark semantically txnToCleanupDetectionTime. Such > bypassing - immediate removal instead of common "mark -> nextTimerIteration > -> remove" is useful for testing purposes. > > txnStateTTL is a new cluster configuration property that defines the > *minimum* lifetime of a transaction state in milliseconds. Real txnState > lifetime will also depend on cleanupTimer intervals which of course not > suitable for test purposes thus we should introduce manual ability to run > TxnResourceCleanupTask immediately. At least we will use it within tests. > h3. Definition of Done > * New configurat
[jira] [Updated] (IGNITE-21759) Prepare general txn cleanup logic: txnCleanupTask along with txnCleanupContext
[ https://issues.apache.org/jira/browse/IGNITE-21759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-21759: - Epic Link: IGNITE-21758 > Prepare general txn cleanup logic: txnCleanupTask along with txnCleanupContext > -- > > Key: IGNITE-21759 > URL: https://issues.apache.org/jira/browse/IGNITE-21759 > Project: Ignite > Issue Type: Improvement >Reporter: Alexander Lapin >Priority: Major > Labels: ignite-3 > > h3. Motivation > Within one of Txn cursros cleanup sub tickets cursor cleanup scheduler was > introduced. Now it's turn to generalize it to handle txn resource cleanup > task. Thus it's expected that two new entities will be introduced: > TxnResourceCleanupTask and TxnResourceCleanupContext, the latter is expected > to be node-local, volatile and be, probably, a part of a TxManager. > Few words about the logic inside TxnResourceCleanupTask that is expected to > be implemented within given ticket (it'll be adjusted later on in other > jiras): > # CleanupTask evaluates txnStateCleanupWatermark as > System.currentTimeMillis. There's no sense in using common clock and thus we > may reduce the contention. > # CleanupTask iterates over txnResourceCleanupContext.txnStatesToCleanup and > if txnState.markForRemovalTimestamp + txnStateTTL (more details a bit later) > >= txnStateCleanupWatermark: > ## Removes given txnId from txnStateVolatileMap. > ## Removes same txnId from txnResourceCleanupContext.txnStatesToCleanup. > # CleanupTask scans txnStateVolatileMap and filters all finished (COMMITED > or ABORTED) transactions. > # Depending on txnStateTTL cleanupTask either removes the finished entry > from the txnStateVolatileMap (if txnStateTTL == 0) or populates the > txnResourceCleanupContext.transactionsToCleanup map with txId -> > txnStateCleanupWatermark semantically txnToCleanupDetectionTime. Such > bypassing - immediate removal instead of common "mark -> nextTimerIteration > -> remove" is useful for testing purposes. > > txnStateTTL is a new cluster configuration property that defines the > *minimum* lifetime of a transaction state in milliseconds. Real txnState > lifetime will also depend on cleanupTimer intervals which of course not > suitable for test purposes thus we should introduce manual ability to run > TxnResourceCleanupTask immediately. At least we will use it within tests. > h3. Definition of Done > * New configuration property txnStateTTL is introduced with default of > 30_000 milliseconds. > * At least two new classes are introduced: TxnResourceCleanupTask and > TxnResourceCleanupContext. > * Cleanup scheduler is generalized and adjusted to run > TxnResourceCleanupTask on every iteration. > * TxnResourceCleanupTask is thread safe, and implemented according to the > description in motivation section. > * Special trigger txnResourceCleanup is introduced within TxManager. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21759) Prepare general txn cleanup logic: txnCleanupTask along with txnCleanupContext
[ https://issues.apache.org/jira/browse/IGNITE-21759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-21759: - Labels: ignite-3 (was: ) > Prepare general txn cleanup logic: txnCleanupTask along with txnCleanupContext > -- > > Key: IGNITE-21759 > URL: https://issues.apache.org/jira/browse/IGNITE-21759 > Project: Ignite > Issue Type: Improvement >Reporter: Alexander Lapin >Priority: Major > Labels: ignite-3 > > h3. Motivation > Within one of Txn cursros cleanup sub tickets cursor cleanup scheduler was > introduced. Now it's turn to generalize it to handle txn resource cleanup > task. Thus it's expected that two new entities will be introduced: > TxnResourceCleanupTask and TxnResourceCleanupContext, the latter is expected > to be node-local, volatile and be, probably, a part of a TxManager. > Few words about the logic inside TxnResourceCleanupTask that is expected to > be implemented within given ticket (it'll be adjusted later on in other > jiras): > # CleanupTask evaluates txnStateCleanupWatermark as > System.currentTimeMillis. There's no sense in using common clock and thus we > may reduce the contention. > # CleanupTask iterates over txnResourceCleanupContext.txnStatesToCleanup and > if txnState.markForRemovalTimestamp + txnStateTTL (more details a bit later) > >= txnStateCleanupWatermark: > ## Removes given txnId from txnStateVolatileMap. > ## Removes same txnId from txnResourceCleanupContext.txnStatesToCleanup. > # CleanupTask scans txnStateVolatileMap and filters all finished (COMMITED > or ABORTED) transactions. > # Depending on txnStateTTL cleanupTask either removes the finished entry > from the txnStateVolatileMap (if txnStateTTL == 0) or populates the > txnResourceCleanupContext.transactionsToCleanup map with txId -> > txnStateCleanupWatermark semantically txnToCleanupDetectionTime. Such > bypassing - immediate removal instead of common "mark -> nextTimerIteration > -> remove" is useful for testing purposes. > > txnStateTTL is a new cluster configuration property that defines the > *minimum* lifetime of a transaction state in milliseconds. Real txnState > lifetime will also depend on cleanupTimer intervals which of course not > suitable for test purposes thus we should introduce manual ability to run > TxnResourceCleanupTask immediately. At least we will use it within tests. > h3. Definition of Done > * New configuration property txnStateTTL is introduced with default of > 30_000 milliseconds. > * At least two new classes are introduced: TxnResourceCleanupTask and > TxnResourceCleanupContext. > * Cleanup scheduler is generalized and adjusted to run > TxnResourceCleanupTask on every iteration. > * TxnResourceCleanupTask is thread safe, and implemented according to the > description in motivation section. > * Special trigger txnResourceCleanup is introduced within TxManager. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21750) Throw a special exception when trying to read from/write to a destroyed IndexStorage
[ https://issues.apache.org/jira/browse/IGNITE-21750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Puchkovskiy updated IGNITE-21750: --- Description: Also: # DESTROYING state is to be renamed to DESTROYED # DESTROYED also becomes a terminal state (like CLOSED), so after a destruction completes the state does not move to CLOSED # After a storage (partition or index) was switched to any of CLOSED/DESTROYED states, it should not allow normal operation # RocksDB-based index storages need to be switched to the DESTROYED state when destroyIndex(int) is called on MvTableStorage was: Also: # DESTROYING state is to be renamed to DESTROYED # DESTROYED also becomes a terminal state (like CLOSED), so after a destruction completes the state does not move to CLOSED # RocksDB-based index storages need to be switched to the DESTROYED state when destroyIndex(int) is called on MvTableStorage > Throw a special exception when trying to read from/write to a destroyed > IndexStorage > > > Key: IGNITE-21750 > URL: https://issues.apache.org/jira/browse/IGNITE-21750 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Assignee: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > Also: > # DESTROYING state is to be renamed to DESTROYED > # DESTROYED also becomes a terminal state (like CLOSED), so after a > destruction completes the state does not move to CLOSED > # After a storage (partition or index) was switched to any of > CLOSED/DESTROYED states, it should not allow normal operation > # RocksDB-based index storages need to be switched to the DESTROYED state > when destroyIndex(int) is called on MvTableStorage -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21750) Throw a special exception when trying to read from/write to a destroyed IndexStorage
[ https://issues.apache.org/jira/browse/IGNITE-21750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Puchkovskiy updated IGNITE-21750: --- Description: Also: # DESTROYING state is to be renamed to DESTROYED # DESTROYED also becomes a terminal state (like CLOSED), so after a destruction completes the state does not move to CLOSED # After a storage (partition or index) was switched to any of CLOSED/DESTROYED states, it should not allow normal operation (probably, its busy lock must be blocked forever) # RocksDB-based index storages need to be switched to the DESTROYED state when destroyIndex(int) is called on MvTableStorage was: Also: # DESTROYING state is to be renamed to DESTROYED # DESTROYED also becomes a terminal state (like CLOSED), so after a destruction completes the state does not move to CLOSED # After a storage (partition or index) was switched to any of CLOSED/DESTROYED states, it should not allow normal operation # RocksDB-based index storages need to be switched to the DESTROYED state when destroyIndex(int) is called on MvTableStorage > Throw a special exception when trying to read from/write to a destroyed > IndexStorage > > > Key: IGNITE-21750 > URL: https://issues.apache.org/jira/browse/IGNITE-21750 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Assignee: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > Also: > # DESTROYING state is to be renamed to DESTROYED > # DESTROYED also becomes a terminal state (like CLOSED), so after a > destruction completes the state does not move to CLOSED > # After a storage (partition or index) was switched to any of > CLOSED/DESTROYED states, it should not allow normal operation (probably, its > busy lock must be blocked forever) > # RocksDB-based index storages need to be switched to the DESTROYED state > when destroyIndex(int) is called on MvTableStorage -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-21761) Within commitPartition mark Mark txnState with a timestamp on cleanup replication finished
Alexander Lapin created IGNITE-21761: Summary: Within commitPartition mark Mark txnState with a timestamp on cleanup replication finished Key: IGNITE-21761 URL: https://issues.apache.org/jira/browse/IGNITE-21761 Project: Ignite Issue Type: Improvement Reporter: Alexander Lapin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21761) Within commitPartition mark Mark txnState with a timestamp on cleanup replication finished
[ https://issues.apache.org/jira/browse/IGNITE-21761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-21761: - Ignite Flags: (was: Docs Required,Release Notes Required) > Within commitPartition mark Mark txnState with a timestamp on cleanup > replication finished > -- > > Key: IGNITE-21761 > URL: https://issues.apache.org/jira/browse/IGNITE-21761 > Project: Ignite > Issue Type: Improvement >Reporter: Alexander Lapin >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21761) Within commitPartition mark Mark txnState with a timestamp on cleanup replication finished
[ https://issues.apache.org/jira/browse/IGNITE-21761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-21761: - Labels: ignite-3 (was: ) > Within commitPartition mark Mark txnState with a timestamp on cleanup > replication finished > -- > > Key: IGNITE-21761 > URL: https://issues.apache.org/jira/browse/IGNITE-21761 > Project: Ignite > Issue Type: Improvement >Reporter: Alexander Lapin >Priority: Major > Labels: ignite-3 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21761) Within commitPartition mark Mark txnState with a timestamp on cleanup replication finished
[ https://issues.apache.org/jira/browse/IGNITE-21761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-21761: - Epic Link: IGNITE-21758 > Within commitPartition mark Mark txnState with a timestamp on cleanup > replication finished > -- > > Key: IGNITE-21761 > URL: https://issues.apache.org/jira/browse/IGNITE-21761 > Project: Ignite > Issue Type: Improvement >Reporter: Alexander Lapin >Priority: Major > Labels: ignite-3 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21761) Within commitPartition mark Mark txnState with a timestamp on cleanup replication finished
[ https://issues.apache.org/jira/browse/IGNITE-21761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-21761: - Description: h3. Motivation Volatile txn state removal is implemented in IGNITE-21759 Worth mentioning that having volatile txnState is mainly an optimization just because it might be lost on node restart. This ticket is about marking txnState persistent state as ready for removal. Basically it is ready for removal when the state is either COMMITED or ABORTED and cleanup is fully replicated over majority of all enlisted partitions. There are some nuances related to txn recovery case that will be covered in another jiras. h3. Definition of Done * In a durable manner (meaning retry on failure) await cleanup replication over majority of all enlisted partitions. * On cleanup done, compute txnState in txnStateVolatileMap with System.currentTimeMillis(). Pay attention that depending on whether given txnState was already removed by TxnResourseCleanupTask or not, it's either an adjusting of an exiting entry in txnStateVolatileMap with timestamp or writing a brand new record. In latter case another iteration of TxnStateCleanupTask will remove newly created entry. Besides that if the volatile state was removed it won't be possible to restore all the meta, which is fine, because all we need is txnId, txnState and System.currentTimeMillies as cleanupReplicationFinished timestamp. * There will be another Jira that will consider such timestamps and remove corresponding states from txnStatePersistentStorage. For know it's out of scope. * There will be another task that will redo the cleanup on node startup in order to restore (or formally re-evaluate) the cleanup replication finished timestamp. For know it's out of scope. > Within commitPartition mark Mark txnState with a timestamp on cleanup > replication finished > -- > > Key: IGNITE-21761 > URL: https://issues.apache.org/jira/browse/IGNITE-21761 > Project: Ignite > Issue Type: Improvement >Reporter: Alexander Lapin >Priority: Major > Labels: ignite-3 > > h3. Motivation > Volatile txn state removal is implemented in IGNITE-21759 Worth mentioning > that having volatile txnState is mainly an optimization just because it might > be lost on node restart. > This ticket is about marking txnState persistent state as ready for removal. > Basically it is ready for removal when the state is either COMMITED or > ABORTED and cleanup is fully replicated over majority of all enlisted > partitions. There are some nuances related to txn recovery case that will be > covered in another jiras. > h3. Definition of Done > * In a durable manner (meaning retry on failure) await cleanup replication > over majority of all enlisted partitions. > * On cleanup done, compute txnState in txnStateVolatileMap with > System.currentTimeMillis(). Pay attention that depending on whether given > txnState was already removed by TxnResourseCleanupTask or not, it's either an > adjusting of an exiting entry in txnStateVolatileMap with timestamp or > writing a brand new record. In latter case another iteration of > TxnStateCleanupTask will remove newly created entry. Besides that if the > volatile state was removed it won't be possible to restore all the meta, > which is fine, because all we need is txnId, txnState and > System.currentTimeMillies as cleanupReplicationFinished timestamp. > * There will be another Jira that will consider such timestamps and remove > corresponding states from txnStatePersistentStorage. For know it's out of > scope. > * There will be another task that will redo the cleanup on node startup in > order to restore (or formally re-evaluate) the cleanup replication finished > timestamp. For know it's out of scope. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21762) Do the txn cleanup procedure for all local finished transactions on commitPartition primary replica election
[ https://issues.apache.org/jira/browse/IGNITE-21762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-21762: - Ignite Flags: (was: Docs Required,Release Notes Required) > Do the txn cleanup procedure for all local finished transactions on > commitPartition primary replica election > > > Key: IGNITE-21762 > URL: https://issues.apache.org/jira/browse/IGNITE-21762 > Project: Ignite > Issue Type: Improvement >Reporter: Alexander Lapin >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-21762) Do the txn cleanup procedure for all local finished transactions on commitPartition primary replica election
Alexander Lapin created IGNITE-21762: Summary: Do the txn cleanup procedure for all local finished transactions on commitPartition primary replica election Key: IGNITE-21762 URL: https://issues.apache.org/jira/browse/IGNITE-21762 Project: Ignite Issue Type: Improvement Reporter: Alexander Lapin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21762) Do the txn cleanup procedure for all local finished transactions on commitPartition primary replica election
[ https://issues.apache.org/jira/browse/IGNITE-21762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-21762: - Labels: ignite-3 (was: ) > Do the txn cleanup procedure for all local finished transactions on > commitPartition primary replica election > > > Key: IGNITE-21762 > URL: https://issues.apache.org/jira/browse/IGNITE-21762 > Project: Ignite > Issue Type: Improvement >Reporter: Alexander Lapin >Priority: Major > Labels: ignite-3 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21762) Do the txn cleanup procedure for all local finished transactions on commitPartition primary replica election
[ https://issues.apache.org/jira/browse/IGNITE-21762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-21762: - Epic Link: IGNITE-21758 > Do the txn cleanup procedure for all local finished transactions on > commitPartition primary replica election > > > Key: IGNITE-21762 > URL: https://issues.apache.org/jira/browse/IGNITE-21762 > Project: Ignite > Issue Type: Improvement >Reporter: Alexander Lapin >Priority: Major > Labels: ignite-3 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21762) Do the txn cleanup procedure for all local finished transactions on commitPartition primary replica election
[ https://issues.apache.org/jira/browse/IGNITE-21762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-21762: - Description: h3. Motivation As mentioned in IGNITE-21761 it's valid to remove finished transaction when cleanup is fully replicated over all enlisted partitions. However it's possible that commitPartition that triggers cleanup after txnState update will drop right after state update but before sending cleanup request, or it'll be killed during cleanup result processing, or commit partition primary replica will be re-elected, so many bad options... All in all that means, that in order to mark persistent txnState as ready for removal it's required to re-send cleanup even if it was previously finished on commit partition primary replica re-election. h3. Definition of Done * Populate txn persistent state with enlisted partitions on txnState update. It's required in order to define the set of nodes where cleanup should be send on commit partition primary replica re-election, including commit parition restart. > Do the txn cleanup procedure for all local finished transactions on > commitPartition primary replica election > > > Key: IGNITE-21762 > URL: https://issues.apache.org/jira/browse/IGNITE-21762 > Project: Ignite > Issue Type: Improvement >Reporter: Alexander Lapin >Priority: Major > Labels: ignite-3 > > h3. Motivation > As mentioned in IGNITE-21761 it's valid to remove finished transaction when > cleanup is fully replicated over all enlisted partitions. However it's > possible that commitPartition that triggers cleanup after txnState update > will drop right after state update but before sending cleanup request, or > it'll be killed during cleanup result processing, or commit partition primary > replica will be re-elected, so many bad options... > All in all that means, that in order to mark persistent txnState as ready for > removal it's required to re-send cleanup even if it was previously finished > on commit partition primary replica re-election. > h3. Definition of Done > * Populate txn persistent state with enlisted partitions on txnState update. > It's required in order to define the set of nodes where cleanup should be > send on commit partition primary replica re-election, including commit > parition restart. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21762) Do the txn cleanup procedure for all local finished transactions on commitPartition primary replica election
[ https://issues.apache.org/jira/browse/IGNITE-21762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-21762: - Description: h3. Motivation As mentioned in IGNITE-21761 it's valid to remove finished transaction when cleanup is fully replicated over all enlisted partitions. However it's possible that commitPartition that triggers cleanup after txnState update will drop right after a state update but before sending a cleanup request, or it'll be killed during cleanup result processing, or commit partition primary replica will be re-elected, so many bad options... All in all that means, that in order to mark persistent txnState as ready for removal it's required to re-send cleanup request on commit partition primary replica election even if it was previously finished. h3. Definition of Done * Populate txn persistent state with enlisted partitions on txnState update. It's required in order to determine the list of nodes to which the cleanup request will need to be sent. * In an asynchronous manner on commit partition primary replica election, scan local persistent txn state storage and send cleanup request in a durable manner. We should not use the thread of primary replica election notification for that. * On cleanup completion, meaning cleanup replication over majorities of all enlisted partitions txnStateVolatileMap will be populated with stamped txnState because of logic introduced in IGNITE-21761 was: h3. Motivation As mentioned in IGNITE-21761 it's valid to remove finished transaction when cleanup is fully replicated over all enlisted partitions. However it's possible that commitPartition that triggers cleanup after txnState update will drop right after state update but before sending cleanup request, or it'll be killed during cleanup result processing, or commit partition primary replica will be re-elected, so many bad options... All in all that means, that in order to mark persistent txnState as ready for removal it's required to re-send cleanup even if it was previously finished on commit partition primary replica re-election. h3. Definition of Done * Populate txn persistent state with enlisted partitions on txnState update. It's required in order to define the set of nodes where cleanup should be send on commit partition primary replica re-election, including commit parition restart. > Do the txn cleanup procedure for all local finished transactions on > commitPartition primary replica election > > > Key: IGNITE-21762 > URL: https://issues.apache.org/jira/browse/IGNITE-21762 > Project: Ignite > Issue Type: Improvement >Reporter: Alexander Lapin >Priority: Major > Labels: ignite-3 > > h3. Motivation > As mentioned in IGNITE-21761 it's valid to remove finished transaction when > cleanup is fully replicated over all enlisted partitions. However it's > possible that commitPartition that triggers cleanup after txnState update > will drop right after a state update but before sending a cleanup request, or > it'll be killed during cleanup result processing, or commit partition primary > replica will be re-elected, so many bad options... > All in all that means, that in order to mark persistent txnState as ready for > removal it's required to re-send cleanup request on commit partition primary > replica election even if it was previously finished. > h3. Definition of Done > * Populate txn persistent state with enlisted partitions on txnState update. > It's required in order to determine the list of nodes to which the cleanup > request will need to be sent. > * In an asynchronous manner on commit partition primary replica election, > scan local persistent txn state storage and send cleanup request in a durable > manner. We should not use the thread of primary replica election notification > for that. > * On cleanup completion, meaning cleanup replication over majorities of all > enlisted partitions txnStateVolatileMap will be populated with stamped > txnState because of logic introduced in IGNITE-21761 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21542) Documentation for Ignite security model
[ https://issues.apache.org/jira/browse/IGNITE-21542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Valuyskiy updated IGNITE-21542: Component/s: documentation > Documentation for Ignite security model > --- > > Key: IGNITE-21542 > URL: https://issues.apache.org/jira/browse/IGNITE-21542 > Project: Ignite > Issue Type: Task > Components: documentation >Reporter: Oleg Valuyskiy >Assignee: Oleg Valuyskiy >Priority: Minor > Labels: ise > Fix For: 2.17 > > Time Spent: 10m > Remaining Estimate: 0h > > It might be helpful to introduce documentation regarding what could be > expected from Ignite installations in terms of security. More specifically, > what Ignite users can expect when it comes to Discovery and Communication > ports inside and outside of the Ignite DMZ. -- This message was sent by Atlassian Jira (v8.20.10#820010)
