[jira] [Updated] (IGNITE-24343) Use ZonePartitionId as commitPartitionId
[ https://issues.apache.org/jira/browse/IGNITE-24343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-24343: - Ignite Flags: (was: Docs Required,Release Notes Required) > Use ZonePartitionId as commitPartitionId > > > Key: IGNITE-24343 > URL: https://issues.apache.org/jira/browse/IGNITE-24343 > Project: Ignite > Issue Type: Improvement >Reporter: Alexander Lapin >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-24343) Use ZonePartitionId as commitPartitionId
Alexander Lapin created IGNITE-24343: Summary: Use ZonePartitionId as commitPartitionId Key: IGNITE-24343 URL: https://issues.apache.org/jira/browse/IGNITE-24343 Project: Ignite Issue Type: Improvement Reporter: Alexander Lapin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24343) Use ZonePartitionId as commitPartitionId
[ https://issues.apache.org/jira/browse/IGNITE-24343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-24343: - Labels: ignite-3 (was: ) > Use ZonePartitionId as commitPartitionId > > > Key: IGNITE-24343 > URL: https://issues.apache.org/jira/browse/IGNITE-24343 > Project: Ignite > Issue Type: Improvement >Reporter: Alexander Lapin >Priority: Major > Labels: ignite-3 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24343) Use ZonePartitionId as commitPartitionId
[ https://issues.apache.org/jira/browse/IGNITE-24343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-24343: - Description: h3. Motivation Side note: we are not consistent in what commitPartitionId means: usually we assume that it's tableId + partitionIndex however in storage API, e.g. in PartitionDataStorage#addWrite commitPartitionId itself means partition index along with dedicated commitTableId. Below I will assume that commitPartitionId is a combination of table/zoneId and partitionIndex. * CommitPartition is a partition where where persisted txnState is stored. The state is atomically switched within TxFinish transaction phase. * CommitPartitionId is used in order to evaluate writeIntents through commitPartition path. * Write Intent(WI) resolution is a process that allows to define whether WI should be considered as applied (either committed or aborted) or ignored because corresponding transaction is in PENDING state. Simply, for WI(txId) ** Tx(txId).isPending() -> ignore WI (return previous value, or null if there's no such value). ** Tx(txId).isAborted() -> ignore WI (return previous value, or null if there's no such value). ** Tx(txId).isCommited() -> consider WI as a common value. * There are 3 different pathes of WI Resolution: ** Local path. Every node that participates in tx flow has volatile txnStateMap, thus it worth checking locally whether we already know the state of corresponding transaction. ** If local path is not applicable we will use Coordinator Path by requesting txn state from cooresponding coordinator. ** If Coordinator Path also not applicable we will use Commit Partition Path. Hence it's required to know where commit partition is. ** For more details please check [IEP-91|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211885498#IEP91:Transactionprotocol-ROtransactions.1] Worth mentioning that IEP is a bit outdated and both RO and RW transaction may see WI and thus do WI Resolution. Seems that we are done with [taxidermy|https://en.wikipedia.org/wiki/Taxonomy]. h3. Definition of Done * Within collocation track we do store zoneId + partitionIndex as commitPartitionId instead of tablerId + partitionIndex. * WI resolution works on top of ZonePartitionId as commitPartitionId. was: h3. Motivation Side note: we are not consistent in what commitPartitionId means: usually we assume that it's tableId + partitionIndex however in storage API, e.g. in PartitionDataStorage#addWrite commitPartitionId itself means partition index along with dedicated commitTableId. Below I will assume that commitPartitionId is a combination of table/zoneId and partitionIndex. * CommitPartition is a partition where where persisted txnState is stored. The state is atomically switched within TxFinish transaction phase. * CommitPartitionId is used in order to evaluate writeIntents through commitPartition path. * Write Intent(WI) resolution is a process that allows to define whether WI should be considered as applied (either committed or aborted) or ignored because corresponding transaction is in PENDING state. Simply, for WI(txId) ** Tx(txId).isPending() -> ignore WI (return previous value, or null if there's no such value). ** Tx(txId).isAborted() -> ignore WI (return previous value, or null if there's no such value). ** Tx(txId).isCommited() -> consider WI as a common value. * There are 3 different pathes of WI Resolution: ** Local path. Every node that participates in tx flow has volatile txnStateMap, thus it worth checking locally whether we already know the state of corresponding transaction. ** If local path is not applicable we will use Coordinator Path by requesting txn state from cooresponding coordinator. ** If Coordinator Path also not applicable we will use Commit Partition Path. Hence it's required to know where commit partition is. ** For more details please check [IEP-91|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211885498#IEP91:Transactionprotocol-ROtransactions.1] Worth mentioning that IEP is a bit outdated and both RO and RW transaction may see WI and thus do WI Resolution. Seems that we are done with [taxidermy|https://en.wikipedia.org/wiki/Taxonomy]. > Use ZonePartitionId as commitPartitionId > > > Key: IGNITE-24343 > URL: https://issues.apache.org/jira/browse/IGNITE-24343 > Project: Ignite > Issue Type: Improvement >Reporter: Alexander Lapin >Priority: Major > Labels: ignite-3 > > h3. Motivation > Side note: we are not consistent in what commitPartitionId means: usually we > assume that it's tableId + partitionIndex however in storage API, e.g. in > PartitionDataStorage#addWrite commitPartitionId itself means partition index > along with dedicated >
[jira] [Resolved] (IGNITE-22112) [SQL] TPC-H q9 query with sc=0.1 takes long time
[ https://issues.apache.org/jira/browse/IGNITE-22112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikita Sivkov resolved IGNITE-22112. Resolution: Cannot Reproduce The issue is not reproducible anymore. Q9 takes around 10 seconds now. > [SQL] TPC-H q9 query with sc=0.1 takes long time > > > Key: IGNITE-22112 > URL: https://issues.apache.org/jira/browse/IGNITE-22112 > Project: Ignite > Issue Type: Bug > Components: sql >Reporter: Nikita Sivkov >Priority: Major > Labels: ignite-3, ignite3_performance > > Benchmark: > [https://github.com/cmu-db/benchbase/tree/main/src/main/java/com/oltpbenchmark/benchmarks/tpch] > > h1. Setup > * 1 server node > * TPC-H with scale factor = 0.1 > h1. Steps > # Start an Ignite node > # Run benchbase with {{-s 1 --create=true --load=true --execute=false}} to > preload data > # Observe via the benchbase log that the data was successfully loaded > # Run {{benchbase with -s 1 --create=false --load=false --execute=true}} to > run the benchmark > h1. Expected result > The benchmark finishes after warmup + duration time in about the same time as > other requests. > h1. Actual result > The > [Q9|https://github.com/cmu-db/benchbase/blob/main/src/main/java/com/oltpbenchmark/benchmarks/tpch/procedures/Q9.java] > query has significantly longer execution time. For example, it may take 10 > minutes, while the rest queries take ~1-15 seconds. > {noformat} > SELECT > nation, > o_year, > SUM(amount) AS sum_profit > FROM > ( > SELECT > n_name AS nation, > EXTRACT(YEAR > FROM > o_orderdate) AS o_year, > l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity AS > amount > FROM > part, > supplier, > lineitem, > partsupp, > orders, > nation > WHERE > s_suppkey = l_suppkey > AND ps_suppkey = l_suppkey > AND ps_partkey = l_partkey > AND p_partkey = l_partkey > AND o_orderkey = l_orderkey > AND s_nationkey = n_nationkey > AND p_name LIKE ? > ) > AS profit > GROUP BY > nation, > o_year > ORDER BY > nation, > o_year DESC{noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24340) Queries permanently causing lock table overflow get retried forever
[ https://issues.apache.org/jira/browse/IGNITE-24340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Puchkovskiy updated IGNITE-24340: --- Description: Here is a scenario (requires implicit PKs): 1. CREATE TABLE integers(i INTEGER) 2. INSERT INTO integers VALUES (0), (0), (0), (0) 3. Repeat the following query a few times (without using explicit transactions): 'INSERT INTO integers SELECT * FROM integers' On a certain iteration, the query from item 3 hangs at least for 10 minutes (probably, it actually hangs forever). This is because HeapLockManager by default has a limited number of slots (by default, it's 1024 * 1024 ~ 1M slots). If there are too many locks taken at the same time, the slots are exhausted and lock attempt fails. It seems that SQL just repeats a query in such a case (at least, for implicit transactions), and there is no repeat limit. As a result, if user's query takes too many locks, it might not succeed at all (with given lock table size), but the actual reason ('too small lock table') will not be communicated to the user. The query will simply hang forever, or, if a statement/query/transaction timeout is used, a timeout will happen. This seems to be a bad UX. > Queries permanently causing lock table overflow get retried forever > --- > > Key: IGNITE-24340 > URL: https://issues.apache.org/jira/browse/IGNITE-24340 > Project: Ignite > Issue Type: Bug >Reporter: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > > Here is a scenario (requires implicit PKs): > 1. CREATE TABLE integers(i INTEGER) > 2. INSERT INTO integers VALUES (0), (0), (0), (0) > 3. Repeat the following query a few times (without using explicit > transactions): 'INSERT INTO integers SELECT * FROM integers' > > On a certain iteration, the query from item 3 hangs at least for 10 minutes > (probably, it actually hangs forever). This is because HeapLockManager by > default has a limited number of slots (by default, it's 1024 * 1024 ~ 1M > slots). If there are too many locks taken at the same time, the slots are > exhausted and lock attempt fails. It seems that SQL just repeats a query in > such a case (at least, for implicit transactions), and there is no repeat > limit. > As a result, if user's query takes too many locks, it might not succeed at > all (with given lock table size), but the actual reason ('too small lock > table') will not be communicated to the user. The query will simply hang > forever, or, if a statement/query/transaction timeout is used, a timeout will > happen. > This seems to be a bad UX. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24343) Use ZonePartitionId as commitPartitionId
[ https://issues.apache.org/jira/browse/IGNITE-24343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-24343: - Description: h3. Motivation Side note: we are not consistent in what commitPartitionId means: usually we assume that it's tableId + partitionIndex however in storage API, e.g. in PartitionDataStorage#addWrite commitPartitionId itself means partition index along with dedicated commitTableId. Below I will assume that commitPartitionId is a combination of table/zoneId and partitionIndex. * CommitPartition is a partition where where persisted txnState is stored. The state is atomically switched within TxFinish transaction phase. * CommitPartitionId is used in order to evaluate writeIntents through commitPartition path. * Write Intent(WI) resolution is a process that allows to define whether WI should be considered as applied (either committed or aborted) or ignored because corresponding transaction is in PENDING state. Simply, for WI(txId) ** Tx(txId).isPending() -> ignore WI (return previous value, or null if there's no such value). ** Tx(txId).isAborted() -> ignore WI (return previous value, or null if there's no such value). ** Tx(txId).isCommited() -> consider WI as a common value. * There are 3 different pathes of WI Resolution: ** Local path. Every node that participates in tx flow has volatile txnStateMap, thus it worth checking locally whether we already know the state of corresponding transaction. ** If local path is not applicable we will use Coordinator Path by requesting txn state from cooresponding coordinator. ** If Coordinator Path also not applicable we will use Commit Partition Path. Hence it's required to know where commit partition is. ** For more details please check [IEP-91|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211885498#IEP91:Transactionprotocol-ROtransactions.1] Worth mentioning that IEP is a bit outdated and both RO and RW transaction may see WI and thus do WI Resolution. Seems that we are done with [taxidermy|https://en.wikipedia.org/wiki/Taxonomy]. > Use ZonePartitionId as commitPartitionId > > > Key: IGNITE-24343 > URL: https://issues.apache.org/jira/browse/IGNITE-24343 > Project: Ignite > Issue Type: Improvement >Reporter: Alexander Lapin >Priority: Major > Labels: ignite-3 > > h3. Motivation > Side note: we are not consistent in what commitPartitionId means: usually we > assume that it's tableId + partitionIndex however in storage API, e.g. in > PartitionDataStorage#addWrite commitPartitionId itself means partition index > along with dedicated > commitTableId. Below I will assume that commitPartitionId is a combination of > table/zoneId and partitionIndex. > * CommitPartition is a partition where where persisted txnState is stored. > The state is atomically switched within TxFinish transaction phase. > * CommitPartitionId is used in order to evaluate writeIntents through > commitPartition path. > * Write Intent(WI) resolution is a process that allows to define whether WI > should be considered as applied (either committed or aborted) or ignored > because corresponding transaction is in PENDING state. Simply, for WI(txId) > ** Tx(txId).isPending() -> ignore WI (return previous value, or null if > there's no such value). > ** Tx(txId).isAborted() -> ignore WI (return previous value, or null if > there's no such value). > ** Tx(txId).isCommited() -> consider WI as a common value. > * There are 3 different pathes of WI Resolution: > ** Local path. Every node that participates in tx flow has volatile > txnStateMap, thus it worth checking locally whether we already know the state > of corresponding transaction. > ** If local path is not applicable we will use Coordinator Path by > requesting txn state from cooresponding coordinator. > ** If Coordinator Path also not applicable we will use Commit Partition > Path. Hence it's required to know where commit partition is. > ** For more details please check > [IEP-91|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211885498#IEP91:Transactionprotocol-ROtransactions.1] > Worth mentioning that IEP is a bit outdated and both RO and RW transaction > may see WI and thus do WI Resolution. > > Seems that we are done with > [taxidermy|https://en.wikipedia.org/wiki/Taxonomy]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-24334) Fix flaky ClientSessionOutboundQueueLimitTest
[ https://issues.apache.org/jira/browse/IGNITE-24334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Petrov reassigned IGNITE-24334: --- Assignee: Mikhail Petrov > Fix flaky ClientSessionOutboundQueueLimitTest > - > > Key: IGNITE-24334 > URL: https://issues.apache.org/jira/browse/IGNITE-24334 > Project: Ignite > Issue Type: Test >Reporter: Mikhail Petrov >Assignee: Mikhail Petrov >Priority: Minor > > See > https://ci2.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=-8176303230924033055&tab=testDetails > for more details. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24343) Use ZonePartitionId as commitPartitionId
[ https://issues.apache.org/jira/browse/IGNITE-24343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-24343: - Epic Link: IGNITE-22115 > Use ZonePartitionId as commitPartitionId > > > Key: IGNITE-24343 > URL: https://issues.apache.org/jira/browse/IGNITE-24343 > Project: Ignite > Issue Type: Improvement >Reporter: Alexander Lapin >Priority: Major > Labels: ignite-3 > > h3. Motivation > Side note: we are not consistent in what commitPartitionId means: usually we > assume that it's tableId + partitionIndex however in storage API, e.g. in > PartitionDataStorage#addWrite commitPartitionId itself means partition index > along with dedicated > commitTableId. Below I will assume that commitPartitionId is a combination of > table/zoneId and partitionIndex. > * CommitPartition is a partition where where persisted txnState is stored. > The state is atomically switched within TxFinish transaction phase. > * CommitPartitionId is used in order to evaluate writeIntents through > commitPartition path. > * Write Intent(WI) resolution is a process that allows to define whether WI > should be considered as applied (either committed or aborted) or ignored > because corresponding transaction is in PENDING state. Simply, for WI(txId) > ** Tx(txId).isPending() -> ignore WI (return previous value, or null if > there's no such value). > ** Tx(txId).isAborted() -> ignore WI (return previous value, or null if > there's no such value). > ** Tx(txId).isCommited() -> consider WI as a common value. > * There are 3 different pathes of WI Resolution: > ** Local path. Every node that participates in tx flow has volatile > txnStateMap, thus it worth checking locally whether we already know the state > of corresponding transaction. > ** If local path is not applicable we will use Coordinator Path by > requesting txn state from cooresponding coordinator. > ** If Coordinator Path also not applicable we will use Commit Partition > Path. Hence it's required to know where commit partition is. > ** For more details please check > [IEP-91|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211885498#IEP91:Transactionprotocol-ROtransactions.1] > Worth mentioning that IEP is a bit outdated and both RO and RW transaction > may see WI and thus do WI Resolution. > > Seems that we are done with > [taxidermy|https://en.wikipedia.org/wiki/Taxonomy]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (IGNITE-24339) Flaky test DdlCommandHandlerExceptionHandlingTest#testZoneNotFoundOnDrop2
[ https://issues.apache.org/jira/browse/IGNITE-24339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Iurii Gerzhedovich reopened IGNITE-24339: - > Flaky test DdlCommandHandlerExceptionHandlingTest#testZoneNotFoundOnDrop2 > - > > Key: IGNITE-24339 > URL: https://issues.apache.org/jira/browse/IGNITE-24339 > Project: Ignite > Issue Type: Improvement > Components: sql >Reporter: Iurii Gerzhedovich >Priority: Major > Labels: ignite-3 > > The test > org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandlerExceptionHandlingTest#testZoneNotFoundOnDrop2 > is flaky with the following stacktrace: > > {code:java} > java.lang.AssertionError: java.util.concurrent.ExecutionException: > java.lang.IllegalArgumentException: Time is out of bounds: 0 at > org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:78) > at > org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:35) > at org.hamcrest.TypeSafeMatcher.matches(TypeSafeMatcher.java:83) at > org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:10) at > org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6) at > org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandlerExceptionHandlingTest.testZoneNotFoundOnDrop2(DdlCommandHandlerExceptionHandlingTest.java:111) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) at > java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at > java.base/java.util.ArrayList.forEach(ArrayList.java:1541) Caused by: > java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: > Time is out of bounds: 0 at > java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) > at > java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2028) > at > org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:74) > ... 8 more Caused by: java.lang.IllegalArgumentException: Time is out of > bounds: 0 at > org.apache.ignite.internal.hlc.HybridTimestamp.(HybridTimestamp.java:111) > at > org.apache.ignite.internal.hlc.HybridTimestamp.hybridTimestamp(HybridTimestamp.java:131) > at > org.apache.ignite.internal.catalog.commands.CatalogUtils.clusterWideEnsuredActivationTimestamp(CatalogUtils.java:537) > at > org.apache.ignite.internal.catalog.CatalogManagerImpl.calcClusterWideEnsureActivationTime(CatalogManagerImpl.java:376) > at > org.apache.ignite.internal.catalog.CatalogManagerImpl.awaitVersionActivation(CatalogManagerImpl.java:370) > at > org.apache.ignite.internal.catalog.CatalogManagerImpl.awaitVersionActivation(CatalogManagerImpl.java:364) > at > java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106) > at > java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2241) > at > org.apache.ignite.internal.catalog.CatalogManagerImpl.saveUpdateAndWaitForActivation(CatalogManagerImpl.java:314) > at > org.apache.ignite.internal.catalog.CatalogManagerImpl.execute(CatalogManagerImpl.java:235) > at > org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandler.handle(DdlCommandHandler.java:76) > ... 4 more{code} > Initial version of Catalog uses {{0L}} as activation time, which is not valid > value according to HybridTimestamp's constructor: > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (IGNITE-24339) Flaky test DdlCommandHandlerExceptionHandlingTest#testZoneNotFoundOnDrop2
[ https://issues.apache.org/jira/browse/IGNITE-24339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Iurii Gerzhedovich resolved IGNITE-24339. - Resolution: Invalid > Flaky test DdlCommandHandlerExceptionHandlingTest#testZoneNotFoundOnDrop2 > - > > Key: IGNITE-24339 > URL: https://issues.apache.org/jira/browse/IGNITE-24339 > Project: Ignite > Issue Type: Improvement > Components: sql >Reporter: Iurii Gerzhedovich >Priority: Major > Labels: ignite-3 > > The test > org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandlerExceptionHandlingTest#testZoneNotFoundOnDrop2 > is flaky with the following stacktrace: > > {code:java} > java.lang.AssertionError: java.util.concurrent.ExecutionException: > java.lang.IllegalArgumentException: Time is out of bounds: 0 at > org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:78) > at > org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:35) > at org.hamcrest.TypeSafeMatcher.matches(TypeSafeMatcher.java:83) at > org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:10) at > org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6) at > org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandlerExceptionHandlingTest.testZoneNotFoundOnDrop2(DdlCommandHandlerExceptionHandlingTest.java:111) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) at > java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at > java.base/java.util.ArrayList.forEach(ArrayList.java:1541) Caused by: > java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: > Time is out of bounds: 0 at > java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) > at > java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2028) > at > org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:74) > ... 8 more Caused by: java.lang.IllegalArgumentException: Time is out of > bounds: 0 at > org.apache.ignite.internal.hlc.HybridTimestamp.(HybridTimestamp.java:111) > at > org.apache.ignite.internal.hlc.HybridTimestamp.hybridTimestamp(HybridTimestamp.java:131) > at > org.apache.ignite.internal.catalog.commands.CatalogUtils.clusterWideEnsuredActivationTimestamp(CatalogUtils.java:537) > at > org.apache.ignite.internal.catalog.CatalogManagerImpl.calcClusterWideEnsureActivationTime(CatalogManagerImpl.java:376) > at > org.apache.ignite.internal.catalog.CatalogManagerImpl.awaitVersionActivation(CatalogManagerImpl.java:370) > at > org.apache.ignite.internal.catalog.CatalogManagerImpl.awaitVersionActivation(CatalogManagerImpl.java:364) > at > java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106) > at > java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2241) > at > org.apache.ignite.internal.catalog.CatalogManagerImpl.saveUpdateAndWaitForActivation(CatalogManagerImpl.java:314) > at > org.apache.ignite.internal.catalog.CatalogManagerImpl.execute(CatalogManagerImpl.java:235) > at > org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandler.handle(DdlCommandHandler.java:76) > ... 4 more{code} > Initial version of Catalog uses {{0L}} as activation time, which is not valid > value according to HybridTimestamp's constructor: > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24294) IgniteCatalog API. Table's IndexDefinition reports incorrect name/sorting of index column
[ https://issues.apache.org/jira/browse/IGNITE-24294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Iurii Gerzhedovich updated IGNITE-24294: Description: Created an index with a column with ASC direction but a method that returns table definition and indices returns DEFAULT instead. Reproducer: {code:java} @Test public void test() { sql("CREATE SCHEMA s1"); sql("CREATE TABLE s1.c1 (" + "id INT PRIMARY KEY," + "val1 INT," + "val2 INT" + ")"); sql("CREATE INDEX s1_c1_idx1 ON s1.c1 USING HASH (val1)"); sql("CREATE INDEX s1_c1_idx2 ON s1.c1 USING SORTED (val1 desc, val2 asc)"); TableDefinition definition = catalog().tableDefinition(QualifiedName.of("S1", "C1")); for (var row : sql("SELECT * FROM SYSTEM.INDEXES WHERE table_name='C1'")) { System.err.println(row); } List indexes = definition.indexes(); assertNotNull(indexes); assertEquals(2, indexes.size()); System.err.println(indexes.get(0).name()); System.err.println(indexes.get(1).name()); List sortedDefs = new ArrayList<>(indexes); sortedDefs.sort(Comparator.comparing(IndexDefinition::name)); { IndexDefinition index = sortedDefs.get(0); assertEquals("S1_C1_IDX1", index.name()); assertEquals(IndexType.HASH, index.type()); List columns = index.columns(); assertEquals(List.of( ColumnSorted.column("VAL1") ), columns); } { IndexDefinition index = sortedDefs.get(1); assertEquals("S1_C1_IDX2", index.name()); assertEquals(IndexType.SORTED, index.type()); List columns = index.columns(); assertEquals(List.of( ColumnSorted.column("VAL1", SortOrder.DESC), ColumnSorted.column("VAL2", SortOrder.ASC) ), columns); } } {code} Output: {noformat} [20, S1_C1_IDX2, 17, C1, 16, S1, SORTED, false, VAL1 DESC, VAL2 ASC, AVAILABLE] [18, C1_PK, 17, C1, 16, S1, HASH, true, ID, AVAILABLE] [19, S1_C1_IDX1, 17, C1, 16, S1, HASH, false, VAL1, AVAILABLE] {noformat} Error: {noformat} org.opentest4j.AssertionFailedError: Expected :[ColumnSorted{columnName='VAL1', sortOrder=DESC}, ColumnSorted{columnName='VAL2', sortOrder=ASC}] Actual :[ColumnSorted{columnName='VAL1 DESC', sortOrder=DEFAULT}, ColumnSorted{columnName='VAL2 ASC', sortOrder=DEFAULT}] {noformat} was: Created an index with a column with ASC direction but a method that returns table definition and indices returns DEFAULT instead. Reproducer: {code:java} @Test public void test() { sql("CREATE SCHEMA s1"); sql("CREATE TABLE s1.c1 (" + "id INT PRIMARY KEY," + "val1 INT," + "val2 INT" + ")"); sql("CREATE INDEX s1_c1_idx1 ON s1.c1 USING HASH (val1)"); sql("CREATE INDEX s1_c1_idx2 ON s1.c1 USING SORTED (val1 desc, val2 asc)"); TableDefinition definition = catalog().tableDefinition(QualifiedName.of("S1", "C1")); for (var row : sql("SELECT * FROM SYSTEM.INDEXES WHERE table_name='C1'")) { System.err.println(row); } List indexes = definition.indexes(); assertNotNull(indexes); assertEquals(2, indexes.size()); System.err.println(indexes.get(0).name()); System.err.println(indexes.get(1).name()); List sortedDefs = new ArrayList<>(indexes); sortedDefs.sort(Comparator.comparing(IndexDefinition::name)); { IndexDefinition index = sortedDefs.get(0); assertEquals("S1_C1_IDX1", index.name()); assertEquals(IndexType.HASH, index.type()); List columns = index.columns(); assertEquals(List.of( ColumnSorted.column("VAL1") ), columns); } { IndexDefinition index = sortedDefs.get(1); assertEquals("S1_C1_IDX2", index.name()); assertEquals(IndexType.SORTED, index.type()); List columns = index.columns(); assertEquals(List.of( ColumnSorted.column("VAL1", SortOrder.DESC), ColumnSorted.column("VAL2", SortOrder.ASC) ), columns); } } {code} Output: {noformat} [20, S1_C1_IDX2, 17, C1, 16, S1, SORTED, false, VAL1 DESC, VAL2 ASC, AVAILABLE] [18, C1_PK, 17, C1, 16, S1, HASH, true, ID, AVAILABLE] [19, S1_C1_IDX1, 17, C1, 16, S1, HASH, false, VAL1, AVAILABLE] {noformat} Error: {noformat} org.opentest4j.AssertionFailedError: Expected :[ColumnSorted{columnName='VAL1', sortOrder=DESC}, ColumnSorted{columnName='VAL2', sortOrder=ASC}] Actual :[ColumnSorted{columnName='VAL1 DESC', sortOrder=DEFAULT}, ColumnS
[jira] [Comment Edited] (IGNITE-24299) Test ticket
[ https://issues.apache.org/jira/browse/IGNITE-24299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17921713#comment-17921713 ] Vyacheslav Koptilin edited comment on IGNITE-24299 at 1/29/25 10:50 AM: Hello [~vdmitrienko] , Could you please clarify what the purpose of this ticket is? was (Author: slava.koptilin): Hello [~vdmitrienko] , Could you please clarify what the purpose of this ticket? > Test ticket > --- > > Key: IGNITE-24299 > URL: https://issues.apache.org/jira/browse/IGNITE-24299 > Project: Ignite > Issue Type: Test >Reporter: Vladimir Dmitrienko >Assignee: Vladimir Dmitrienko >Priority: Trivial > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-22621) Implement transaction storage for the zone replica listener
[ https://issues.apache.org/jira/browse/IGNITE-22621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-22621: - Summary: Implement transaction storage for the zone replica listener (was: Implement transacation storage for the zone replica listener) > Implement transaction storage for the zone replica listener > --- > > Key: IGNITE-22621 > URL: https://issues.apache.org/jira/browse/IGNITE-22621 > Project: Ignite > Issue Type: Improvement >Reporter: Kirill Gusakov >Assignee: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > Attachments: Screenshot 2025-01-29 at 10.29.28 AM.png > > > h3. Motivation > Few words about general context. Tx Finish flow is a crucial step in AI3 Tx > Protocol. Generally it consists of two phases along with some prerequisites. > 0. Prerequisites: schema validation, tx state check. > 1. Txn state switch phase. Atomic transaction commit/rollback implemented > through replicating persistent txn state(COMMITTED/ABORTED) to > commitPartition. Once it's done, transaction is considered as finished. > 2. Cleanup phase. Releasing locks, converting writeIntents to common values > or removing them depending on whether it's commit or abort. > h3. !Screenshot 2025-01-29 at 10.29.28 AM.png! Definition of Done > * Move processing TxFinishReplicaRequest from > PartitionReplicaListener#processTxFinishAction to > ZonePartitionRaftListener#processTxFinishAction by introducing corresponding > method. In order to do that, it's required to add some dependencies to > ZonePartitionRaftListener: > * > ** SchemaCompatibilityValidator > ** TxStateStorage // Adding the storage isn't trivial. > ** TxManager > ** RaftCommandRunner // In order to run TxFinishCommand -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-24342) [Flaky] Cannot reliably start 3-nodes cluster on a single Windows machine
Andrey Khitrin created IGNITE-24342: --- Summary: [Flaky] Cannot reliably start 3-nodes cluster on a single Windows machine Key: IGNITE-24342 URL: https://issues.apache.org/jira/browse/IGNITE-24342 Project: Ignite Issue Type: Bug Affects Versions: 3.0 Environment: A single Windows 10 machine with 32 Gb of RAM Reporter: Andrey Khitrin Attachments: logs.tgz This issue doesn't have a 100% reproducibility rate, but is frequent enough to observe. How to reproduce: # Try to start 3 AI nodes with a static `nodeFinder` on a single machine (configs are attached) {code:java} nodeFinder { netClusterNodes=[ "127.0.0.1:3344", "127.0.0.1:3345", "127.0.0.1:3346" ] type=STATIC } {code} Expected result: all nodes are up. Actual result: 2 of 3 nodes terminated with thread dumps, cannot initialize cluster. Key exceptions in logs: # "IllegalStateException: cannot send more responses than requests" (see attachment) # Various RAFT-related and timeout errors: {code:java} 2025-01-28 06:03:05:471 -0600 [ERROR][%TablesAmountCapacityMultiNodeTest_cluster_1%JRaft-Response-Processor-8][AbstractClientService] Fail to connect TablesAmountCapacityMultiNodeTest_cluster_0, exception: java.util.concurrent.TimeoutException. 2025-01-28 06:03:05:815 -0600 [INFO][%TablesAmountCapacityMultiNodeTest_cluster_1%JRaft-Request-Processor-24][NodeImpl] Node ignore PreVoteRequest from TablesAmountCapacityMultiNodeTest_cluster_0, term=2, currTerm=1, because the leader TablesAmountCapacityMultiNodeTest_cluster_1's lease is still valid. 2025-01-28 06:03:05:815 -0600 [ERROR][%TablesAmountCapacityMultiNodeTest_cluster_1%JRaft-Response-Processor-8][ReplicatorGroupImpl] Fail to check replicator connection to peer=TablesAmountCapacityMultiNodeTest_cluster_0, replicatorType=Follower. 2025-01-28 06:03:05:836 -0600 [ERROR][%TablesAmountCapacityMultiNodeTest_cluster_1%JRaft-Response-Processor-8][NodeImpl] Fail to add a replicator, peer=TablesAmountCapacityMultiNodeTest_cluster_0. {code} # Thread dumps in logs for 2 of 3 nodes (see attachment) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24275) Sql. Rework or replace HybridTimestampTracker.clientTracker
[ https://issues.apache.org/jira/browse/IGNITE-24275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Orlov updated IGNITE-24275: -- Issue Type: Bug (was: Improvement) > Sql. Rework or replace HybridTimestampTracker.clientTracker > --- > > Key: IGNITE-24275 > URL: https://issues.apache.org/jira/browse/IGNITE-24275 > Project: Ignite > Issue Type: Bug > Components: sql >Reporter: Pavel Pereslegin >Priority: Major > Labels: ignite-3 > > In summary, the requirements for a {{HybridTimestampTracker}} (*observable > time tracker*) are as follows: > * Observed time can be updated multiple times > * Observed time can be updated from different threads > * The updated value should be visible from different threads > * When updating the time, the tracker must provide a guarantee that the > updated value is monotonically increasing > From that perspective current implementation of > {{HybridTimestampTracker.clientTracker}} is broken. > * The method {{get()}} returns only initial value (so tracker cannot be used > multiple times, for example for SQL script processing) > * Consumer usually stores value to non-volatile field > * There is no check that the new value is greater than the previous one > We need to rework {{clientTracker}} to meet requirement or replace it with > {{atomicTracker}}. > p.s. check TODO in codebase -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-24275) Sql. Rework or replace HybridTimestampTracker.clientTracker
[ https://issues.apache.org/jira/browse/IGNITE-24275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922053#comment-17922053 ] Konstantin Orlov commented on IGNITE-24275: --- Below are two tests which failed due to the bug (add these to any subclass of {{ClusterPerClassIntegrationTest}}): {code} @Override protected void configureInitParameters(InitParametersBuilder builder) { builder.clusterConfiguration(TestIgnitionManager.PRODUCTION_CLUSTER_CONFIG_STRING); } @Test void test1() { String address = "127.0.0.1:10800"; try ( IgniteClient client1 = IgniteClient.builder() .addresses(address) .build(); IgniteClient client2 = IgniteClient.builder() .addresses(address) .build(); ) { client1.sql().execute(null, "CREATE TABLE my_table (id INT PRIMARY KEY, val INT)"); // Statement below is failed with [IGN-TX-11 Table was dropped], while the same case executed // through embedded client or JDBC will succeed. try (ResultSet rs = client2.sql().execute(null, "SELECT * FROM my_table")) { while (rs.hasNext()) { rs.next(); } } } } @Test void test2() { String address = "127.0.0.1:10800"; try ( IgniteClient client1 = IgniteClient.builder() .addresses(address) .build(); ) { // Statement below is failed with [IGN-SQL-4 Failed to validate query. From line 1, column 104 to line 1, column 111: Object 'MY_TABLE' not found] client1.sql().executeScript("" + "CREATE TABLE my_table (id INT PRIMARY KEY, val INT);" + "START TRANSACTION READ ONLY;" + "SELECT COUNT(val) FROM my_table;" + "COMMIT;" ); } } {code} > Sql. Rework or replace HybridTimestampTracker.clientTracker > --- > > Key: IGNITE-24275 > URL: https://issues.apache.org/jira/browse/IGNITE-24275 > Project: Ignite > Issue Type: Bug > Components: sql >Reporter: Pavel Pereslegin >Priority: Major > Labels: ignite-3 > > In summary, the requirements for a {{HybridTimestampTracker}} (*observable > time tracker*) are as follows: > * Observed time can be updated multiple times > * Observed time can be updated from different threads > * The updated value should be visible from different threads > * When updating the time, the tracker must provide a guarantee that the > updated value is monotonically increasing > From that perspective current implementation of > {{HybridTimestampTracker.clientTracker}} is broken. > * The method {{get()}} returns only initial value (so tracker cannot be used > multiple times, for example for SQL script processing) > * Consumer usually stores value to non-volatile field > * There is no check that the new value is greater than the previous one > We need to rework {{clientTracker}} to meet requirement or replace it with > {{atomicTracker}}. > p.s. check TODO in codebase -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24304) Sql. System views. Introduce a system view for index columns
[ https://issues.apache.org/jira/browse/IGNITE-24304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Iurii Gerzhedovich updated IGNITE-24304: Description: Index system view stores index columns as a string, which makes in difficult to work with and requires manual parsing (e.g. IgniteCatalog provides index information - https://issues.apache.org/jira/browse/IGNITE-24294 ), it would be better to introduce a separate system view for index columns (or extend table columns view to include index columns). System view name: * index_columns System view columns: * schema_id * schema_name * table_id * table_name * index_id * index_name * column_name * column_ordinal - a position in an index definition * collation (desc, asc, null first, null last, etc, see IgniteCatalog SortOrder) (must be null for hash indices) was: Index system view stores index columns as a string, which makes in difficult to work with and requires manual parsing (e.g. IgniteCatalog provides index information - https://issues.apache.org/jira/browse/IGNITE-24294 ), it would be better to introduce a separate system view for index columns (or extend table columns view to include index columns). System view name: * index_columns System view columns: * schema_id * schema_name * table_id * table_name * index_id * index_name * column_ordinal - a position in an index definition * collation (desc, asc, null first, null last, etc, see IgniteCatalog SortOrder) (must be null for hash indices) > Sql. System views. Introduce a system view for index columns > > > Key: IGNITE-24304 > URL: https://issues.apache.org/jira/browse/IGNITE-24304 > Project: Ignite > Issue Type: Improvement > Components: sql >Reporter: Maksim Zhuravkov >Priority: Major > Labels: ignite-3 > > Index system view stores index columns as a string, which makes in difficult > to work with and requires manual parsing (e.g. IgniteCatalog provides index > information - https://issues.apache.org/jira/browse/IGNITE-24294 ), it would > be better to introduce a separate system view for index columns (or extend > table columns view to include index columns). > System view name: > * index_columns > System view columns: > * schema_id > * schema_name > * table_id > * table_name > * index_id > * index_name > * column_name > * column_ordinal - a position in an index definition > * collation (desc, asc, null first, null last, etc, see IgniteCatalog > SortOrder) (must be null for hash indices) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24275) Sql. Rework or replace HybridTimestampTracker.clientTracker
[ https://issues.apache.org/jira/browse/IGNITE-24275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Pereslegin updated IGNITE-24275: -- Description: In summary, the requirements for a {{HybridTimestampTracker}} (*observable time tracker*) are as follows: * Observed time can be updated multiple times * Observed time can be updated from different threads * The updated value should be visible from different threads * When updating the time, the tracker must provide a guarantee that the updated value is monotonically increasing >From that perspective current implementation of >{{HybridTimestampTracker.clientTracker}} is broken. * The method {{get()}} returns only initial value (so tracker cannot be used multiple times, for example for SQL script processing) * Consumer usually stores value to non-volatile field * There is no check that the new value is greater than the previous one We need to rework {{clientTracker}} to meet requirement or replace it with {{atomicTracker}}. p.s. check TODO in codebase was: In summary, the requirements for a {{HybridTimestampTracker}} (*observable time tracker*) are as follows: * Observed time can be updated multiple times * Observed time can be updated from different threads * The updated value should be visible from different threads * When updating the time, the tracker must provide a guarantee that the updated value is monotonically increasing >From that perspective current implementation of >{{HybridTimestampTracker.clientTracker}} is broken. * The method {{get()}} returns only initial value (so tracker cannot be used multiple times, for example for SQL script processing) * Consumer usually stores value to non-volatile field * There is no check that the new value is greater than the previous one We need to rework {{clientTracker}} to meet requirement or replace it where with {{atomicTracker}}. p.s. check TODO in codebase > Sql. Rework or replace HybridTimestampTracker.clientTracker > --- > > Key: IGNITE-24275 > URL: https://issues.apache.org/jira/browse/IGNITE-24275 > Project: Ignite > Issue Type: Improvement > Components: sql >Reporter: Pavel Pereslegin >Priority: Major > Labels: ignite-3 > > In summary, the requirements for a {{HybridTimestampTracker}} (*observable > time tracker*) are as follows: > * Observed time can be updated multiple times > * Observed time can be updated from different threads > * The updated value should be visible from different threads > * When updating the time, the tracker must provide a guarantee that the > updated value is monotonically increasing > From that perspective current implementation of > {{HybridTimestampTracker.clientTracker}} is broken. > * The method {{get()}} returns only initial value (so tracker cannot be used > multiple times, for example for SQL script processing) > * Consumer usually stores value to non-volatile field > * There is no check that the new value is greater than the previous one > We need to rework {{clientTracker}} to meet requirement or replace it with > {{atomicTracker}}. > p.s. check TODO in codebase -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-24341) Sql. System views. Add ordinal column to table_columns
Maksim Zhuravkov created IGNITE-24341: - Summary: Sql. System views. Add ordinal column to table_columns Key: IGNITE-24341 URL: https://issues.apache.org/jira/browse/IGNITE-24341 Project: Ignite Issue Type: Improvement Components: sql Reporter: Maksim Zhuravkov TABLE_COLUMNS system view does not include information on column position in a table definition. Let's add `column_ordinal` column to TABLE_COLUMNS view that contains it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-24299) Test ticket
[ https://issues.apache.org/jira/browse/IGNITE-24299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922060#comment-17922060 ] Vyacheslav Koptilin commented on IGNITE-24299: -- Ok. Thank you for the clarification! Please don't forget to close the ticket then the issue is resolved. > Test ticket > --- > > Key: IGNITE-24299 > URL: https://issues.apache.org/jira/browse/IGNITE-24299 > Project: Ignite > Issue Type: Test >Reporter: Vladimir Dmitrienko >Assignee: Vladimir Dmitrienko >Priority: Trivial > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-24299) Test ticket
[ https://issues.apache.org/jira/browse/IGNITE-24299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922061#comment-17922061 ] Vladimir Dmitrienko commented on IGNITE-24299: -- Sure (y) > Test ticket > --- > > Key: IGNITE-24299 > URL: https://issues.apache.org/jira/browse/IGNITE-24299 > Project: Ignite > Issue Type: Test >Reporter: Vladimir Dmitrienko >Assignee: Vladimir Dmitrienko >Priority: Trivial > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-24339) Flaky test DdlCommandHandlerExceptionHandlingTest#testZoneNotFoundOnDrop2
Iurii Gerzhedovich created IGNITE-24339: --- Summary: Flaky test DdlCommandHandlerExceptionHandlingTest#testZoneNotFoundOnDrop2 Key: IGNITE-24339 URL: https://issues.apache.org/jira/browse/IGNITE-24339 Project: Ignite Issue Type: Improvement Components: sql Reporter: Iurii Gerzhedovich The test org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandlerExceptionHandlingTest#testZoneNotFoundOnDrop2 is flaky with the following stacktrace: {code:java} java.lang.AssertionError: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Time is out of bounds: 0 at org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:78) at org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:35) at org.hamcrest.TypeSafeMatcher.matches(TypeSafeMatcher.java:83) at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:10) at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6) at org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandlerExceptionHandlingTest.testZoneNotFoundOnDrop2(DdlCommandHandlerExceptionHandlingTest.java:111) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Time is out of bounds: 0 at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2028) at org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:74) ... 8 more Caused by: java.lang.IllegalArgumentException: Time is out of bounds: 0 at org.apache.ignite.internal.hlc.HybridTimestamp.(HybridTimestamp.java:111) at org.apache.ignite.internal.hlc.HybridTimestamp.hybridTimestamp(HybridTimestamp.java:131) at org.apache.ignite.internal.catalog.commands.CatalogUtils.clusterWideEnsuredActivationTimestamp(CatalogUtils.java:537) at org.apache.ignite.internal.catalog.CatalogManagerImpl.calcClusterWideEnsureActivationTime(CatalogManagerImpl.java:376) at org.apache.ignite.internal.catalog.CatalogManagerImpl.awaitVersionActivation(CatalogManagerImpl.java:370) at org.apache.ignite.internal.catalog.CatalogManagerImpl.awaitVersionActivation(CatalogManagerImpl.java:364) at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106) at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2241) at org.apache.ignite.internal.catalog.CatalogManagerImpl.saveUpdateAndWaitForActivation(CatalogManagerImpl.java:314) at org.apache.ignite.internal.catalog.CatalogManagerImpl.execute(CatalogManagerImpl.java:235) at org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandler.handle(DdlCommandHandler.java:76) ... 4 more{code} Initial version of Catalog uses {{0L}} as activation time, which is not valid value according to HybridTimestamp's constructor: -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24330) Revise DisasterRecoveryManager API in alignment with Colocation track
[ https://issues.apache.org/jira/browse/IGNITE-24330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mirza Aliev updated IGNITE-24330: - Description: h3. Motivation In [collocation|https://ggsystems.atlassian.net/issues/IGN-24435] track we change the way how partitions are aligned with tables, so partitions will be the part of zones and tables in one zone will share common partitions from a zone. Currently, the DisasterRecoveryManager API for reset/restart partition operations is based on the assumption that partitions are tied to table entities. Example of {{resetPartitions}} method description: {code:java} * @param zoneName Name of the distribution zone. Case-sensitive, without quotes. * @param schemaName Schema name. Case-sensitive, without quotes. * @param tableName Table name. Case-sensitive, without quotes. * @param partitionIds IDs of partitions to reset. If empty, reset all zone's partitions. */ private CompletableFuture resetPartitions( String zoneName, String schemaName, String tableName, Set partitionIds, ... ) { {code} Problems with the current API: * The current API assumes partitions are part of a specific table, making it unclear how to handle partition resets in the new colocation model. * Users don't want to affect partition distribution across all tables in a zone when specifying a reset for a particular table. We need to provide a way to use the DisasterRecoveryManager API in a way that reflects the new colocation model, where partitions belong to zones rather than being tied to individual tables. As a solution, we could introduce a revised DisasterRecoveryManagerV2 API that removes direct table references and operates at the zone level. Once the colocation track is completed, we can switch to DisasterRecoveryManagerV2 and deprecate the existing implementation. However, it is still unclear which methods will be required and how they will impact table distribution. h3. Definition of Done * DisasterRecoveryManager API is enriched with the methods to work with zones partitions was: h3. Motivation In [collocation|https://ggsystems.atlassian.net/issues/IGN-24435] track we change the way how partitions are aligned with tables, so partitions will be the part of zones and tables in one zone will share common partitions from a zone. Currently, the DisasterRecoveryManager API for reset/restart partition operations is based on the assumption that partitions are tied to table entities. Example of {{resetPartitions}} method description: {code:java} * @param zoneName Name of the distribution zone. Case-sensitive, without quotes. * @param schemaName Schema name. Case-sensitive, without quotes. * @param tableName Table name. Case-sensitive, without quotes. * @param partitionIds IDs of partitions to reset. If empty, reset all zone's partitions. */ private CompletableFuture resetPartitions( String zoneName, String schemaName, String tableName, Set partitionIds, ... ) { {code} Problems with the current API: * The current API assumes partitions are part of a specific table, making it unclear how to handle partition resets in the new colocation model. * Users don't want to affect partition distribution across all tables in a zone when specifying a reset for a particular table. We need to provide a way to use the DisasterRecoveryManager API in a way that reflects the new colocation model, where partitions belong to zones rather than being tied to individual tables. As a solution, we could introduce a revised DisasterRecoveryManagerV2 API that removes direct table references and operates at the zone level. Once the colocation track is completed, we can switch to DisasterRecoveryManagerV2 and deprecate the existing implementation. However, it is still unclear which methods will be required and how they will impact table distribution. > Revise DisasterRecoveryManager API in alignment with Colocation track > - > > Key: IGNITE-24330 > URL: https://issues.apache.org/jira/browse/IGNITE-24330 > Project: Ignite > Issue Type: Improvement >Reporter: Mirza Aliev >Priority: Major > Labels: ignite-3 > > h3. Motivation > In [collocation|https://ggsystems.atlassian.net/issues/IGN-24435] track we > change the way how partitions are aligned with tables, so partitions will be > the part of zones and tables in one zone will share common partitions from a > zone. > Currently, the DisasterRecoveryManager API for reset/restart partition > operations is based on the assumption that partitions are tied to table > entities. > Example of {{resetPart
[jira] [Updated] (IGNITE-24232) Implement correct node restart behaviour when forced pending
[ https://issues.apache.org/jira/browse/IGNITE-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mirza Aliev updated IGNITE-24232: - Epic Link: (was: IGNITE-22115) > Implement correct node restart behaviour when forced pending > > > Key: IGNITE-24232 > URL: https://issues.apache.org/jira/browse/IGNITE-24232 > Project: Ignite > Issue Type: Improvement >Reporter: Mirza Aliev >Priority: Major > Labels: ignite-3 > > h3. Motivation > In https://issues.apache.org/jira/browse/IGNITE-23780 we have improved > correct node behaviour when node restarts according to > stable/pending(forced/not forced) keys. Several places in the code where we > start partitions where improved. > The same improvements must be done based on collocation > stable/pending(forced/not forced) keys, where zones partitions are started. > Details could be found in the original ticket > https://issues.apache.org/jira/browse/IGNITE-23780 > h3. Definition of Done > * All changes from https://issues.apache.org/jira/browse/IGNITE-23780 in the > partition restart flow must be adopted for the collocation rebalance flow. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24230) Support Disaster Recovery methods for the collocation case
[ https://issues.apache.org/jira/browse/IGNITE-24230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mirza Aliev updated IGNITE-24230: - Epic Link: (was: IGNITE-22115) > Support Disaster Recovery methods for the collocation case > -- > > Key: IGNITE-24230 > URL: https://issues.apache.org/jira/browse/IGNITE-24230 > Project: Ignite > Issue Type: Improvement >Reporter: Mirza Aliev >Priority: Major > Labels: ignite-3 > > h3. Motivation > DisasterRecoveryManager has logic for reset partitions, which sets force > pending assignments. We need to support such functionality for collocation > pending keys. > Also there is logic which reacts on HA zone topology reduced > `DisasterRecoveryManager#onHaZoneTopologyReduce`. It calculates set of > partitions which lost majority, based on stable assignments. The same logic > must be done based on collocation stable keys. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24230) Support Disaster Recovery methods for the collocation case
[ https://issues.apache.org/jira/browse/IGNITE-24230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mirza Aliev updated IGNITE-24230: - Description: h3. Motivation Once we revise DisasterRecoveryManager API in alignment with Colocation track https://issues.apache.org/jira/browse/IGNITE-24330 and implement all related tickets, so DisasterRecoveryManager API is improved with new methods that work with zones' partitions, we need to implement REST API for all new methods. See {{DisasterRecoveryController}} h3. Definition of done * REST API of DisasterRecoveryManager is improved with new methods that were implemented for zones' partitions. was: h3. Motivation DisasterRecoveryManager has logic for reset partitions, which sets force pending assignments. We need to support such functionality for collocation pending keys. Also there is logic which reacts on HA zone topology reduced `DisasterRecoveryManager#onHaZoneTopologyReduce`. It calculates set of partitions which lost majority, based on stable assignments. The same logic must be done based on collocation stable keys. > Support Disaster Recovery methods for the collocation case > -- > > Key: IGNITE-24230 > URL: https://issues.apache.org/jira/browse/IGNITE-24230 > Project: Ignite > Issue Type: Improvement >Reporter: Mirza Aliev >Priority: Major > Labels: ignite-3 > > h3. Motivation > Once we revise DisasterRecoveryManager API in alignment with Colocation track > https://issues.apache.org/jira/browse/IGNITE-24330 and implement all related > tickets, so DisasterRecoveryManager API is improved with new methods that > work with zones' partitions, > we need to implement REST API for all new methods. See > {{DisasterRecoveryController}} > h3. Definition of done > * REST API of DisasterRecoveryManager is improved with new methods that were > implemented for zones' partitions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24331) Support DisasterRecoveryManager#localPartitionStates and globalPartitionStates in Collocation track
[ https://issues.apache.org/jira/browse/IGNITE-24331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mirza Aliev updated IGNITE-24331: - Summary: Support DisasterRecoveryManager#localPartitionStates and globalPartitionStates in Collocation track (was: Support DisasterRecoveryManager#localPartitionStates and globalPartitionStates in Colocation track) > Support DisasterRecoveryManager#localPartitionStates and > globalPartitionStates in Collocation track > --- > > Key: IGNITE-24331 > URL: https://issues.apache.org/jira/browse/IGNITE-24331 > Project: Ignite > Issue Type: Improvement >Reporter: Mirza Aliev >Priority: Major > Labels: ignite-3 > > h3. Motivation > This ticket is a part of the implementation track for > https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised > DisasterRecoveryManager API in alignment with Colocation track > In this ticket we have to provide a correct behaviour for > {{DisasterRecoveryManager#localPartitionStates}} and close method > {{DisasterRecoveryManager#globalPartitionStates}} which uses > {{localPartitionStates}}. > Currently, those methods return a mapping of {{TablePartitionId}} to the > mapping between a node name and a partition state. > To conform Collocation track, those methods must use {{ZonePartitionId}} and > hence return states of a zone's partition rather than table's. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24331) Support DisasterRecoveryManager#localPartitionStates and globalPartitionStates in Collocation track
[ https://issues.apache.org/jira/browse/IGNITE-24331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mirza Aliev updated IGNITE-24331: - Description: h3. Motivation This ticket is a part of the implementation track for https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised DisasterRecoveryManager API in alignment with Colocation track In this ticket we have to provide a correct behaviour for {{DisasterRecoveryManager#localPartitionStates}} and close method {{DisasterRecoveryManager#globalPartitionStates}} which uses {{localPartitionStates}}. Currently, those methods return a mapping of {{TablePartitionId}} to the mapping between a node name and a partition state. To conform Collocation track, those methods must use {{ZonePartitionId}} and hence return states of a zone's partition rather than table's. h3. Definition of Done * DisasterRecoveryManager API is enriched with {{localPartitionStates}} and {{globalPartitionStates}} that are work with zones' based partitions was: h3. Motivation This ticket is a part of the implementation track for https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised DisasterRecoveryManager API in alignment with Colocation track In this ticket we have to provide a correct behaviour for {{DisasterRecoveryManager#localPartitionStates}} and close method {{DisasterRecoveryManager#globalPartitionStates}} which uses {{localPartitionStates}}. Currently, those methods return a mapping of {{TablePartitionId}} to the mapping between a node name and a partition state. To conform Collocation track, those methods must use {{ZonePartitionId}} and hence return states of a zone's partition rather than table's. > Support DisasterRecoveryManager#localPartitionStates and > globalPartitionStates in Collocation track > --- > > Key: IGNITE-24331 > URL: https://issues.apache.org/jira/browse/IGNITE-24331 > Project: Ignite > Issue Type: Improvement >Reporter: Mirza Aliev >Priority: Major > Labels: ignite-3 > > h3. Motivation > This ticket is a part of the implementation track for > https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised > DisasterRecoveryManager API in alignment with Colocation track > In this ticket we have to provide a correct behaviour for > {{DisasterRecoveryManager#localPartitionStates}} and close method > {{DisasterRecoveryManager#globalPartitionStates}} which uses > {{localPartitionStates}}. > Currently, those methods return a mapping of {{TablePartitionId}} to the > mapping between a node name and a partition state. > To conform Collocation track, those methods must use {{ZonePartitionId}} and > hence return states of a zone's partition rather than table's. > h3. Definition of Done > * DisasterRecoveryManager API is enriched with {{localPartitionStates}} and > {{globalPartitionStates}} that are work with zones' based partitions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24332) Support DisasterRecoveryManager#resetPartitions in Collocation track
[ https://issues.apache.org/jira/browse/IGNITE-24332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mirza Aliev updated IGNITE-24332: - Summary: Support DisasterRecoveryManager#resetPartitions in Collocation track (was: Support DisasterRecoveryManager#resetPartitions in Colocation track) > Support DisasterRecoveryManager#resetPartitions in Collocation track > > > Key: IGNITE-24332 > URL: https://issues.apache.org/jira/browse/IGNITE-24332 > Project: Ignite > Issue Type: Improvement >Reporter: Mirza Aliev >Priority: Major > Labels: ignite-3 > > h3. Motivation > This ticket is a part of the implementation track for > https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised > DisasterRecoveryManager API in alignment with Colocation track > In this ticket we have to provide a correct behaviour for > {{DisasterRecoveryManager#resetPartitions}} for partitions from zones. > h3. Implementation notes > Taking into account that prerequisite with {{localPartitionStates}} is > already implemented (https://issues.apache.org/jira/browse/IGNITE-24331), in > this ticket we must just reuse code from the current > {{DisasterRecoveryManager#resetPartitions}}, use correct > {{localPartitionStates}} method and change > {{ZoneRebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}} key rather than > {{RebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24331) Support DisasterRecoveryManager#localPartitionStates and globalPartitionStates in Collocation track
[ https://issues.apache.org/jira/browse/IGNITE-24331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mirza Aliev updated IGNITE-24331: - Description: h3. Motivation This ticket is a part of the implementation track for https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised DisasterRecoveryManager API in alignment with Colocation track In this ticket we have to provide a correct behaviour for {{DisasterRecoveryManager#localPartitionStates}} and close method {{DisasterRecoveryManager#globalPartitionStates}} which uses {{localPartitionStates}}. Currently, those methods return a mapping of {{TablePartitionId}} to the mapping between a node name and a partition state. To conform Collocation track, those methods must use {{ZonePartitionId}} and hence return states of a zone's partition rather than table's. h3. Definition of Done * DisasterRecoveryManager API is enriched with {{localPartitionStates}} and {{globalPartitionStates}} methods that are work with zones' based partitions was: h3. Motivation This ticket is a part of the implementation track for https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised DisasterRecoveryManager API in alignment with Colocation track In this ticket we have to provide a correct behaviour for {{DisasterRecoveryManager#localPartitionStates}} and close method {{DisasterRecoveryManager#globalPartitionStates}} which uses {{localPartitionStates}}. Currently, those methods return a mapping of {{TablePartitionId}} to the mapping between a node name and a partition state. To conform Collocation track, those methods must use {{ZonePartitionId}} and hence return states of a zone's partition rather than table's. h3. Definition of Done * DisasterRecoveryManager API is enriched with {{localPartitionStates}} and {{globalPartitionStates}} that are work with zones' based partitions > Support DisasterRecoveryManager#localPartitionStates and > globalPartitionStates in Collocation track > --- > > Key: IGNITE-24331 > URL: https://issues.apache.org/jira/browse/IGNITE-24331 > Project: Ignite > Issue Type: Improvement >Reporter: Mirza Aliev >Priority: Major > Labels: ignite-3 > > h3. Motivation > This ticket is a part of the implementation track for > https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised > DisasterRecoveryManager API in alignment with Colocation track > In this ticket we have to provide a correct behaviour for > {{DisasterRecoveryManager#localPartitionStates}} and close method > {{DisasterRecoveryManager#globalPartitionStates}} which uses > {{localPartitionStates}}. > Currently, those methods return a mapping of {{TablePartitionId}} to the > mapping between a node name and a partition state. > To conform Collocation track, those methods must use {{ZonePartitionId}} and > hence return states of a zone's partition rather than table's. > h3. Definition of Done > * DisasterRecoveryManager API is enriched with {{localPartitionStates}} and > {{globalPartitionStates}} methods that are work with zones' based partitions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24332) Support DisasterRecoveryManager#resetPartitions in Collocation track
[ https://issues.apache.org/jira/browse/IGNITE-24332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mirza Aliev updated IGNITE-24332: - Description: h3. Motivation This ticket is a part of the implementation track for https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised DisasterRecoveryManager API in alignment with Colocation track In this ticket we have to provide a correct behaviour for {{DisasterRecoveryManager#resetPartitions}} for partitions from zones. h3. Implementation notes Taking into account that prerequisite with {{localPartitionStates}} is already implemented (https://issues.apache.org/jira/browse/IGNITE-24331), in this ticket we must just reuse code from the current {{DisasterRecoveryManager#resetPartitions}}, use correct {{localPartitionStates}} method and change {{ZoneRebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}} key rather than {{RebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}} h3. Definition of Done * DisasterRecoveryManager API is enriched with was: h3. Motivation This ticket is a part of the implementation track for https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised DisasterRecoveryManager API in alignment with Colocation track In this ticket we have to provide a correct behaviour for {{DisasterRecoveryManager#resetPartitions}} for partitions from zones. h3. Implementation notes Taking into account that prerequisite with {{localPartitionStates}} is already implemented (https://issues.apache.org/jira/browse/IGNITE-24331), in this ticket we must just reuse code from the current {{DisasterRecoveryManager#resetPartitions}}, use correct {{localPartitionStates}} method and change {{ZoneRebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}} key rather than {{RebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}} > Support DisasterRecoveryManager#resetPartitions in Collocation track > > > Key: IGNITE-24332 > URL: https://issues.apache.org/jira/browse/IGNITE-24332 > Project: Ignite > Issue Type: Improvement >Reporter: Mirza Aliev >Priority: Major > Labels: ignite-3 > > h3. Motivation > This ticket is a part of the implementation track for > https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised > DisasterRecoveryManager API in alignment with Colocation track > In this ticket we have to provide a correct behaviour for > {{DisasterRecoveryManager#resetPartitions}} for partitions from zones. > h3. Implementation notes > Taking into account that prerequisite with {{localPartitionStates}} is > already implemented (https://issues.apache.org/jira/browse/IGNITE-24331), in > this ticket we must just reuse code from the current > {{DisasterRecoveryManager#resetPartitions}}, use correct > {{localPartitionStates}} method and change > {{ZoneRebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}} key rather than > {{RebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}} > h3. Definition of Done > * DisasterRecoveryManager API is enriched with -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24331) Support DisasterRecoveryManager#localPartitionStates and globalPartitionStates in Collocation track
[ https://issues.apache.org/jira/browse/IGNITE-24331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mirza Aliev updated IGNITE-24331: - Description: h3. Motivation This ticket is a part of the implementation track for https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised DisasterRecoveryManager API in alignment with Colocation track In this ticket we have to provide a correct behaviour for {{DisasterRecoveryManager#localPartitionStates}} and close method {{DisasterRecoveryManager#globalPartitionStates}} which uses {{localPartitionStates}}. Currently, those methods return a mapping of {{TablePartitionId}} to the mapping between a node name and a partition state. To conform Collocation track, those methods must use {{ZonePartitionId}} and hence return states of a zone's partition rather than table's. h3. Definition of Done * DisasterRecoveryManager API is enriched with {{localPartitionStates}} and {{globalPartitionStates}} methods that work with zones' based partitions was: h3. Motivation This ticket is a part of the implementation track for https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised DisasterRecoveryManager API in alignment with Colocation track In this ticket we have to provide a correct behaviour for {{DisasterRecoveryManager#localPartitionStates}} and close method {{DisasterRecoveryManager#globalPartitionStates}} which uses {{localPartitionStates}}. Currently, those methods return a mapping of {{TablePartitionId}} to the mapping between a node name and a partition state. To conform Collocation track, those methods must use {{ZonePartitionId}} and hence return states of a zone's partition rather than table's. h3. Definition of Done * DisasterRecoveryManager API is enriched with {{localPartitionStates}} and {{globalPartitionStates}} methods that are work with zones' based partitions > Support DisasterRecoveryManager#localPartitionStates and > globalPartitionStates in Collocation track > --- > > Key: IGNITE-24331 > URL: https://issues.apache.org/jira/browse/IGNITE-24331 > Project: Ignite > Issue Type: Improvement >Reporter: Mirza Aliev >Priority: Major > Labels: ignite-3 > > h3. Motivation > This ticket is a part of the implementation track for > https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised > DisasterRecoveryManager API in alignment with Colocation track > In this ticket we have to provide a correct behaviour for > {{DisasterRecoveryManager#localPartitionStates}} and close method > {{DisasterRecoveryManager#globalPartitionStates}} which uses > {{localPartitionStates}}. > Currently, those methods return a mapping of {{TablePartitionId}} to the > mapping between a node name and a partition state. > To conform Collocation track, those methods must use {{ZonePartitionId}} and > hence return states of a zone's partition rather than table's. > h3. Definition of Done > * DisasterRecoveryManager API is enriched with {{localPartitionStates}} and > {{globalPartitionStates}} methods that work with zones' based partitions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24332) Support DisasterRecoveryManager#resetPartitions in Collocation track
[ https://issues.apache.org/jira/browse/IGNITE-24332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mirza Aliev updated IGNITE-24332: - Description: h3. Motivation This ticket is a part of the implementation track for https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised DisasterRecoveryManager API in alignment with Colocation track In this ticket we have to provide a correct behaviour for {{DisasterRecoveryManager#resetPartitions}} for partitions from zones. h3. Implementation notes Taking into account that prerequisite with {{localPartitionStates}} is already implemented (https://issues.apache.org/jira/browse/IGNITE-24331), in this ticket we must just reuse code from the current {{DisasterRecoveryManager#resetPartitions}}, use correct {{localPartitionStates}} method and change {{ZoneRebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}} key rather than {{RebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}} h3. Definition of Done * DisasterRecoveryManager API is enriched with {{resetPartitions}} method that works with zones' based partitions was: h3. Motivation This ticket is a part of the implementation track for https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised DisasterRecoveryManager API in alignment with Colocation track In this ticket we have to provide a correct behaviour for {{DisasterRecoveryManager#resetPartitions}} for partitions from zones. h3. Implementation notes Taking into account that prerequisite with {{localPartitionStates}} is already implemented (https://issues.apache.org/jira/browse/IGNITE-24331), in this ticket we must just reuse code from the current {{DisasterRecoveryManager#resetPartitions}}, use correct {{localPartitionStates}} method and change {{ZoneRebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}} key rather than {{RebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}} h3. Definition of Done * DisasterRecoveryManager API is enriched with > Support DisasterRecoveryManager#resetPartitions in Collocation track > > > Key: IGNITE-24332 > URL: https://issues.apache.org/jira/browse/IGNITE-24332 > Project: Ignite > Issue Type: Improvement >Reporter: Mirza Aliev >Priority: Major > Labels: ignite-3 > > h3. Motivation > This ticket is a part of the implementation track for > https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised > DisasterRecoveryManager API in alignment with Colocation track > In this ticket we have to provide a correct behaviour for > {{DisasterRecoveryManager#resetPartitions}} for partitions from zones. > h3. Implementation notes > Taking into account that prerequisite with {{localPartitionStates}} is > already implemented (https://issues.apache.org/jira/browse/IGNITE-24331), in > this ticket we must just reuse code from the current > {{DisasterRecoveryManager#resetPartitions}}, use correct > {{localPartitionStates}} method and change > {{ZoneRebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}} key rather than > {{RebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}} > h3. Definition of Done > * DisasterRecoveryManager API is enriched with {{resetPartitions}} method > that works with zones' based partitions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24335) Support DisasterRecoveryManager#restartPartition in Collocation track
[ https://issues.apache.org/jira/browse/IGNITE-24335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mirza Aliev updated IGNITE-24335: - Description: h3. Motivation This ticket is a part of the implementation track for https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised DisasterRecoveryManager API in alignment with Collocation track In this ticket we have to provide a correct behaviour for {{DisasterRecoveryManager#restartPartition}} for partitions from zones. h3. Implementation notes As usual for the such methods from DisasterRecoveryManager, we must switch to {{ZonePartitionId}}. Also this mechanism uses under the hood {{TableManager#restartPartition}}. Seems that we need to have similar method in {{PartitionReplicaLifecycleManager}}, which has all prerequisites like {{PartitionReplicaLifecycleManager#weakStopPartition}}, {{PartitionReplicaLifecycleManager#waitForMetadataCompleteness}} h3. Definition of Done * DisasterRecoveryManager API is enriched with {{restartPartition}} method that works with zones' based partitions was: h3. Motivation This ticket is a part of the implementation track for https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised DisasterRecoveryManager API in alignment with Collocation track In this ticket we have to provide a correct behaviour for {{DisasterRecoveryManager#restartPartition}} for partitions from zones. h3. Implementation notes As usual for the such methods from DisasterRecoveryManager, we must switch to {{ZonePartitionId}}. Also this mechanism uses under the hood {{TableManager#restartPartition}}. Seems that we need to have similar method in {{PartitionReplicaLifecycleManager}}, which has all prerequisites like {{PartitionReplicaLifecycleManager#weakStopPartition}}, {{PartitionReplicaLifecycleManager#waitForMetadataCompleteness}} > Support DisasterRecoveryManager#restartPartition in Collocation track > - > > Key: IGNITE-24335 > URL: https://issues.apache.org/jira/browse/IGNITE-24335 > Project: Ignite > Issue Type: Improvement >Reporter: Mirza Aliev >Priority: Major > Labels: ignite-3 > > h3. Motivation > This ticket is a part of the implementation track for > https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised > DisasterRecoveryManager API in alignment with Collocation track > In this ticket we have to provide a correct behaviour for > {{DisasterRecoveryManager#restartPartition}} for partitions from zones. > h3. Implementation notes > As usual for the such methods from DisasterRecoveryManager, we must switch to > {{ZonePartitionId}}. Also this mechanism uses under the hood > {{TableManager#restartPartition}}. Seems that we need to have similar method > in {{PartitionReplicaLifecycleManager}}, which has all prerequisites like > {{PartitionReplicaLifecycleManager#weakStopPartition}}, > {{PartitionReplicaLifecycleManager#waitForMetadataCompleteness}} > h3. Definition of Done > * DisasterRecoveryManager API is enriched with {{restartPartition}} method > that works with zones' based partitions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24230) Enrich Disaster Recovery REST API with new methods for the collocation case
[ https://issues.apache.org/jira/browse/IGNITE-24230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mirza Aliev updated IGNITE-24230: - Summary: Enrich Disaster Recovery REST API with new methods for the collocation case (was: Support Disaster Recovery methods for the collocation case) > Enrich Disaster Recovery REST API with new methods for the collocation case > --- > > Key: IGNITE-24230 > URL: https://issues.apache.org/jira/browse/IGNITE-24230 > Project: Ignite > Issue Type: Improvement >Reporter: Mirza Aliev >Priority: Major > Labels: ignite-3 > > h3. Motivation > Once we revise DisasterRecoveryManager API in alignment with Colocation track > https://issues.apache.org/jira/browse/IGNITE-24330 and implement all related > tickets, so DisasterRecoveryManager API is improved with new methods that > work with zones' partitions, > we need to implement REST API for all new methods. See > {{DisasterRecoveryController}} > h3. Definition of done > * REST API of DisasterRecoveryManager is improved with new methods that were > implemented for zones' partitions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-24323) Calcite engine. Query execution inside UDF leads to deadlock
[ https://issues.apache.org/jira/browse/IGNITE-24323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922026#comment-17922026 ] Aleksey Plekhanov commented on IGNITE-24323: Implementation details: Regular {{IgniteThreadPoolExecutor}} is used instead of striped pool executor. But with custom {{BlockingQueue}}. This blocking queue implemented as wrapper over the new data structure {{QueryTasksQueue}}. On {{poll}} from this queue query is blocked by ID and no more task with the same query ID can be polled until query is unblocked explicitely. Query is unblocked in {{finally}} block after the task completes execution. So no tasks can be executed concurrently for the same query and we still provide initial tasks ordering in scope of one query. > Calcite engine. Query execution inside UDF leads to deadlock > > > Key: IGNITE-24323 > URL: https://issues.apache.org/jira/browse/IGNITE-24323 > Project: Ignite > Issue Type: Bug >Reporter: Aleksey Plekhanov >Assignee: Aleksey Plekhanov >Priority: Major > Labels: calcite, ise > Time Spent: 0.5h > Remaining Estimate: 0h > > We have striped executor to execute query tasks. Thread for query/fragment on > each node is fixed and defined by hash of query ID and fragment ID. If user > created user defined function and try to execute some query inside it, it can > lead to deadlock, since task execution thread blocked by new synchronous > query execution request and the new query can be assigned to the same thread. > For H2-based SQL engine threads are no binded to the query and there is no > such problem. > For Calcite-based SQL engine we use non thread safe datastructures in tasks. > To workaround this problem we can use regular IgniteThreadPoolExecutor, but > need to ensure that there are no tasks concurrently executing for the same > query/fragment. > Also, perhaps this approach can affect performance and should be enabled by > some property explicitly only for users that require this mode. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-24307) Query plan does not use the most optimal table index
[ https://issues.apache.org/jira/browse/IGNITE-24307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922027#comment-17922027 ] Iurii Gerzhedovich commented on IGNITE-24307: - [~vdmitrienko] I can just say that no difference for this case in terms of the time of execution. You can try to investigate this and find bounds when it will be significant > Query plan does not use the most optimal table index > > > Key: IGNITE-24307 > URL: https://issues.apache.org/jira/browse/IGNITE-24307 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 3.0.0-beta1 > Environment: 3 nodes (each node is CMG, each node > {color:#067d17}"-Xms4096m"{color}, {color:#067d17}"-Xmx4096m"{color}), each > on separate host. Each host vCPU: 4, Memory: 32GB. >Reporter: Vladimir Dmitrienko >Priority: Major > Labels: ignite-3 > > *Steps to reproduce:* > Execute the following SQL script: > {code:java} > CREATE TABLE IF NOT EXISTS test_table (id int primary key, field_1 int, > field_2 int, field_3 int); > CREATE INDEX IF NOT EXISTS one_field_index ON test_table (field_1 ASC); > CREATE INDEX IF NOT EXISTS three_field_index ON test_table (field_1 ASC, > field_2 ASC, field_3 ASC); > INSERT INTO test_table VALUES (1, 2, 3, 4); > EXPLAIN PLAN FOR SELECT field_1, field_2, field_3 FROM test_table WHERE > field_1 = 2 AND field_2 = 3 AND field_3 = 4;{code} > > *Actual result:* > Execution plan uses the {{ONE_FIELD_INDEX}} even if the more appropriate > {{THREE_FIELD_INDEX}} exists; > > {code:java} > PLAN > > > > > Exchange(distribution=[single]): rowcount = 1.0, cumulative cost = IgniteCost > [rowCount=2.0, cpu=2.0, memory=1.0, io=1.0, network=13.0], id = 362 > > > > IndexScan(table=[[PUBLIC, TEST_TABLE]], tableId=[35], > index=[ONE_FIELD_INDEX], type=[SORTED], searchBounds=[[ExactBounds > [bound=2]]], filters=[AND(=($t0, 2), =($t1, 3), =($t2, 4))], > requiredColumns=[{1, 2, 3}], collation=[[1]]): rowcount = 1.0, cumulative > cost = IgniteCost [rowCount=1.0, cpu=1.0, memory=1.0, io=1.0, network=1.0], > id = 361 {code} > > *Expected result:* > {{THREE_FIELD_INDEX}} used. > *Note:* > if id column added to the query, {{THREE_FIELD_INDEX}} is used. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-22621) Implement transacation storage for the zone replica listener
[ https://issues.apache.org/jira/browse/IGNITE-22621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-22621: - Ignite Flags: (was: Docs Required,Release Notes Required) > Implement transacation storage for the zone replica listener > > > Key: IGNITE-22621 > URL: https://issues.apache.org/jira/browse/IGNITE-22621 > Project: Ignite > Issue Type: Improvement >Reporter: Kirill Gusakov >Priority: Major > Labels: ignite-3 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-23749) CancelServiceTask permissions
[ https://issues.apache.org/jira/browse/IGNITE-23749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikita Amelchev updated IGNITE-23749: - Fix Version/s: 2.18 > CancelServiceTask permissions > - > > Key: IGNITE-23749 > URL: https://issues.apache.org/jira/browse/IGNITE-23749 > Project: Ignite > Issue Type: Improvement >Reporter: Egor Fomin >Assignee: Egor Fomin >Priority: Minor > Fix For: 2.18 > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-23749) CancelServiceTask permissions
[ https://issues.apache.org/jira/browse/IGNITE-23749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikita Amelchev updated IGNITE-23749: - Ignite Flags: Release Notes Required (was: Docs Required,Release Notes Required) > CancelServiceTask permissions > - > > Key: IGNITE-23749 > URL: https://issues.apache.org/jira/browse/IGNITE-23749 > Project: Ignite > Issue Type: Improvement >Reporter: Egor Fomin >Assignee: Egor Fomin >Priority: Minor > Fix For: 2.18 > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-23749) CancelServiceTask permissions
[ https://issues.apache.org/jira/browse/IGNITE-23749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikita Amelchev updated IGNITE-23749: - Release Note: Removed redundant permission check to cancel services > CancelServiceTask permissions > - > > Key: IGNITE-23749 > URL: https://issues.apache.org/jira/browse/IGNITE-23749 > Project: Ignite > Issue Type: Improvement >Reporter: Egor Fomin >Assignee: Egor Fomin >Priority: Minor > Fix For: 2.18 > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-23749) CancelServiceTask permissions
[ https://issues.apache.org/jira/browse/IGNITE-23749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikita Amelchev updated IGNITE-23749: - Description: Remove redundant ADMIN_OPS permission check to cancel services > CancelServiceTask permissions > - > > Key: IGNITE-23749 > URL: https://issues.apache.org/jira/browse/IGNITE-23749 > Project: Ignite > Issue Type: Improvement >Reporter: Egor Fomin >Assignee: Egor Fomin >Priority: Minor > Fix For: 2.18 > > Time Spent: 50m > Remaining Estimate: 0h > > Remove redundant ADMIN_OPS permission check to cancel services -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-24323) Calcite engine. Query execution inside UDF leads to deadlock
[ https://issues.apache.org/jira/browse/IGNITE-24323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922263#comment-17922263 ] Aleksey Plekhanov commented on IGNITE-24323: Green TC bot visa: https://tcbot2.sbt-ignite-dev.ru/pr.html?serverId=apache&suiteId=IgniteTests24Java8_RunAll&branchForTc=pull/11833/head&action=Latest Something broken in TC bot and "Comment JIRA" is not working now. > Calcite engine. Query execution inside UDF leads to deadlock > > > Key: IGNITE-24323 > URL: https://issues.apache.org/jira/browse/IGNITE-24323 > Project: Ignite > Issue Type: Bug >Reporter: Aleksey Plekhanov >Assignee: Aleksey Plekhanov >Priority: Major > Labels: calcite, ise > Time Spent: 0.5h > Remaining Estimate: 0h > > We have striped executor to execute query tasks. Thread for query/fragment on > each node is fixed and defined by hash of query ID and fragment ID. If user > created user defined function and try to execute some query inside it, it can > lead to deadlock, since task execution thread blocked by new synchronous > query execution request and the new query can be assigned to the same thread. > For H2-based SQL engine threads are no binded to the query and there is no > such problem. > For Calcite-based SQL engine we use non thread safe datastructures in tasks. > To workaround this problem we can use regular IgniteThreadPoolExecutor, but > need to ensure that there are no tasks concurrently executing for the same > query/fragment. > Also, perhaps this approach can affect performance and should be enabled by > some property explicitly only for users that require this mode. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-24139) Update Ignite dependency jetty
[ https://issues.apache.org/jira/browse/IGNITE-24139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922270#comment-17922270 ] Julia Bakulina commented on IGNITE-24139: - [TCbot2 green visa|https://tcbot2.sbt-ignite-dev.ru/pr.html?serverId=apache&suiteId=IgniteTests24Java8_RunAll&branchForTc=pull/11830/head&action=Latest] > Update Ignite dependency jetty > -- > > Key: IGNITE-24139 > URL: https://issues.apache.org/jira/browse/IGNITE-24139 > Project: Ignite > Issue Type: Task >Reporter: Julia Bakulina >Assignee: Julia Bakulina >Priority: Minor > Labels: ise > Time Spent: 10m > Remaining Estimate: 0h > > Update Ignite jetty dependency to 11 version -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24344) Rename types related to transaction state storages
[ https://issues.apache.org/jira/browse/IGNITE-24344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Puchkovskiy updated IGNITE-24344: --- Description: Currently, there is per-table TxStateTableStorage and per-partition (where partitions are table partitions) TxStateStorage. We are going to switch to zones, so 'table' term should go. TxStateTableStorage is not specific to tables, it can serve as zone-wide storage, but we need to rename it. The idea is that the new name is not specific about tables or zones; TxStateStorage seems to be ok in this regard. This means that TxStateStorage (that now corresponds to one partition) needs to be renamed, TxStatePartitionStorage seems to be a good name. It also aligns well with MvPartitionStorage which we already have. {*}Definition of done{*}: TxStateStorage gets renamed to TxStatePartitionStorage, and TxStateTableStorage to TxStateStorage. was:Currently, there is per-table TxState > Rename types related to transaction state storages > -- > > Key: IGNITE-24344 > URL: https://issues.apache.org/jira/browse/IGNITE-24344 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Assignee: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > > Currently, there is per-table TxStateTableStorage and per-partition (where > partitions are table partitions) TxStateStorage. > We are going to switch to zones, so 'table' term should go. > TxStateTableStorage is not specific to tables, it can serve as zone-wide > storage, but we need to rename it. The idea is that the new name is not > specific about tables or zones; TxStateStorage seems to be ok in this regard. > This means that TxStateStorage (that now corresponds to one partition) needs > to be renamed, TxStatePartitionStorage seems to be a good name. It also > aligns well with MvPartitionStorage which we already have. > {*}Definition of done{*}: TxStateStorage gets renamed to > TxStatePartitionStorage, and TxStateTableStorage to TxStateStorage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24344) Rename types related to transaction state storages
[ https://issues.apache.org/jira/browse/IGNITE-24344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Puchkovskiy updated IGNITE-24344: --- Epic Link: IGNITE-22115 > Rename types related to transaction state storages > -- > > Key: IGNITE-24344 > URL: https://issues.apache.org/jira/browse/IGNITE-24344 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Assignee: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > > Currently, there is per-table TxStateTableStorage and per-partition (where > partitions are table partitions) TxStateStorage. > We are going to switch to zones, so 'table' term should go. > TxStateTableStorage is not specific to tables, it can serve as zone-wide > storage, but we need to rename it. The idea is that the new name is not > specific about tables or zones; TxStateStorage seems to be ok in this regard. > This means that TxStateStorage (that now corresponds to one partition) needs > to be renamed, TxStatePartitionStorage seems to be a good name. It also > aligns well with MvPartitionStorage which we already have. > {*}Definition of done{*}: TxStateStorage gets renamed to > TxStatePartitionStorage, and TxStateTableStorage to TxStateStorage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-24345) Defer zone destruction after drop
Roman Puchkovskiy created IGNITE-24345: -- Summary: Defer zone destruction after drop Key: IGNITE-24345 URL: https://issues.apache.org/jira/browse/IGNITE-24345 Project: Ignite Issue Type: Improvement Reporter: Roman Puchkovskiy -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-24346) Do not destroy tx state storage while its content might be needed for write intent resolution
Roman Puchkovskiy created IGNITE-24346: -- Summary: Do not destroy tx state storage while its content might be needed for write intent resolution Key: IGNITE-24346 URL: https://issues.apache.org/jira/browse/IGNITE-24346 Project: Ignite Issue Type: Improvement Reporter: Roman Puchkovskiy -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24346) Do not destroy tx state storage while its content might be needed for write intent resolution
[ https://issues.apache.org/jira/browse/IGNITE-24346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Puchkovskiy updated IGNITE-24346: --- Description: Tx finish records stored in tx state storage have to be retained until all write intents corresponding to the finish record's transaction are cleaned up (either converted to normal tuple versions or removed). It is possible that, in the current implementation, we might erase such records too early when destroying a dropped table. The suspected scenario is: # tx1 is started, it writes to tables A and B, the it gets committed, commit partition is a partition of A # Cleanup of tx1 is deferred for some reason (like high amount of work the cleanuper does) # A is dropped # After the moment of A's drop sinks under LWM, storages of A's partitions get destroyed (including the commit partition of tx1) # Cleanup of tx1 is attempted, but the tx state storage is destroyed, so the cleanup cannot proceed -> write intents of tx1 will remain forever unresolved We need to check whether this is a possible scenario and, if it's possible, make sure that we keep tx state storages available for cleanup activities until all their transactions get cleaned up, even if this means deferring table partition destruction. For per-zone tx state storages to which we are currently switching (see IGNITE-22621), nothing changes: a transaction might span multiple zones and then the zone hosting the commit partition might have been dropped and destroyed (IGNITE-24345). > Do not destroy tx state storage while its content might be needed for write > intent resolution > - > > Key: IGNITE-24346 > URL: https://issues.apache.org/jira/browse/IGNITE-24346 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > > Tx finish records stored in tx state storage have to be retained until all > write intents corresponding to the finish record's transaction are cleaned up > (either converted to normal tuple versions or removed). It is possible that, > in the current implementation, we might erase such records too early when > destroying a dropped table. > The suspected scenario is: > # tx1 is started, it writes to tables A and B, the it gets committed, commit > partition is a partition of A > # Cleanup of tx1 is deferred for some reason (like high amount of work the > cleanuper does) > # A is dropped > # After the moment of A's drop sinks under LWM, storages of A's partitions > get destroyed (including the commit partition of tx1) > # Cleanup of tx1 is attempted, but the tx state storage is destroyed, so the > cleanup cannot proceed -> write intents of tx1 will remain forever unresolved > We need to check whether this is a possible scenario and, if it's possible, > make sure that we keep tx state storages available for cleanup activities > until all their transactions get cleaned up, even if this means deferring > table partition destruction. > For per-zone tx state storages to which we are currently switching (see > IGNITE-22621), nothing changes: a transaction might span multiple zones and > then the zone hosting the commit partition might have been dropped and > destroyed (IGNITE-24345). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24345) Defer zone destruction after drop
[ https://issues.apache.org/jira/browse/IGNITE-24345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Puchkovskiy updated IGNITE-24345: --- Description: When a table is dropped, table partition storages are not destroyed immediately. Instead, this is deferred until table drop moment sinks below the LWM. As partitions will become per-zone (instead of per-table), we need to implement the same logic for zones. That is, zone partitions need to be destroyed only after zone drop moment sinks below the LWM. > Defer zone destruction after drop > - > > Key: IGNITE-24345 > URL: https://issues.apache.org/jira/browse/IGNITE-24345 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > > When a table is dropped, table partition storages are not destroyed > immediately. Instead, this is deferred until table drop moment sinks below > the LWM. > As partitions will become per-zone (instead of per-table), we need to > implement the same logic for zones. That is, zone partitions need to be > destroyed only after zone drop moment sinks below the LWM. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24344) Rename types related to transaction meta storages
[ https://issues.apache.org/jira/browse/IGNITE-24344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Puchkovskiy updated IGNITE-24344: --- Description: Currently, there is per-table TxState > Rename types related to transaction meta storages > - > > Key: IGNITE-24344 > URL: https://issues.apache.org/jira/browse/IGNITE-24344 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Assignee: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > > Currently, there is per-table TxState -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-24344) Rename types related to transaction meta storages
Roman Puchkovskiy created IGNITE-24344: -- Summary: Rename types related to transaction meta storages Key: IGNITE-24344 URL: https://issues.apache.org/jira/browse/IGNITE-24344 Project: Ignite Issue Type: Improvement Reporter: Roman Puchkovskiy Assignee: Roman Puchkovskiy -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24344) Rename types related to transaction state storages
[ https://issues.apache.org/jira/browse/IGNITE-24344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Puchkovskiy updated IGNITE-24344: --- Summary: Rename types related to transaction state storages (was: Rename types related to transaction meta storages) > Rename types related to transaction state storages > -- > > Key: IGNITE-24344 > URL: https://issues.apache.org/jira/browse/IGNITE-24344 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Assignee: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > > Currently, there is per-table TxState -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-24347) Table drop should not cause zone partitions destruction
Roman Puchkovskiy created IGNITE-24347: -- Summary: Table drop should not cause zone partitions destruction Key: IGNITE-24347 URL: https://issues.apache.org/jira/browse/IGNITE-24347 Project: Ignite Issue Type: Improvement Reporter: Roman Puchkovskiy -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24345) Defer zone destruction after drop
[ https://issues.apache.org/jira/browse/IGNITE-24345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Puchkovskiy updated IGNITE-24345: --- Description: When a table is dropped, table partition storages are not destroyed immediately. Instead, this is deferred until table drop moment sinks below the LWM. As partitions will become per-zone (instead of per-table), we need to implement the same logic for zones. That is, zone partitions need to be destroyed only after zone drop moment sinks below the LWM. IGNITE-24345 defines that table drop should stop causing eventual storage destruction. was: When a table is dropped, table partition storages are not destroyed immediately. Instead, this is deferred until table drop moment sinks below the LWM. As partitions will become per-zone (instead of per-table), we need to implement the same logic for zones. That is, zone partitions need to be destroyed only after zone drop moment sinks below the LWM. > Defer zone destruction after drop > - > > Key: IGNITE-24345 > URL: https://issues.apache.org/jira/browse/IGNITE-24345 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > > When a table is dropped, table partition storages are not destroyed > immediately. Instead, this is deferred until table drop moment sinks below > the LWM. > As partitions will become per-zone (instead of per-table), we need to > implement the same logic for zones. That is, zone partitions need to be > destroyed only after zone drop moment sinks below the LWM. > IGNITE-24345 defines that table drop should stop causing eventual storage > destruction. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (IGNITE-24347) Table drop should not cause zone partitions destruction
[ https://issues.apache.org/jira/browse/IGNITE-24347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Puchkovskiy resolved IGNITE-24347. Resolution: Invalid The issue is invalid: as MV partition storages will remain per-table for now, table drop still should cause an eventual destruction of such storages belonging to the dropped table. > Table drop should not cause zone partitions destruction > --- > > Key: IGNITE-24347 > URL: https://issues.apache.org/jira/browse/IGNITE-24347 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > > As in IGNITE-24345 partition destruction will be moved to zone-related > events, table drop should not cause partition destruction anymore (even in > its deferred variety). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24347) Table drop should not cause zone partitions destruction
[ https://issues.apache.org/jira/browse/IGNITE-24347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Puchkovskiy updated IGNITE-24347: --- Description: As in IGNITE-24345 partition destruction will be moved to zone-related events, table drop should not cause partition destruction anymore (even in its deferred variety). > Table drop should not cause zone partitions destruction > --- > > Key: IGNITE-24347 > URL: https://issues.apache.org/jira/browse/IGNITE-24347 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > > As in IGNITE-24345 partition destruction will be moved to zone-related > events, table drop should not cause partition destruction anymore (even in > its deferred variety). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24345) Defer zone destruction after drop
[ https://issues.apache.org/jira/browse/IGNITE-24345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Puchkovskiy updated IGNITE-24345: --- Description: When a table is dropped, table partition storages are not destroyed immediately. Instead, this is deferred until table drop moment sinks below the LWM. As partitions will become per-zone (instead of per-table), we need to implement the same logic for zones. That is, zone partitions need to be destroyed only after zone drop moment sinks below the LWM. MV partition storages will remain per-table (at least in this Epic), so they will still need to be destroyed eventually as a result of table drop. was: When a table is dropped, table partition storages are not destroyed immediately. Instead, this is deferred until table drop moment sinks below the LWM. As partitions will become per-zone (instead of per-table), we need to implement the same logic for zones. That is, zone partitions need to be destroyed only after zone drop moment sinks below the LWM. IGNITE-24345 defines that table drop should stop causing eventual storage destruction. > Defer zone destruction after drop > - > > Key: IGNITE-24345 > URL: https://issues.apache.org/jira/browse/IGNITE-24345 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > > When a table is dropped, table partition storages are not destroyed > immediately. Instead, this is deferred until table drop moment sinks below > the LWM. > As partitions will become per-zone (instead of per-table), we need to > implement the same logic for zones. That is, zone partitions need to be > destroyed only after zone drop moment sinks below the LWM. > MV partition storages will remain per-table (at least in this Epic), so they > will still need to be destroyed eventually as a result of table drop. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24336) Calcite. Add support of user-defined table functions.
[ https://issues.apache.org/jira/browse/IGNITE-24336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Steshin updated IGNITE-24336: -- Release Note: User-defined SQL table functions are now available with Calcite engine. Description: We should implement user-defined table functions. They already have an implementation in _Calcite_, _TableFunctionImpl_. And we also have user-defined scalar functions. Table functions should be described similarly to the existing configuration of scalar functions, via _CacheConfiguration#setSqlFunctionClasses()_ and annotation _@QuerySqlTableFunction_. Or we might use another annotation. The annotation should describe the returned columns and their data types. *Example:* {code:java} public static final class TableFunctions { @QuerySqlFunction(tableColumnTypes = {Integer.class, String.class}, tableColumnNames = {"INT_COL", "STR_COL"}) public static Iterable table_fun(int i) { return Arrays.asList( new Object[]{i, "code_" + i}, new Object[]{i * 10, "code_" + (i * 10)} ); } } ignite.getOrCreateCache(new CacheConfiguration("test_cache").setSqlFunctionClasses(TableFunctions.class)); sql("select STR_COL from TABLE_FUN(1) where INT_COL>1") {code} Table function implementation must return an _Iterable_ representing table rows. Each its element is an _Object[]_ or an _Iterable_ representing a table row. The number of row elements must match the number of the defined columns. *Known limitations:* Only _Calcite_ engine will support table functions first. was: We should implement user-defined table functions. They already have a implementation in _Calcite_, _TableFunctionImpl_. And we also have user-defined scalar functions. Table functions should be described similarly to the existing configuration of scalar functions, via _CacheConfiguration#setSqlFunctionClasses()_ and annotation _@QuerySqlTableFunction_. Or we might use another annotation. The annotation should describe the returned columns and their data types. *Example:* {code:java} public static final class TableFunctions { @QuerySqlFunction(tableColumnTypes = {Integer.class, String.class}, tableColumnNames = {"INT_COL", "STR_COL"}) public static Iterable table_fun(int i) { return Arrays.asList( new Object[]{i, "code_" + i}, new Object[]{i * 10, "code_" + (i * 10)} ); } } ignite.getOrCreateCache(new CacheConfiguration("test_cache").setSqlFunctionClasses(TableFunctions.class)); sql("select STR_COL from TABLE_FUN(1) where INT_COL>1") {code} Table function implementation must return an _Iterable_ representing table rows. Each its element is an _Object[]_ or an _Iterable_ representing a table row. The number of row elements must match the number of the defined columns. *Known limitations:* Only _Calcite_ engine will support table functions first. > Calcite. Add support of user-defined table functions. > - > > Key: IGNITE-24336 > URL: https://issues.apache.org/jira/browse/IGNITE-24336 > Project: Ignite > Issue Type: Improvement >Reporter: Vladimir Steshin >Assignee: Vladimir Steshin >Priority: Major > Labels: calcite, ignite-2, ise, sql > > We should implement user-defined table functions. They already have an > implementation in _Calcite_, _TableFunctionImpl_. And we also have > user-defined scalar functions. > Table functions should be described similarly to the existing configuration > of scalar functions, via _CacheConfiguration#setSqlFunctionClasses()_ and > annotation _@QuerySqlTableFunction_. Or we might use another annotation. The > annotation should describe the returned columns and their data types. > *Example:* > {code:java} > public static final class TableFunctions { > @QuerySqlFunction(tableColumnTypes = {Integer.class, String.class}, > tableColumnNames = {"INT_COL", "STR_COL"}) > public static Iterable table_fun(int i) { > return Arrays.asList( > new Object[]{i, "code_" + i}, > new Object[]{i * 10, "code_" + (i * 10)} > ); > } > } > ignite.getOrCreateCache(new > CacheConfiguration("test_cache").setSqlFunctionClasses(TableFunctions.class)); > sql("select STR_COL from TABLE_FUN(1) where INT_COL>1") > {code} > Table function implementation must return an _Iterable_ representing table > rows. Each its element is an _Object[]_ or an _Iterable_ representing a table > row. The number of row elements must match the number of the defined columns. > *Known limitations:* > Only _Calcite_ engine will support table functions first. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24336) Calcite. Add support of user-defined table functions.
[ https://issues.apache.org/jira/browse/IGNITE-24336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Steshin updated IGNITE-24336: -- Labels: calcite ignite-2 ise (was: calcite ignite-2 ise sql) > Calcite. Add support of user-defined table functions. > - > > Key: IGNITE-24336 > URL: https://issues.apache.org/jira/browse/IGNITE-24336 > Project: Ignite > Issue Type: Improvement >Reporter: Vladimir Steshin >Assignee: Vladimir Steshin >Priority: Major > Labels: calcite, ignite-2, ise > > We should implement user-defined table functions. They already have an > implementation in _Calcite_, _TableFunctionImpl_. And we also have > user-defined scalar functions. > Table functions should be described similarly to the existing configuration > of scalar functions, via _CacheConfiguration#setSqlFunctionClasses()_ and > annotation _@QuerySqlTableFunction_. Or we might use another annotation. The > annotation should describe the returned columns and their data types. > *Example:* > {code:java} > public static final class TableFunctions { > @QuerySqlFunction(tableColumnTypes = {Integer.class, String.class}, > tableColumnNames = {"INT_COL", "STR_COL"}) > public static Iterable table_fun(int i) { > return Arrays.asList( > new Object[]{i, "code_" + i}, > new Object[]{i * 10, "code_" + (i * 10)} > ); > } > } > ignite.getOrCreateCache(new > CacheConfiguration("test_cache").setSqlFunctionClasses(TableFunctions.class)); > sql("select STR_COL from TABLE_FUN(1) where INT_COL>1") > {code} > Table function implementation must return an _Iterable_ representing table > rows. Each its element is an _Object[]_ or an _Iterable_ representing a table > row. The number of row elements must match the number of the defined columns. > *Known limitations:* > Only _Calcite_ engine will support table functions first. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-22621) Implement transacation storage for the zone replica listener
[ https://issues.apache.org/jira/browse/IGNITE-22621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-22621: - Attachment: Screenshot 2025-01-29 at 10.29.28 AM.png > Implement transacation storage for the zone replica listener > > > Key: IGNITE-22621 > URL: https://issues.apache.org/jira/browse/IGNITE-22621 > Project: Ignite > Issue Type: Improvement >Reporter: Kirill Gusakov >Priority: Major > Labels: ignite-3 > Attachments: Screenshot 2025-01-29 at 10.29.28 AM.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-22621) Implement transacation storage for the zone replica listener
[ https://issues.apache.org/jira/browse/IGNITE-22621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin reassigned IGNITE-22621: Assignee: Roman Puchkovskiy > Implement transacation storage for the zone replica listener > > > Key: IGNITE-22621 > URL: https://issues.apache.org/jira/browse/IGNITE-22621 > Project: Ignite > Issue Type: Improvement >Reporter: Kirill Gusakov >Assignee: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > Attachments: Screenshot 2025-01-29 at 10.29.28 AM.png > > > h3. Motivation > Few words about general context. Tx Finish flow is a crucial step in AI3 Tx > Protocol. Generally it consists of two phases along with some prerequisites. > 0. Prerequisites: schema validation, tx state check. > 1. Txn state switch phase. Atomic transaction commit/rollback implemented > through replicating persistent txn state(COMMITTED/ABORTED) to > commitPartition. Once it's done, transaction is considered as finished. > 2. Cleanup phase. Releasing locks, converting writeIntents to common values > or removing them depending on whether it's commit or abort. > h3. !Screenshot 2025-01-29 at 10.29.28 AM.png! Definition of Done > * Move processing TxFinishReplicaRequest from > PartitionReplicaListener#processTxFinishAction to > ZonePartitionRaftListener#processTxFinishAction by introducing corresponding > method. In order to do that, it's required to add some dependencies to > ZonePartitionRaftListener: > * > ** SchemaCompatibilityValidator > ** TxStateStorage // Adding the storage isn't trivial. > ** TxManager > ** RaftCommandRunner // In order to run TxFinishCommand -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-22621) Implement transacation storage for the zone replica listener
[ https://issues.apache.org/jira/browse/IGNITE-22621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-22621: - Description: h3. Motivation Few words about general context. Tx Finish flow is a crucial step in AI3 Tx Protocol. Generally it consists of two phases along with some prerequisites. 0. Prerequisites: schema validation, tx state check. 1. Txn state switch phase. Atomic transaction commit/rollback implemented through replicating persistent txn state(COMMITTED/ABORTED) to commitPartition. Once it's done, transaction is considered as finished. 2. Cleanup phase. Releasing locks, converting writeIntents to common values or removing them depending on whether it's commit or abort. h3. !Screenshot 2025-01-29 at 10.29.28 AM.png! Definition of Done * Move processing TxFinishReplicaRequest from PartitionReplicaListener#processTxFinishAction to ZonePartitionRaftListener#processTxFinishAction by introducing corresponding method. In order to do that, it's required to add some dependencies to ZonePartitionRaftListener: * ** SchemaCompatibilityValidator ** TxStateStorage // Adding the storage isn't trivial. ** TxManager ** RaftCommandRunner // In order to run TxFinishCommand > Implement transacation storage for the zone replica listener > > > Key: IGNITE-22621 > URL: https://issues.apache.org/jira/browse/IGNITE-22621 > Project: Ignite > Issue Type: Improvement >Reporter: Kirill Gusakov >Priority: Major > Labels: ignite-3 > Attachments: Screenshot 2025-01-29 at 10.29.28 AM.png > > > h3. Motivation > Few words about general context. Tx Finish flow is a crucial step in AI3 Tx > Protocol. Generally it consists of two phases along with some prerequisites. > 0. Prerequisites: schema validation, tx state check. > 1. Txn state switch phase. Atomic transaction commit/rollback implemented > through replicating persistent txn state(COMMITTED/ABORTED) to > commitPartition. Once it's done, transaction is considered as finished. > 2. Cleanup phase. Releasing locks, converting writeIntents to common values > or removing them depending on whether it's commit or abort. > h3. !Screenshot 2025-01-29 at 10.29.28 AM.png! Definition of Done > * Move processing TxFinishReplicaRequest from > PartitionReplicaListener#processTxFinishAction to > ZonePartitionRaftListener#processTxFinishAction by introducing corresponding > method. In order to do that, it's required to add some dependencies to > ZonePartitionRaftListener: > * > ** SchemaCompatibilityValidator > ** TxStateStorage // Adding the storage isn't trivial. > ** TxManager > ** RaftCommandRunner // In order to run TxFinishCommand -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24339) Flaky test DdlCommandHandlerExceptionHandlingTest#testZoneNotFoundOnDrop2
[ https://issues.apache.org/jira/browse/IGNITE-24339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Iurii Gerzhedovich updated IGNITE-24339: Description: The test org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandlerExceptionHandlingTest#testZoneNotFoundOnDrop2 is flaky with the following stacktrace: {code:java} java.lang.AssertionError: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Time is out of bounds: 0 at org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:78) at org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:35) at org.hamcrest.TypeSafeMatcher.matches(TypeSafeMatcher.java:83) at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:10) at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6) at org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandlerExceptionHandlingTest.testZoneNotFoundOnDrop2(DdlCommandHandlerExceptionHandlingTest.java:111) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Time is out of bounds: 0 at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2028) at org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:74) ... 8 more Caused by: java.lang.IllegalArgumentException: Time is out of bounds: 0 at org.apache.ignite.internal.hlc.HybridTimestamp.(HybridTimestamp.java:111) at org.apache.ignite.internal.hlc.HybridTimestamp.hybridTimestamp(HybridTimestamp.java:131) at org.apache.ignite.internal.catalog.commands.CatalogUtils.clusterWideEnsuredActivationTimestamp(CatalogUtils.java:537) at org.apache.ignite.internal.catalog.CatalogManagerImpl.calcClusterWideEnsureActivationTime(CatalogManagerImpl.java:376) at org.apache.ignite.internal.catalog.CatalogManagerImpl.awaitVersionActivation(CatalogManagerImpl.java:370) at org.apache.ignite.internal.catalog.CatalogManagerImpl.awaitVersionActivation(CatalogManagerImpl.java:364) at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106) at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2241) at org.apache.ignite.internal.catalog.CatalogManagerImpl.saveUpdateAndWaitForActivation(CatalogManagerImpl.java:314) at org.apache.ignite.internal.catalog.CatalogManagerImpl.execute(CatalogManagerImpl.java:235) at org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandler.handle(DdlCommandHandler.java:76) ... 4 more{code} Initial version of Catalog uses {{0L}} as activation time, which is not valid value according to HybridTimestamp's constructor: Let's change activationTimestamp for initial version of catalog in CatalogManagerImpl from 0 to 1. But we need to rework tests after that was: The test org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandlerExceptionHandlingTest#testZoneNotFoundOnDrop2 is flaky with the following stacktrace: {code:java} java.lang.AssertionError: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Time is out of bounds: 0 at org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:78) at org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:35) at org.hamcrest.TypeSafeMatcher.matches(TypeSafeMatcher.java:83) at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:10) at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6) at org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandlerExceptionHandlingTest.testZoneNotFoundOnDrop2(DdlCommandHandlerExceptionHandlingTest.java:111) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Time is out of bounds: 0 at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2028) at org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:74) ... 8 more Caused by: java.lang.IllegalArgumentException: Time is out of bounds: 0 at org.apache.ignite.internal.hlc.HybridTimestamp.(HybridTimestamp.java:111) at org.apa
[jira] [Updated] (IGNITE-24337) Don't throw NoSuchFileException in IgniteUtils.deleteIfExistsThrowable
[ https://issues.apache.org/jira/browse/IGNITE-24337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philipp Shergalis updated IGNITE-24337: --- Description: It is definitely not correct that deleteIfExists method throws error if the directory doesn't exist. Checking if file exists before calling this method is race-prone (it could get deleted by another thread after this check) > Don't throw NoSuchFileException in IgniteUtils.deleteIfExistsThrowable > -- > > Key: IGNITE-24337 > URL: https://issues.apache.org/jira/browse/IGNITE-24337 > Project: Ignite > Issue Type: Improvement >Reporter: Philipp Shergalis >Assignee: Philipp Shergalis >Priority: Major > Labels: ignite-3 > > It is definitely not correct that deleteIfExists method throws error if the > directory doesn't exist. Checking if file exists before calling this method > is race-prone (it could get deleted by another thread after this check) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-24340) Queries permanently causing lock table overflow get retried forever
Roman Puchkovskiy created IGNITE-24340: -- Summary: Queries permanently causing lock table overflow get retried forever Key: IGNITE-24340 URL: https://issues.apache.org/jira/browse/IGNITE-24340 Project: Ignite Issue Type: Bug Reporter: Roman Puchkovskiy -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-24343) Use ZonePartitionId as commitPartitionId
[ https://issues.apache.org/jira/browse/IGNITE-24343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-24343: - Description: h3. Motivation Side note: we are not consistent in what commitPartitionId means: usually we assume that it's tableId + partitionIndex however in storage API, e.g. in PartitionDataStorage#addWrite commitPartitionId itself means partition index along with dedicated commitTableId. Below I will assume that commitPartitionId is a combination of table/zoneId and partitionIndex. * CommitPartition is a partition where where persisted txnState is stored. The state is atomically switched within TxFinish transaction phase. * CommitPartitionId is used in order to evaluate writeIntents through commitPartition path. * Write Intent(WI) resolution is a process that allows to define whether WI should be considered as applied (either committed or aborted) or ignored because corresponding transaction is in PENDING state. Simply, for WI(txId) ** Tx(txId).isPending() -> ignore WI (return previous value, or null if there's no such value). ** Tx(txId).isAborted() -> ignore WI (return previous value, or null if there's no such value). ** Tx(txId).isCommited() -> consider WI as a common value. * There are 3 different pathes of WI Resolution: ** Local path. Every node that participates in tx flow has volatile txnStateMap, thus it worth checking locally whether we already know the state of corresponding transaction. ** If local path is not applicable we will use Coordinator Path by requesting txn state from cooresponding coordinator. ** If Coordinator Path also not applicable we will use Commit Partition Path. Hence it's required to know where commit partition is. ** For more details please check [IEP-91|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211885498#IEP91:Transactionprotocol-ROtransactions.1] Worth mentioning that IEP is a bit outdated and both RO and RW transaction may see WI and thus do WI Resolution. Seems that we are done with [taxidermy|https://en.wikipedia.org/wiki/Taxonomy]. h3. Definition of Done * Within collocation track we do store zoneId + partitionIndex as commitPartitionId instead of tablerId + partitionIndex. * WI resolution works on top of ZonePartitionId as commitPartitionId. h3. Implementation Notes * As an entry point we may start with PartitionDataStorage#addWrite. Basically depending on whether it's collocation track or common one it's possible to propagate either zoneId or tableId as int without updating internal storage structure. * That means that UpdateCommand.tablePartitionId should be adjusted or extended with zonePartitionId, that required to adjust ReadWriteReplicaRequest.commitPartitionId in a similar manner which on its turn requires to adjust InternalTransaction.commitPartition. Long story short, we will find ourselves in methods like org.apache.ignite.internal.sql.engine.prepare.IgniteRelShuttle#enlist(org.apache.ignite.internal.sql.engine.rel.SourceAwareIgniteRel) where we retrieve tableId from relation int tableId = rel.getTable().unwrap(IgniteTable.class).id(); Seems that we have two options here, either propagate zoneId from the very beginning of the stack or add Catalog dependency to ExecutionService and similar components in order to convert tableId to zoneId in idempotent manner. I mean that in theory it's possible to move tables between zones, thus it's required to be consistent in defining zoneId by tableId, e.g. by requesting catalog on tx start time. * On the other hand ReadResult has int commitTableId that should be renamed to commitZoneId or some generalized option. * Corresponding changes in WI flow should also be implemented. was: h3. Motivation Side note: we are not consistent in what commitPartitionId means: usually we assume that it's tableId + partitionIndex however in storage API, e.g. in PartitionDataStorage#addWrite commitPartitionId itself means partition index along with dedicated commitTableId. Below I will assume that commitPartitionId is a combination of table/zoneId and partitionIndex. * CommitPartition is a partition where where persisted txnState is stored. The state is atomically switched within TxFinish transaction phase. * CommitPartitionId is used in order to evaluate writeIntents through commitPartition path. * Write Intent(WI) resolution is a process that allows to define whether WI should be considered as applied (either committed or aborted) or ignored because corresponding transaction is in PENDING state. Simply, for WI(txId) ** Tx(txId).isPending() -> ignore WI (return previous value, or null if there's no such value). ** Tx(txId).isAborted() -> ignore WI (return previous value, or null if there's no such value). ** Tx(txId).isCommited() -> consider WI as a common value. * There are 3 different pathes of WI Resolution: ** Local pa