[jira] [Updated] (IGNITE-24343) Use ZonePartitionId as commitPartitionId

2025-01-29 Thread Alexander Lapin (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Lapin updated IGNITE-24343:
-
Ignite Flags:   (was: Docs Required,Release Notes Required)

> Use ZonePartitionId as commitPartitionId
> 
>
> Key: IGNITE-24343
> URL: https://issues.apache.org/jira/browse/IGNITE-24343
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Alexander Lapin
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (IGNITE-24343) Use ZonePartitionId as commitPartitionId

2025-01-29 Thread Alexander Lapin (Jira)
Alexander Lapin created IGNITE-24343:


 Summary: Use ZonePartitionId as commitPartitionId
 Key: IGNITE-24343
 URL: https://issues.apache.org/jira/browse/IGNITE-24343
 Project: Ignite
  Issue Type: Improvement
Reporter: Alexander Lapin






--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24343) Use ZonePartitionId as commitPartitionId

2025-01-29 Thread Alexander Lapin (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Lapin updated IGNITE-24343:
-
Labels: ignite-3  (was: )

> Use ZonePartitionId as commitPartitionId
> 
>
> Key: IGNITE-24343
> URL: https://issues.apache.org/jira/browse/IGNITE-24343
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Alexander Lapin
>Priority: Major
>  Labels: ignite-3
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24343) Use ZonePartitionId as commitPartitionId

2025-01-29 Thread Alexander Lapin (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Lapin updated IGNITE-24343:
-
Description: 
h3. Motivation

Side note: we are not consistent in what commitPartitionId means: usually we 
assume that it's tableId + partitionIndex however in storage API, e.g. in 
PartitionDataStorage#addWrite commitPartitionId itself means partition index 
along with dedicated 
commitTableId. Below I will assume that commitPartitionId is a combination of 
table/zoneId and partitionIndex.
  * CommitPartition is a partition where where persisted txnState is stored. 
The state is atomically switched within TxFinish transaction phase.
 * CommitPartitionId is used in order to evaluate writeIntents through 
commitPartition path.
 * Write Intent(WI) resolution is a process that allows to define whether WI 
should be considered as applied (either committed or aborted) or ignored 
because corresponding transaction is in PENDING state. Simply, for WI(txId)
 ** Tx(txId).isPending() -> ignore WI (return previous value, or null if 
there's no such value).
 ** Tx(txId).isAborted() -> ignore WI (return previous value, or null if 
there's no such value).
 ** Tx(txId).isCommited() -> consider WI as a common value.
 * There are 3 different pathes of WI Resolution:
 ** Local path. Every node that participates in tx flow has volatile 
txnStateMap, thus it worth checking locally whether we already know the state 
of corresponding transaction.
 ** If local path is not applicable we will use Coordinator Path by requesting 
txn state from cooresponding coordinator.
 ** If Coordinator Path also not applicable we will use Commit Partition Path. 
Hence it's required to know where commit partition is.
 ** For more details please check 
[IEP-91|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211885498#IEP91:Transactionprotocol-ROtransactions.1]
 Worth mentioning that IEP is a bit outdated and both RO and RW transaction may 
see WI and thus do WI Resolution.

 
Seems that we are done with [taxidermy|https://en.wikipedia.org/wiki/Taxonomy].
h3. Definition of Done
 * Within collocation track we do store zoneId + partitionIndex as 
commitPartitionId instead of tablerId + partitionIndex.
 * WI resolution works on top of ZonePartitionId as commitPartitionId.

  was:
h3. Motivation

Side note: we are not consistent in what commitPartitionId means: usually we 
assume that it's tableId + partitionIndex however in storage API, e.g. in 
PartitionDataStorage#addWrite commitPartitionId itself means partition index 
along with dedicated 
commitTableId. Below I will assume that commitPartitionId is a combination of 
table/zoneId and partitionIndex.
  * CommitPartition is a partition where where persisted txnState is stored. 
The state is atomically switched within TxFinish transaction phase.
 * CommitPartitionId is used in order to evaluate writeIntents through 
commitPartition path.
 * Write Intent(WI) resolution is a process that allows to define whether WI 
should be considered as applied (either committed or aborted) or ignored 
because corresponding transaction is in PENDING state. Simply, for WI(txId)
 ** Tx(txId).isPending() -> ignore WI (return previous value, or null if 
there's no such value).
 ** Tx(txId).isAborted() -> ignore WI (return previous value, or null if 
there's no such value).
 ** Tx(txId).isCommited() -> consider WI as a common value.
 * There are 3 different pathes of WI Resolution:
 ** Local path. Every node that participates in tx flow has volatile 
txnStateMap, thus it worth checking locally whether we already know the state 
of corresponding transaction.
 ** If local path is not applicable we will use Coordinator Path by requesting 
txn state from cooresponding coordinator.
 ** If Coordinator Path also not applicable we will use Commit Partition Path. 
Hence it's required to know where commit partition is.
 ** For more details please check 
[IEP-91|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211885498#IEP91:Transactionprotocol-ROtransactions.1]
 Worth mentioning that IEP is a bit outdated and both RO and RW transaction may 
see WI and thus do WI Resolution.

 
Seems that we are done with [taxidermy|https://en.wikipedia.org/wiki/Taxonomy].


> Use ZonePartitionId as commitPartitionId
> 
>
> Key: IGNITE-24343
> URL: https://issues.apache.org/jira/browse/IGNITE-24343
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Alexander Lapin
>Priority: Major
>  Labels: ignite-3
>
> h3. Motivation
> Side note: we are not consistent in what commitPartitionId means: usually we 
> assume that it's tableId + partitionIndex however in storage API, e.g. in 
> PartitionDataStorage#addWrite commitPartitionId itself means partition index 
> along with dedicated 
>

[jira] [Resolved] (IGNITE-22112) [SQL] TPC-H q9 query with sc=0.1 takes long time

2025-01-29 Thread Nikita Sivkov (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-22112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikita Sivkov resolved IGNITE-22112.

Resolution: Cannot Reproduce

The issue is not reproducible anymore. Q9 takes around 10 seconds now.

> [SQL] TPC-H q9 query with sc=0.1 takes long time
> 
>
> Key: IGNITE-22112
> URL: https://issues.apache.org/jira/browse/IGNITE-22112
> Project: Ignite
>  Issue Type: Bug
>  Components: sql
>Reporter: Nikita Sivkov
>Priority: Major
>  Labels: ignite-3, ignite3_performance
>
> Benchmark: 
> [https://github.com/cmu-db/benchbase/tree/main/src/main/java/com/oltpbenchmark/benchmarks/tpch]
>  
> h1. Setup
>  * 1 server node
>  * TPC-H with scale factor = 0.1
> h1. Steps
>  # Start an Ignite node
>  # Run benchbase with {{-s 1 --create=true --load=true --execute=false}} to 
> preload data
>  # Observe via the benchbase log that the data was successfully loaded
>  # Run {{benchbase with -s 1 --create=false --load=false --execute=true}} to 
> run the benchmark
> h1. Expected result
> The benchmark finishes after warmup + duration time in about the same time as 
> other requests.
> h1. Actual result
> The 
> [Q9|https://github.com/cmu-db/benchbase/blob/main/src/main/java/com/oltpbenchmark/benchmarks/tpch/procedures/Q9.java]
>  query has significantly longer execution time. For example, it may take 10 
> minutes, while the rest queries take ~1-15 seconds.
> {noformat}
> SELECT
>    nation,
>    o_year,
>    SUM(amount) AS sum_profit
> FROM
>    (
>       SELECT
>          n_name AS nation,
>          EXTRACT(YEAR
>       FROM
>          o_orderdate) AS o_year,
>          l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity AS 
> amount
>       FROM
>          part,
>          supplier,
>          lineitem,
>          partsupp,
>          orders,
>          nation
>       WHERE
>          s_suppkey = l_suppkey
>          AND ps_suppkey = l_suppkey
>          AND ps_partkey = l_partkey
>          AND p_partkey = l_partkey
>          AND o_orderkey = l_orderkey
>          AND s_nationkey = n_nationkey
>          AND p_name LIKE ?
>    )
>    AS profit
> GROUP BY
>    nation,
>    o_year
> ORDER BY
>    nation,
>    o_year DESC{noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24340) Queries permanently causing lock table overflow get retried forever

2025-01-29 Thread Roman Puchkovskiy (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roman Puchkovskiy updated IGNITE-24340:
---
Description: 
Here is a scenario (requires implicit PKs):
1. CREATE TABLE integers(i INTEGER)
2. INSERT INTO integers VALUES (0), (0), (0), (0)
3. Repeat the following query a few times (without using explicit 
transactions): 'INSERT INTO integers SELECT * FROM integers'
 
On a certain iteration, the query from item 3 hangs at least for 10 minutes 
(probably, it actually hangs forever). This is because HeapLockManager by 
default has a limited number of slots (by default, it's 1024 * 1024 ~ 1M 
slots). If there are too many locks taken at the same time, the slots are 
exhausted and lock attempt fails. It seems that SQL just repeats a query in 
such a case (at least, for implicit transactions), and there is no repeat limit.

As a result, if user's query takes too many locks, it might not succeed at all 
(with given lock table size), but the actual reason ('too small lock table') 
will not be communicated to the user. The query will simply hang forever, or, 
if a statement/query/transaction timeout is used, a timeout will happen.

This seems to be a bad UX.

> Queries permanently causing lock table overflow get retried forever
> ---
>
> Key: IGNITE-24340
> URL: https://issues.apache.org/jira/browse/IGNITE-24340
> Project: Ignite
>  Issue Type: Bug
>Reporter: Roman Puchkovskiy
>Priority: Major
>  Labels: ignite-3
>
> Here is a scenario (requires implicit PKs):
> 1. CREATE TABLE integers(i INTEGER)
> 2. INSERT INTO integers VALUES (0), (0), (0), (0)
> 3. Repeat the following query a few times (without using explicit 
> transactions): 'INSERT INTO integers SELECT * FROM integers'
>  
> On a certain iteration, the query from item 3 hangs at least for 10 minutes 
> (probably, it actually hangs forever). This is because HeapLockManager by 
> default has a limited number of slots (by default, it's 1024 * 1024 ~ 1M 
> slots). If there are too many locks taken at the same time, the slots are 
> exhausted and lock attempt fails. It seems that SQL just repeats a query in 
> such a case (at least, for implicit transactions), and there is no repeat 
> limit.
> As a result, if user's query takes too many locks, it might not succeed at 
> all (with given lock table size), but the actual reason ('too small lock 
> table') will not be communicated to the user. The query will simply hang 
> forever, or, if a statement/query/transaction timeout is used, a timeout will 
> happen.
> This seems to be a bad UX.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24343) Use ZonePartitionId as commitPartitionId

2025-01-29 Thread Alexander Lapin (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Lapin updated IGNITE-24343:
-
Description: 
h3. Motivation

Side note: we are not consistent in what commitPartitionId means: usually we 
assume that it's tableId + partitionIndex however in storage API, e.g. in 
PartitionDataStorage#addWrite commitPartitionId itself means partition index 
along with dedicated 
commitTableId. Below I will assume that commitPartitionId is a combination of 
table/zoneId and partitionIndex.
  * CommitPartition is a partition where where persisted txnState is stored. 
The state is atomically switched within TxFinish transaction phase.
 * CommitPartitionId is used in order to evaluate writeIntents through 
commitPartition path.
 * Write Intent(WI) resolution is a process that allows to define whether WI 
should be considered as applied (either committed or aborted) or ignored 
because corresponding transaction is in PENDING state. Simply, for WI(txId)
 ** Tx(txId).isPending() -> ignore WI (return previous value, or null if 
there's no such value).
 ** Tx(txId).isAborted() -> ignore WI (return previous value, or null if 
there's no such value).
 ** Tx(txId).isCommited() -> consider WI as a common value.
 * There are 3 different pathes of WI Resolution:
 ** Local path. Every node that participates in tx flow has volatile 
txnStateMap, thus it worth checking locally whether we already know the state 
of corresponding transaction.
 ** If local path is not applicable we will use Coordinator Path by requesting 
txn state from cooresponding coordinator.
 ** If Coordinator Path also not applicable we will use Commit Partition Path. 
Hence it's required to know where commit partition is.
 ** For more details please check 
[IEP-91|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211885498#IEP91:Transactionprotocol-ROtransactions.1]
 Worth mentioning that IEP is a bit outdated and both RO and RW transaction may 
see WI and thus do WI Resolution.

 
Seems that we are done with [taxidermy|https://en.wikipedia.org/wiki/Taxonomy].

> Use ZonePartitionId as commitPartitionId
> 
>
> Key: IGNITE-24343
> URL: https://issues.apache.org/jira/browse/IGNITE-24343
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Alexander Lapin
>Priority: Major
>  Labels: ignite-3
>
> h3. Motivation
> Side note: we are not consistent in what commitPartitionId means: usually we 
> assume that it's tableId + partitionIndex however in storage API, e.g. in 
> PartitionDataStorage#addWrite commitPartitionId itself means partition index 
> along with dedicated 
> commitTableId. Below I will assume that commitPartitionId is a combination of 
> table/zoneId and partitionIndex.
>   * CommitPartition is a partition where where persisted txnState is stored. 
> The state is atomically switched within TxFinish transaction phase.
>  * CommitPartitionId is used in order to evaluate writeIntents through 
> commitPartition path.
>  * Write Intent(WI) resolution is a process that allows to define whether WI 
> should be considered as applied (either committed or aborted) or ignored 
> because corresponding transaction is in PENDING state. Simply, for WI(txId)
>  ** Tx(txId).isPending() -> ignore WI (return previous value, or null if 
> there's no such value).
>  ** Tx(txId).isAborted() -> ignore WI (return previous value, or null if 
> there's no such value).
>  ** Tx(txId).isCommited() -> consider WI as a common value.
>  * There are 3 different pathes of WI Resolution:
>  ** Local path. Every node that participates in tx flow has volatile 
> txnStateMap, thus it worth checking locally whether we already know the state 
> of corresponding transaction.
>  ** If local path is not applicable we will use Coordinator Path by 
> requesting txn state from cooresponding coordinator.
>  ** If Coordinator Path also not applicable we will use Commit Partition 
> Path. Hence it's required to know where commit partition is.
>  ** For more details please check 
> [IEP-91|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211885498#IEP91:Transactionprotocol-ROtransactions.1]
>  Worth mentioning that IEP is a bit outdated and both RO and RW transaction 
> may see WI and thus do WI Resolution.
>  
> Seems that we are done with 
> [taxidermy|https://en.wikipedia.org/wiki/Taxonomy].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Assigned] (IGNITE-24334) Fix flaky ClientSessionOutboundQueueLimitTest

2025-01-29 Thread Mikhail Petrov (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Petrov reassigned IGNITE-24334:
---

Assignee: Mikhail Petrov

> Fix flaky ClientSessionOutboundQueueLimitTest
> -
>
> Key: IGNITE-24334
> URL: https://issues.apache.org/jira/browse/IGNITE-24334
> Project: Ignite
>  Issue Type: Test
>Reporter: Mikhail Petrov
>Assignee: Mikhail Petrov
>Priority: Minor
>
> See 
> https://ci2.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=-8176303230924033055&tab=testDetails
>  for more details.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24343) Use ZonePartitionId as commitPartitionId

2025-01-29 Thread Alexander Lapin (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Lapin updated IGNITE-24343:
-
Epic Link: IGNITE-22115

> Use ZonePartitionId as commitPartitionId
> 
>
> Key: IGNITE-24343
> URL: https://issues.apache.org/jira/browse/IGNITE-24343
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Alexander Lapin
>Priority: Major
>  Labels: ignite-3
>
> h3. Motivation
> Side note: we are not consistent in what commitPartitionId means: usually we 
> assume that it's tableId + partitionIndex however in storage API, e.g. in 
> PartitionDataStorage#addWrite commitPartitionId itself means partition index 
> along with dedicated 
> commitTableId. Below I will assume that commitPartitionId is a combination of 
> table/zoneId and partitionIndex.
>   * CommitPartition is a partition where where persisted txnState is stored. 
> The state is atomically switched within TxFinish transaction phase.
>  * CommitPartitionId is used in order to evaluate writeIntents through 
> commitPartition path.
>  * Write Intent(WI) resolution is a process that allows to define whether WI 
> should be considered as applied (either committed or aborted) or ignored 
> because corresponding transaction is in PENDING state. Simply, for WI(txId)
>  ** Tx(txId).isPending() -> ignore WI (return previous value, or null if 
> there's no such value).
>  ** Tx(txId).isAborted() -> ignore WI (return previous value, or null if 
> there's no such value).
>  ** Tx(txId).isCommited() -> consider WI as a common value.
>  * There are 3 different pathes of WI Resolution:
>  ** Local path. Every node that participates in tx flow has volatile 
> txnStateMap, thus it worth checking locally whether we already know the state 
> of corresponding transaction.
>  ** If local path is not applicable we will use Coordinator Path by 
> requesting txn state from cooresponding coordinator.
>  ** If Coordinator Path also not applicable we will use Commit Partition 
> Path. Hence it's required to know where commit partition is.
>  ** For more details please check 
> [IEP-91|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211885498#IEP91:Transactionprotocol-ROtransactions.1]
>  Worth mentioning that IEP is a bit outdated and both RO and RW transaction 
> may see WI and thus do WI Resolution.
>  
> Seems that we are done with 
> [taxidermy|https://en.wikipedia.org/wiki/Taxonomy].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Reopened] (IGNITE-24339) Flaky test DdlCommandHandlerExceptionHandlingTest#testZoneNotFoundOnDrop2

2025-01-29 Thread Iurii Gerzhedovich (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Iurii Gerzhedovich reopened IGNITE-24339:
-

> Flaky test DdlCommandHandlerExceptionHandlingTest#testZoneNotFoundOnDrop2
> -
>
> Key: IGNITE-24339
> URL: https://issues.apache.org/jira/browse/IGNITE-24339
> Project: Ignite
>  Issue Type: Improvement
>  Components: sql
>Reporter: Iurii Gerzhedovich
>Priority: Major
>  Labels: ignite-3
>
> The test 
> org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandlerExceptionHandlingTest#testZoneNotFoundOnDrop2
>  is flaky with the following stacktrace:
>  
> {code:java}
> java.lang.AssertionError: java.util.concurrent.ExecutionException: 
> java.lang.IllegalArgumentException: Time is out of bounds: 0   at 
> org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:78)
>  at 
> org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:35)
>  at org.hamcrest.TypeSafeMatcher.matches(TypeSafeMatcher.java:83) at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:10) at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6) at 
> org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandlerExceptionHandlingTest.testZoneNotFoundOnDrop2(DdlCommandHandlerExceptionHandlingTest.java:111)
>  at java.base/java.lang.reflect.Method.invoke(Method.java:566) at 
> java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at 
> java.base/java.util.ArrayList.forEach(ArrayList.java:1541) Caused by: 
> java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: 
> Time is out of bounds: 0 at 
> java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
>  at 
> java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2028)
>  at 
> org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:74)
>  ... 8 more Caused by: java.lang.IllegalArgumentException: Time is out of 
> bounds: 0 at 
> org.apache.ignite.internal.hlc.HybridTimestamp.(HybridTimestamp.java:111)
>  at 
> org.apache.ignite.internal.hlc.HybridTimestamp.hybridTimestamp(HybridTimestamp.java:131)
>  at 
> org.apache.ignite.internal.catalog.commands.CatalogUtils.clusterWideEnsuredActivationTimestamp(CatalogUtils.java:537)
>  at 
> org.apache.ignite.internal.catalog.CatalogManagerImpl.calcClusterWideEnsureActivationTime(CatalogManagerImpl.java:376)
>  at 
> org.apache.ignite.internal.catalog.CatalogManagerImpl.awaitVersionActivation(CatalogManagerImpl.java:370)
>  at 
> org.apache.ignite.internal.catalog.CatalogManagerImpl.awaitVersionActivation(CatalogManagerImpl.java:364)
>  at 
> java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
>  at 
> java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2241)
>  at 
> org.apache.ignite.internal.catalog.CatalogManagerImpl.saveUpdateAndWaitForActivation(CatalogManagerImpl.java:314)
>  at 
> org.apache.ignite.internal.catalog.CatalogManagerImpl.execute(CatalogManagerImpl.java:235)
>  at 
> org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandler.handle(DdlCommandHandler.java:76)
>  ... 4 more{code}
> Initial version of Catalog uses {{0L}} as activation time, which is not valid 
> value according to HybridTimestamp's constructor:
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (IGNITE-24339) Flaky test DdlCommandHandlerExceptionHandlingTest#testZoneNotFoundOnDrop2

2025-01-29 Thread Iurii Gerzhedovich (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Iurii Gerzhedovich resolved IGNITE-24339.
-
Resolution: Invalid

> Flaky test DdlCommandHandlerExceptionHandlingTest#testZoneNotFoundOnDrop2
> -
>
> Key: IGNITE-24339
> URL: https://issues.apache.org/jira/browse/IGNITE-24339
> Project: Ignite
>  Issue Type: Improvement
>  Components: sql
>Reporter: Iurii Gerzhedovich
>Priority: Major
>  Labels: ignite-3
>
> The test 
> org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandlerExceptionHandlingTest#testZoneNotFoundOnDrop2
>  is flaky with the following stacktrace:
>  
> {code:java}
> java.lang.AssertionError: java.util.concurrent.ExecutionException: 
> java.lang.IllegalArgumentException: Time is out of bounds: 0   at 
> org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:78)
>  at 
> org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:35)
>  at org.hamcrest.TypeSafeMatcher.matches(TypeSafeMatcher.java:83) at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:10) at 
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6) at 
> org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandlerExceptionHandlingTest.testZoneNotFoundOnDrop2(DdlCommandHandlerExceptionHandlingTest.java:111)
>  at java.base/java.lang.reflect.Method.invoke(Method.java:566) at 
> java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at 
> java.base/java.util.ArrayList.forEach(ArrayList.java:1541) Caused by: 
> java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: 
> Time is out of bounds: 0 at 
> java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
>  at 
> java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2028)
>  at 
> org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:74)
>  ... 8 more Caused by: java.lang.IllegalArgumentException: Time is out of 
> bounds: 0 at 
> org.apache.ignite.internal.hlc.HybridTimestamp.(HybridTimestamp.java:111)
>  at 
> org.apache.ignite.internal.hlc.HybridTimestamp.hybridTimestamp(HybridTimestamp.java:131)
>  at 
> org.apache.ignite.internal.catalog.commands.CatalogUtils.clusterWideEnsuredActivationTimestamp(CatalogUtils.java:537)
>  at 
> org.apache.ignite.internal.catalog.CatalogManagerImpl.calcClusterWideEnsureActivationTime(CatalogManagerImpl.java:376)
>  at 
> org.apache.ignite.internal.catalog.CatalogManagerImpl.awaitVersionActivation(CatalogManagerImpl.java:370)
>  at 
> org.apache.ignite.internal.catalog.CatalogManagerImpl.awaitVersionActivation(CatalogManagerImpl.java:364)
>  at 
> java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
>  at 
> java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2241)
>  at 
> org.apache.ignite.internal.catalog.CatalogManagerImpl.saveUpdateAndWaitForActivation(CatalogManagerImpl.java:314)
>  at 
> org.apache.ignite.internal.catalog.CatalogManagerImpl.execute(CatalogManagerImpl.java:235)
>  at 
> org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandler.handle(DdlCommandHandler.java:76)
>  ... 4 more{code}
> Initial version of Catalog uses {{0L}} as activation time, which is not valid 
> value according to HybridTimestamp's constructor:
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24294) IgniteCatalog API. Table's IndexDefinition reports incorrect name/sorting of index column

2025-01-29 Thread Iurii Gerzhedovich (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Iurii Gerzhedovich updated IGNITE-24294:

Description: 
Created an index with a column with ASC direction but a method that returns 
table definition and indices returns DEFAULT instead. 

Reproducer:
{code:java}
@Test
public void test() {
sql("CREATE SCHEMA s1");

sql("CREATE TABLE s1.c1 ("
+ "id INT PRIMARY KEY,"
+ "val1 INT,"
+ "val2 INT"
+ ")");

sql("CREATE INDEX s1_c1_idx1 ON s1.c1 USING HASH (val1)");
sql("CREATE INDEX s1_c1_idx2 ON s1.c1 USING SORTED (val1 desc, val2 
asc)");

TableDefinition definition = 
catalog().tableDefinition(QualifiedName.of("S1", "C1"));

for (var row : sql("SELECT * FROM SYSTEM.INDEXES WHERE 
table_name='C1'")) {
System.err.println(row);
}

List indexes = definition.indexes();
assertNotNull(indexes);

assertEquals(2, indexes.size());

System.err.println(indexes.get(0).name());
System.err.println(indexes.get(1).name());

List sortedDefs = new ArrayList<>(indexes);
sortedDefs.sort(Comparator.comparing(IndexDefinition::name));

{
IndexDefinition index = sortedDefs.get(0);
assertEquals("S1_C1_IDX1", index.name());
assertEquals(IndexType.HASH, index.type());

List columns = index.columns();
assertEquals(List.of(
ColumnSorted.column("VAL1")
), columns);
}

{
IndexDefinition index = sortedDefs.get(1);
assertEquals("S1_C1_IDX2", index.name());
assertEquals(IndexType.SORTED, index.type());

List columns = index.columns();
assertEquals(List.of(
ColumnSorted.column("VAL1", SortOrder.DESC),
ColumnSorted.column("VAL2", SortOrder.ASC)
), columns);
}
}
{code}
Output:
{noformat}
[20, S1_C1_IDX2, 17, C1, 16, S1, SORTED, false, VAL1 DESC, VAL2 ASC, AVAILABLE]
[18, C1_PK, 17, C1, 16, S1, HASH, true, ID, AVAILABLE]
[19, S1_C1_IDX1, 17, C1, 16, S1, HASH, false, VAL1, AVAILABLE]
{noformat}
Error:
{noformat}
org.opentest4j.AssertionFailedError: 
Expected :[ColumnSorted{columnName='VAL1', sortOrder=DESC}, 
ColumnSorted{columnName='VAL2', sortOrder=ASC}]
Actual   :[ColumnSorted{columnName='VAL1 DESC', sortOrder=DEFAULT}, 
ColumnSorted{columnName='VAL2 ASC', sortOrder=DEFAULT}]
{noformat}

  was:
Created an index with a column with ASC direction but a method that returns 
table definition and indices returns DEFAULT instead.

Reproducer:

{code:java}
@Test
public void test() {
sql("CREATE SCHEMA s1");

sql("CREATE TABLE s1.c1 ("
+ "id INT PRIMARY KEY,"
+ "val1 INT,"
+ "val2 INT"
+ ")");

sql("CREATE INDEX s1_c1_idx1 ON s1.c1 USING HASH (val1)");
sql("CREATE INDEX s1_c1_idx2 ON s1.c1 USING SORTED (val1 desc, val2 
asc)");

TableDefinition definition = 
catalog().tableDefinition(QualifiedName.of("S1", "C1"));

for (var row : sql("SELECT * FROM SYSTEM.INDEXES WHERE 
table_name='C1'")) {
System.err.println(row);
}

List indexes = definition.indexes();
assertNotNull(indexes);

assertEquals(2, indexes.size());

System.err.println(indexes.get(0).name());
System.err.println(indexes.get(1).name());

List sortedDefs = new ArrayList<>(indexes);
sortedDefs.sort(Comparator.comparing(IndexDefinition::name));

{
IndexDefinition index = sortedDefs.get(0);
assertEquals("S1_C1_IDX1", index.name());
assertEquals(IndexType.HASH, index.type());

List columns = index.columns();
assertEquals(List.of(
ColumnSorted.column("VAL1")
), columns);
}

{
IndexDefinition index = sortedDefs.get(1);
assertEquals("S1_C1_IDX2", index.name());
assertEquals(IndexType.SORTED, index.type());

List columns = index.columns();
assertEquals(List.of(
ColumnSorted.column("VAL1", SortOrder.DESC),
ColumnSorted.column("VAL2", SortOrder.ASC)
), columns);
}
}
{code}

Output:

{noformat}
[20, S1_C1_IDX2, 17, C1, 16, S1, SORTED, false, VAL1 DESC, VAL2 ASC, AVAILABLE]
[18, C1_PK, 17, C1, 16, S1, HASH, true, ID, AVAILABLE]
[19, S1_C1_IDX1, 17, C1, 16, S1, HASH, false, VAL1, AVAILABLE]
{noformat}

Error:

{noformat}
org.opentest4j.AssertionFailedError: 
Expected :[ColumnSorted{columnName='VAL1', sortOrder=DESC}, 
ColumnSorted{columnName='VAL2', sortOrder=ASC}]
Actual   :[ColumnSorted{columnName='VAL1 DESC', sortOrder=DEFAULT}, 
ColumnS

[jira] [Comment Edited] (IGNITE-24299) Test ticket

2025-01-29 Thread Vyacheslav Koptilin (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-24299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17921713#comment-17921713
 ] 

Vyacheslav Koptilin edited comment on IGNITE-24299 at 1/29/25 10:50 AM:


Hello [~vdmitrienko] ,

 

Could you please clarify what the purpose of this ticket is?


was (Author: slava.koptilin):
Hello [~vdmitrienko] ,

 

Could you please clarify what the purpose of this ticket?

> Test ticket
> ---
>
> Key: IGNITE-24299
> URL: https://issues.apache.org/jira/browse/IGNITE-24299
> Project: Ignite
>  Issue Type: Test
>Reporter: Vladimir Dmitrienko
>Assignee: Vladimir Dmitrienko
>Priority: Trivial
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-22621) Implement transaction storage for the zone replica listener

2025-01-29 Thread Alexander Lapin (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-22621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Lapin updated IGNITE-22621:
-
Summary: Implement transaction storage for the zone replica listener  (was: 
Implement transacation storage for the zone replica listener)

> Implement transaction storage for the zone replica listener
> ---
>
> Key: IGNITE-22621
> URL: https://issues.apache.org/jira/browse/IGNITE-22621
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Kirill Gusakov
>Assignee: Roman Puchkovskiy
>Priority: Major
>  Labels: ignite-3
> Attachments: Screenshot 2025-01-29 at 10.29.28 AM.png
>
>
> h3. Motivation
> Few words about general context. Tx Finish flow is a crucial step in AI3 Tx 
> Protocol. Generally it consists of two phases along with some prerequisites.
> 0. Prerequisites: schema validation, tx state check.
> 1. Txn state switch phase. Atomic transaction commit/rollback implemented 
> through replicating persistent txn state(COMMITTED/ABORTED) to 
> commitPartition. Once it's done, transaction is considered as finished.
> 2. Cleanup phase. Releasing locks, converting writeIntents to common values 
> or removing them depending on whether it's commit or abort.
> h3. !Screenshot 2025-01-29 at 10.29.28 AM.png! Definition of Done
>  * Move processing TxFinishReplicaRequest from 
> PartitionReplicaListener#processTxFinishAction to 
> ZonePartitionRaftListener#processTxFinishAction by introducing corresponding 
> method. In order to do that, it's required to add some dependencies to 
> ZonePartitionRaftListener:
>  * 
>  ** SchemaCompatibilityValidator
>  ** TxStateStorage // Adding the storage isn't trivial.
>  **  TxManager
>  **  RaftCommandRunner // In order to run TxFinishCommand



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (IGNITE-24342) [Flaky] Cannot reliably start 3-nodes cluster on a single Windows machine

2025-01-29 Thread Andrey Khitrin (Jira)
Andrey Khitrin created IGNITE-24342:
---

 Summary: [Flaky] Cannot reliably start 3-nodes cluster on a single 
Windows machine
 Key: IGNITE-24342
 URL: https://issues.apache.org/jira/browse/IGNITE-24342
 Project: Ignite
  Issue Type: Bug
Affects Versions: 3.0
 Environment: A single Windows 10 machine with 32 Gb of RAM
Reporter: Andrey Khitrin
 Attachments: logs.tgz

This issue doesn't have a 100% reproducibility rate, but is frequent enough to 
observe.

How to reproduce:
 # Try to start 3 AI nodes with a static `nodeFinder` on a single machine 
(configs are attached)

{code:java}
nodeFinder {
netClusterNodes=[
"127.0.0.1:3344",
"127.0.0.1:3345",
"127.0.0.1:3346"
]
type=STATIC
}
{code}
Expected result: all nodes are up.

Actual result: 2 of 3 nodes terminated with thread dumps, cannot initialize 
cluster.

Key exceptions in logs:
 # "IllegalStateException: cannot send more responses than requests" (see 
attachment)
 # Various RAFT-related and timeout errors:

{code:java}
2025-01-28 06:03:05:471 -0600 
[ERROR][%TablesAmountCapacityMultiNodeTest_cluster_1%JRaft-Response-Processor-8][AbstractClientService]
 Fail to connect TablesAmountCapacityMultiNodeTest_cluster_0, exception: 
java.util.concurrent.TimeoutException.
2025-01-28 06:03:05:815 -0600 
[INFO][%TablesAmountCapacityMultiNodeTest_cluster_1%JRaft-Request-Processor-24][NodeImpl]
 Node  ignore 
PreVoteRequest from TablesAmountCapacityMultiNodeTest_cluster_0, term=2, 
currTerm=1, because the leader TablesAmountCapacityMultiNodeTest_cluster_1's 
lease is still valid.
2025-01-28 06:03:05:815 -0600 
[ERROR][%TablesAmountCapacityMultiNodeTest_cluster_1%JRaft-Response-Processor-8][ReplicatorGroupImpl]
 Fail to check replicator connection to 
peer=TablesAmountCapacityMultiNodeTest_cluster_0, replicatorType=Follower.
2025-01-28 06:03:05:836 -0600 
[ERROR][%TablesAmountCapacityMultiNodeTest_cluster_1%JRaft-Response-Processor-8][NodeImpl]
 Fail to add a replicator, peer=TablesAmountCapacityMultiNodeTest_cluster_0.
{code}
 # Thread dumps in logs for 2 of 3 nodes (see attachment)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24275) Sql. Rework or replace HybridTimestampTracker.clientTracker

2025-01-29 Thread Konstantin Orlov (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Orlov updated IGNITE-24275:
--
Issue Type: Bug  (was: Improvement)

> Sql. Rework or replace HybridTimestampTracker.clientTracker
> ---
>
> Key: IGNITE-24275
> URL: https://issues.apache.org/jira/browse/IGNITE-24275
> Project: Ignite
>  Issue Type: Bug
>  Components: sql
>Reporter: Pavel Pereslegin
>Priority: Major
>  Labels: ignite-3
>
> In summary, the requirements for a {{HybridTimestampTracker}} (*observable 
> time tracker*) are as follows:
> * Observed time can be updated multiple times
> * Observed time can be updated from different threads
> * The updated value should be visible from different threads
> * When updating the time, the tracker must provide a guarantee that the 
> updated value is monotonically increasing
> From that perspective current implementation of 
> {{HybridTimestampTracker.clientTracker}} is broken.
> * The method {{get()}} returns only initial value (so tracker cannot be used 
> multiple times, for example for SQL script processing)
> * Consumer usually stores value to non-volatile field
> * There is no check that the new value is greater than the previous one
> We need to rework {{clientTracker}} to meet requirement or replace it with 
> {{atomicTracker}}.
> p.s. check TODO in codebase



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (IGNITE-24275) Sql. Rework or replace HybridTimestampTracker.clientTracker

2025-01-29 Thread Konstantin Orlov (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-24275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922053#comment-17922053
 ] 

Konstantin Orlov commented on IGNITE-24275:
---

Below are two tests which failed due to the bug (add these to any subclass of 
{{ClusterPerClassIntegrationTest}}):

{code}
@Override
protected void configureInitParameters(InitParametersBuilder builder) {

builder.clusterConfiguration(TestIgnitionManager.PRODUCTION_CLUSTER_CONFIG_STRING);
}

@Test
void test1() {
String address = "127.0.0.1:10800";

try (
IgniteClient client1 = IgniteClient.builder()
.addresses(address)
.build();

IgniteClient client2 = IgniteClient.builder()
.addresses(address)
.build();
) {

client1.sql().execute(null, "CREATE TABLE my_table (id INT PRIMARY 
KEY, val INT)");

// Statement below is failed with [IGN-TX-11 Table was dropped], 
while the same case executed
// through embedded client or JDBC will succeed.
try (ResultSet rs = client2.sql().execute(null, "SELECT * FROM 
my_table")) {
while (rs.hasNext()) {
rs.next();
}
}
}
}

@Test
void test2() {
String address = "127.0.0.1:10800";

try (
IgniteClient client1 = IgniteClient.builder()
.addresses(address)
.build();
) {
// Statement below is failed with [IGN-SQL-4 Failed to validate 
query. From line 1, column 104 to line 1, column 111: Object 'MY_TABLE' not 
found]
client1.sql().executeScript(""
+ "CREATE TABLE my_table (id INT PRIMARY KEY, val INT);" 
+ "START TRANSACTION READ ONLY;" 
+ "SELECT COUNT(val) FROM my_table;" 
+ "COMMIT;"
);
}
}
{code}

> Sql. Rework or replace HybridTimestampTracker.clientTracker
> ---
>
> Key: IGNITE-24275
> URL: https://issues.apache.org/jira/browse/IGNITE-24275
> Project: Ignite
>  Issue Type: Bug
>  Components: sql
>Reporter: Pavel Pereslegin
>Priority: Major
>  Labels: ignite-3
>
> In summary, the requirements for a {{HybridTimestampTracker}} (*observable 
> time tracker*) are as follows:
> * Observed time can be updated multiple times
> * Observed time can be updated from different threads
> * The updated value should be visible from different threads
> * When updating the time, the tracker must provide a guarantee that the 
> updated value is monotonically increasing
> From that perspective current implementation of 
> {{HybridTimestampTracker.clientTracker}} is broken.
> * The method {{get()}} returns only initial value (so tracker cannot be used 
> multiple times, for example for SQL script processing)
> * Consumer usually stores value to non-volatile field
> * There is no check that the new value is greater than the previous one
> We need to rework {{clientTracker}} to meet requirement or replace it with 
> {{atomicTracker}}.
> p.s. check TODO in codebase



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24304) Sql. System views. Introduce a system view for index columns

2025-01-29 Thread Iurii Gerzhedovich (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Iurii Gerzhedovich updated IGNITE-24304:

Description: 
Index system view stores index columns as a string, which makes in difficult to 
work with and requires manual parsing (e.g. IgniteCatalog provides index 
information - https://issues.apache.org/jira/browse/IGNITE-24294 ), it would be 
better to introduce a separate system view for index columns (or extend table 
columns view to include index columns).

System view name:
 * index_columns

System view columns:
 * schema_id
 * schema_name
 * table_id
 * table_name
 * index_id
 * index_name
 * column_name
 * column_ordinal - a position in an index definition
 * collation (desc, asc, null first, null last, etc, see IgniteCatalog 
SortOrder) (must be null for hash indices)

  was:
Index system view stores index columns as a string, which makes in difficult to 
work with and requires manual parsing (e.g. IgniteCatalog provides index 
information - https://issues.apache.org/jira/browse/IGNITE-24294 ), it would be 
better to introduce a separate system view for index columns (or extend table 
columns view to include index columns).

System view name: 
* index_columns

System view columns:
* schema_id
* schema_name
* table_id
* table_name
* index_id
* index_name
* column_ordinal - a position in an index definition
* collation (desc, asc, null first, null last, etc, see IgniteCatalog 
SortOrder) (must be null for hash indices)
 


> Sql. System views. Introduce a system view for index columns
> 
>
> Key: IGNITE-24304
> URL: https://issues.apache.org/jira/browse/IGNITE-24304
> Project: Ignite
>  Issue Type: Improvement
>  Components: sql
>Reporter: Maksim Zhuravkov
>Priority: Major
>  Labels: ignite-3
>
> Index system view stores index columns as a string, which makes in difficult 
> to work with and requires manual parsing (e.g. IgniteCatalog provides index 
> information - https://issues.apache.org/jira/browse/IGNITE-24294 ), it would 
> be better to introduce a separate system view for index columns (or extend 
> table columns view to include index columns).
> System view name:
>  * index_columns
> System view columns:
>  * schema_id
>  * schema_name
>  * table_id
>  * table_name
>  * index_id
>  * index_name
>  * column_name
>  * column_ordinal - a position in an index definition
>  * collation (desc, asc, null first, null last, etc, see IgniteCatalog 
> SortOrder) (must be null for hash indices)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24275) Sql. Rework or replace HybridTimestampTracker.clientTracker

2025-01-29 Thread Pavel Pereslegin (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pavel Pereslegin updated IGNITE-24275:
--
Description: 
In summary, the requirements for a {{HybridTimestampTracker}} (*observable time 
tracker*) are as follows:

* Observed time can be updated multiple times
* Observed time can be updated from different threads
* The updated value should be visible from different threads
* When updating the time, the tracker must provide a guarantee that the updated 
value is monotonically increasing

>From that perspective current implementation of 
>{{HybridTimestampTracker.clientTracker}} is broken.

* The method {{get()}} returns only initial value (so tracker cannot be used 
multiple times, for example for SQL script processing)
* Consumer usually stores value to non-volatile field
* There is no check that the new value is greater than the previous one

We need to rework {{clientTracker}} to meet requirement or replace it with 
{{atomicTracker}}.

p.s. check TODO in codebase

  was:
In summary, the requirements for a {{HybridTimestampTracker}} (*observable time 
tracker*) are as follows:

* Observed time can be updated multiple times
* Observed time can be updated from different threads
* The updated value should be visible from different threads
* When updating the time, the tracker must provide a guarantee that the updated 
value is monotonically increasing

>From that perspective current implementation of 
>{{HybridTimestampTracker.clientTracker}} is broken.

* The method {{get()}} returns only initial value (so tracker cannot be used 
multiple times, for example for SQL script processing)
* Consumer usually stores value to non-volatile field
* There is no check that the new value is greater than the previous one

We need to rework {{clientTracker}} to meet requirement or replace it where 
with {{atomicTracker}}.

p.s. check TODO in codebase


> Sql. Rework or replace HybridTimestampTracker.clientTracker
> ---
>
> Key: IGNITE-24275
> URL: https://issues.apache.org/jira/browse/IGNITE-24275
> Project: Ignite
>  Issue Type: Improvement
>  Components: sql
>Reporter: Pavel Pereslegin
>Priority: Major
>  Labels: ignite-3
>
> In summary, the requirements for a {{HybridTimestampTracker}} (*observable 
> time tracker*) are as follows:
> * Observed time can be updated multiple times
> * Observed time can be updated from different threads
> * The updated value should be visible from different threads
> * When updating the time, the tracker must provide a guarantee that the 
> updated value is monotonically increasing
> From that perspective current implementation of 
> {{HybridTimestampTracker.clientTracker}} is broken.
> * The method {{get()}} returns only initial value (so tracker cannot be used 
> multiple times, for example for SQL script processing)
> * Consumer usually stores value to non-volatile field
> * There is no check that the new value is greater than the previous one
> We need to rework {{clientTracker}} to meet requirement or replace it with 
> {{atomicTracker}}.
> p.s. check TODO in codebase



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (IGNITE-24341) Sql. System views. Add ordinal column to table_columns

2025-01-29 Thread Maksim Zhuravkov (Jira)
Maksim Zhuravkov created IGNITE-24341:
-

 Summary: Sql. System views. Add ordinal column to table_columns
 Key: IGNITE-24341
 URL: https://issues.apache.org/jira/browse/IGNITE-24341
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Reporter: Maksim Zhuravkov


TABLE_COLUMNS  system view does not include information on column position in a 
table definition. Let's add `column_ordinal` column to TABLE_COLUMNS view that 
contains it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (IGNITE-24299) Test ticket

2025-01-29 Thread Vyacheslav Koptilin (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-24299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922060#comment-17922060
 ] 

Vyacheslav Koptilin commented on IGNITE-24299:
--

Ok. Thank you for the clarification! Please don't forget to close the ticket 
then the issue is resolved.

> Test ticket
> ---
>
> Key: IGNITE-24299
> URL: https://issues.apache.org/jira/browse/IGNITE-24299
> Project: Ignite
>  Issue Type: Test
>Reporter: Vladimir Dmitrienko
>Assignee: Vladimir Dmitrienko
>Priority: Trivial
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (IGNITE-24299) Test ticket

2025-01-29 Thread Vladimir Dmitrienko (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-24299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922061#comment-17922061
 ] 

Vladimir Dmitrienko commented on IGNITE-24299:
--

Sure (y)

> Test ticket
> ---
>
> Key: IGNITE-24299
> URL: https://issues.apache.org/jira/browse/IGNITE-24299
> Project: Ignite
>  Issue Type: Test
>Reporter: Vladimir Dmitrienko
>Assignee: Vladimir Dmitrienko
>Priority: Trivial
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (IGNITE-24339) Flaky test DdlCommandHandlerExceptionHandlingTest#testZoneNotFoundOnDrop2

2025-01-29 Thread Iurii Gerzhedovich (Jira)
Iurii Gerzhedovich created IGNITE-24339:
---

 Summary: Flaky test 
DdlCommandHandlerExceptionHandlingTest#testZoneNotFoundOnDrop2
 Key: IGNITE-24339
 URL: https://issues.apache.org/jira/browse/IGNITE-24339
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Reporter: Iurii Gerzhedovich


The test 
org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandlerExceptionHandlingTest#testZoneNotFoundOnDrop2
 is flaky with the following stacktrace:
 
{code:java}
java.lang.AssertionError: java.util.concurrent.ExecutionException: 
java.lang.IllegalArgumentException: Time is out of bounds: 0   at 
org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:78)
 at 
org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:35)
 at org.hamcrest.TypeSafeMatcher.matches(TypeSafeMatcher.java:83) at 
org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:10) at 
org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6) at 
org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandlerExceptionHandlingTest.testZoneNotFoundOnDrop2(DdlCommandHandlerExceptionHandlingTest.java:111)
 at java.base/java.lang.reflect.Method.invoke(Method.java:566) at 
java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at 
java.base/java.util.ArrayList.forEach(ArrayList.java:1541) Caused by: 
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: 
Time is out of bounds: 0 at 
java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
 at 
java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2028)
 at 
org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:74)
 ... 8 more Caused by: java.lang.IllegalArgumentException: Time is out of 
bounds: 0 at 
org.apache.ignite.internal.hlc.HybridTimestamp.(HybridTimestamp.java:111) 
at 
org.apache.ignite.internal.hlc.HybridTimestamp.hybridTimestamp(HybridTimestamp.java:131)
 at 
org.apache.ignite.internal.catalog.commands.CatalogUtils.clusterWideEnsuredActivationTimestamp(CatalogUtils.java:537)
 at 
org.apache.ignite.internal.catalog.CatalogManagerImpl.calcClusterWideEnsureActivationTime(CatalogManagerImpl.java:376)
 at 
org.apache.ignite.internal.catalog.CatalogManagerImpl.awaitVersionActivation(CatalogManagerImpl.java:370)
 at 
org.apache.ignite.internal.catalog.CatalogManagerImpl.awaitVersionActivation(CatalogManagerImpl.java:364)
 at 
java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
 at 
java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2241)
 at 
org.apache.ignite.internal.catalog.CatalogManagerImpl.saveUpdateAndWaitForActivation(CatalogManagerImpl.java:314)
 at 
org.apache.ignite.internal.catalog.CatalogManagerImpl.execute(CatalogManagerImpl.java:235)
 at 
org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandler.handle(DdlCommandHandler.java:76)
 ... 4 more{code}

Initial version of Catalog uses {{0L}} as activation time, which is not valid 
value according to HybridTimestamp's constructor:
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24330) Revise DisasterRecoveryManager API in alignment with Colocation track

2025-01-29 Thread Mirza Aliev (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mirza Aliev updated IGNITE-24330:
-
Description: 
h3. Motivation
In [collocation|https://ggsystems.atlassian.net/issues/IGN-24435] track we 
change the way how partitions are aligned with tables, so partitions will be 
the part of zones and tables in one zone will share common partitions from a 
zone.

Currently, the DisasterRecoveryManager API for reset/restart partition 
operations is based on the assumption that partitions are tied to table 
entities.

Example of {{resetPartitions}} method description:

{code:java}
 * @param zoneName Name of the distribution zone. Case-sensitive, without 
quotes.
 * @param schemaName Schema name. Case-sensitive, without quotes.
 * @param tableName Table name. Case-sensitive, without quotes.
 * @param partitionIds IDs of partitions to reset. If empty, reset all 
zone's partitions.
 
 */
private CompletableFuture resetPartitions(
String zoneName,
String schemaName,
String tableName,
Set partitionIds,
...
) {
{code}

Problems with the current API:

* The current API assumes partitions are part of a specific table, making it 
unclear how to handle partition resets in the new colocation model.
* Users don't want to affect partition distribution across all tables in a zone 
when specifying a reset for a particular table.
  
We need to provide a way to use the DisasterRecoveryManager API in a way that 
reflects the new colocation model, where partitions belong to zones rather than 
being tied to individual tables.

As a solution, we could introduce a revised DisasterRecoveryManagerV2 API that 
removes direct table references and operates at the zone level. Once the 
colocation track is completed, we can switch to DisasterRecoveryManagerV2 and 
deprecate the existing implementation. However, it is still unclear which 
methods will be required and how they will impact table distribution.

h3. Definition of Done

* DisasterRecoveryManager API is enriched with the methods to work with zones 
partitions 


  was:
h3. Motivation
In [collocation|https://ggsystems.atlassian.net/issues/IGN-24435] track we 
change the way how partitions are aligned with tables, so partitions will be 
the part of zones and tables in one zone will share common partitions from a 
zone.

Currently, the DisasterRecoveryManager API for reset/restart partition 
operations is based on the assumption that partitions are tied to table 
entities.

Example of {{resetPartitions}} method description:

{code:java}
 * @param zoneName Name of the distribution zone. Case-sensitive, without 
quotes.
 * @param schemaName Schema name. Case-sensitive, without quotes.
 * @param tableName Table name. Case-sensitive, without quotes.
 * @param partitionIds IDs of partitions to reset. If empty, reset all 
zone's partitions.
 
 */
private CompletableFuture resetPartitions(
String zoneName,
String schemaName,
String tableName,
Set partitionIds,
...
) {
{code}

Problems with the current API:

* The current API assumes partitions are part of a specific table, making it 
unclear how to handle partition resets in the new colocation model.
* Users don't want to affect partition distribution across all tables in a zone 
when specifying a reset for a particular table.
  
We need to provide a way to use the DisasterRecoveryManager API in a way that 
reflects the new colocation model, where partitions belong to zones rather than 
being tied to individual tables.

As a solution, we could introduce a revised DisasterRecoveryManagerV2 API that 
removes direct table references and operates at the zone level. Once the 
colocation track is completed, we can switch to DisasterRecoveryManagerV2 and 
deprecate the existing implementation. However, it is still unclear which 
methods will be required and how they will impact table distribution.




> Revise DisasterRecoveryManager API in alignment with Colocation track
> -
>
> Key: IGNITE-24330
> URL: https://issues.apache.org/jira/browse/IGNITE-24330
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Mirza Aliev
>Priority: Major
>  Labels: ignite-3
>
> h3. Motivation
> In [collocation|https://ggsystems.atlassian.net/issues/IGN-24435] track we 
> change the way how partitions are aligned with tables, so partitions will be 
> the part of zones and tables in one zone will share common partitions from a 
> zone.
> Currently, the DisasterRecoveryManager API for reset/restart partition 
> operations is based on the assumption that partitions are tied to table 
> entities.
> Example of {{resetPart

[jira] [Updated] (IGNITE-24232) Implement correct node restart behaviour when forced pending

2025-01-29 Thread Mirza Aliev (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mirza Aliev updated IGNITE-24232:
-
Epic Link:   (was: IGNITE-22115)

> Implement correct node restart behaviour when forced pending
> 
>
> Key: IGNITE-24232
> URL: https://issues.apache.org/jira/browse/IGNITE-24232
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Mirza Aliev
>Priority: Major
>  Labels: ignite-3
>
> h3. Motivation
> In https://issues.apache.org/jira/browse/IGNITE-23780 we have improved 
> correct node behaviour when node restarts according to 
> stable/pending(forced/not forced) keys. Several places in the code where we 
> start partitions where improved.
> The same improvements must be done based on collocation 
> stable/pending(forced/not forced) keys, where zones partitions are started. 
> Details could be found in the original ticket 
> https://issues.apache.org/jira/browse/IGNITE-23780 
> h3. Definition of Done
> * All changes from https://issues.apache.org/jira/browse/IGNITE-23780 in the 
> partition restart flow must be adopted for the collocation rebalance flow.  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24230) Support Disaster Recovery methods for the collocation case

2025-01-29 Thread Mirza Aliev (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mirza Aliev updated IGNITE-24230:
-
Epic Link:   (was: IGNITE-22115)

> Support Disaster Recovery methods for the collocation case
> --
>
> Key: IGNITE-24230
> URL: https://issues.apache.org/jira/browse/IGNITE-24230
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Mirza Aliev
>Priority: Major
>  Labels: ignite-3
>
> h3. Motivation 
> DisasterRecoveryManager has logic for reset partitions, which sets force 
> pending assignments. We need to support such functionality for collocation 
> pending keys.
> Also there is logic which reacts on HA zone topology reduced 
> `DisasterRecoveryManager#onHaZoneTopologyReduce`. It calculates set of 
> partitions which lost majority, based on stable assignments. The same logic 
> must be done based on collocation stable keys.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24230) Support Disaster Recovery methods for the collocation case

2025-01-29 Thread Mirza Aliev (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mirza Aliev updated IGNITE-24230:
-
Description: 
h3. Motivation 

Once we revise DisasterRecoveryManager API in alignment with Colocation track 
https://issues.apache.org/jira/browse/IGNITE-24330 and implement all related 
tickets, so DisasterRecoveryManager API is improved with new methods that work 
with zones' partitions, 
we need to implement REST API for all new methods. See 
{{DisasterRecoveryController}}

h3. Definition of done 
*  REST API of DisasterRecoveryManager is improved with new methods that were 
implemented for zones' partitions.

  was:
h3. Motivation 

DisasterRecoveryManager has logic for reset partitions, which sets force 
pending assignments. We need to support such functionality for collocation 
pending keys.

Also there is logic which reacts on HA zone topology reduced 
`DisasterRecoveryManager#onHaZoneTopologyReduce`. It calculates set of 
partitions which lost majority, based on stable assignments. The same logic 
must be done based on collocation stable keys.



> Support Disaster Recovery methods for the collocation case
> --
>
> Key: IGNITE-24230
> URL: https://issues.apache.org/jira/browse/IGNITE-24230
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Mirza Aliev
>Priority: Major
>  Labels: ignite-3
>
> h3. Motivation 
> Once we revise DisasterRecoveryManager API in alignment with Colocation track 
> https://issues.apache.org/jira/browse/IGNITE-24330 and implement all related 
> tickets, so DisasterRecoveryManager API is improved with new methods that 
> work with zones' partitions, 
> we need to implement REST API for all new methods. See 
> {{DisasterRecoveryController}}
> h3. Definition of done 
> *  REST API of DisasterRecoveryManager is improved with new methods that were 
> implemented for zones' partitions.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24331) Support DisasterRecoveryManager#localPartitionStates and globalPartitionStates in Collocation track

2025-01-29 Thread Mirza Aliev (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mirza Aliev updated IGNITE-24331:
-
Summary: Support DisasterRecoveryManager#localPartitionStates and 
globalPartitionStates in Collocation track  (was: Support 
DisasterRecoveryManager#localPartitionStates and globalPartitionStates in 
Colocation track)

> Support DisasterRecoveryManager#localPartitionStates and 
> globalPartitionStates in Collocation track
> ---
>
> Key: IGNITE-24331
> URL: https://issues.apache.org/jira/browse/IGNITE-24331
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Mirza Aliev
>Priority: Major
>  Labels: ignite-3
>
> h3. Motivation
> This ticket is a part of the implementation track for 
> https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised 
> DisasterRecoveryManager API in alignment with Colocation track
> In this ticket we have to provide a correct behaviour for 
> {{DisasterRecoveryManager#localPartitionStates}} and close method 
> {{DisasterRecoveryManager#globalPartitionStates}} which uses 
> {{localPartitionStates}}.
> Currently, those methods return a mapping of {{TablePartitionId}} to the 
> mapping between a node name and a partition state. 
> To conform Collocation track, those methods must use {{ZonePartitionId}} and 
> hence return states of a zone's partition rather than table's.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24331) Support DisasterRecoveryManager#localPartitionStates and globalPartitionStates in Collocation track

2025-01-29 Thread Mirza Aliev (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mirza Aliev updated IGNITE-24331:
-
Description: 
h3. Motivation
This ticket is a part of the implementation track for 
https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised 
DisasterRecoveryManager API in alignment with Colocation track

In this ticket we have to provide a correct behaviour for 
{{DisasterRecoveryManager#localPartitionStates}} and close method 
{{DisasterRecoveryManager#globalPartitionStates}} which uses 
{{localPartitionStates}}.
Currently, those methods return a mapping of {{TablePartitionId}} to the 
mapping between a node name and a partition state. 

To conform Collocation track, those methods must use {{ZonePartitionId}} and 
hence return states of a zone's partition rather than table's.

h3. Definition of Done 

* DisasterRecoveryManager API is enriched with {{localPartitionStates}} and 
{{globalPartitionStates}} that are work with zones' based partitions  

  was:
h3. Motivation
This ticket is a part of the implementation track for 
https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised 
DisasterRecoveryManager API in alignment with Colocation track

In this ticket we have to provide a correct behaviour for 
{{DisasterRecoveryManager#localPartitionStates}} and close method 
{{DisasterRecoveryManager#globalPartitionStates}} which uses 
{{localPartitionStates}}.
Currently, those methods return a mapping of {{TablePartitionId}} to the 
mapping between a node name and a partition state. 

To conform Collocation track, those methods must use {{ZonePartitionId}} and 
hence return states of a zone's partition rather than table's.


> Support DisasterRecoveryManager#localPartitionStates and 
> globalPartitionStates in Collocation track
> ---
>
> Key: IGNITE-24331
> URL: https://issues.apache.org/jira/browse/IGNITE-24331
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Mirza Aliev
>Priority: Major
>  Labels: ignite-3
>
> h3. Motivation
> This ticket is a part of the implementation track for 
> https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised 
> DisasterRecoveryManager API in alignment with Colocation track
> In this ticket we have to provide a correct behaviour for 
> {{DisasterRecoveryManager#localPartitionStates}} and close method 
> {{DisasterRecoveryManager#globalPartitionStates}} which uses 
> {{localPartitionStates}}.
> Currently, those methods return a mapping of {{TablePartitionId}} to the 
> mapping between a node name and a partition state. 
> To conform Collocation track, those methods must use {{ZonePartitionId}} and 
> hence return states of a zone's partition rather than table's.
> h3. Definition of Done 
> * DisasterRecoveryManager API is enriched with {{localPartitionStates}} and 
> {{globalPartitionStates}} that are work with zones' based partitions  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24332) Support DisasterRecoveryManager#resetPartitions in Collocation track

2025-01-29 Thread Mirza Aliev (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mirza Aliev updated IGNITE-24332:
-
Summary: Support DisasterRecoveryManager#resetPartitions in Collocation 
track  (was: Support DisasterRecoveryManager#resetPartitions in Colocation 
track)

> Support DisasterRecoveryManager#resetPartitions in Collocation track
> 
>
> Key: IGNITE-24332
> URL: https://issues.apache.org/jira/browse/IGNITE-24332
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Mirza Aliev
>Priority: Major
>  Labels: ignite-3
>
> h3. Motivation
> This ticket is a part of the implementation track for 
> https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised 
> DisasterRecoveryManager API in alignment with Colocation track
> In this ticket we have to provide a correct behaviour for 
> {{DisasterRecoveryManager#resetPartitions}} for partitions from zones.
> h3. Implementation notes
> Taking into account that prerequisite with {{localPartitionStates}} is 
> already implemented (https://issues.apache.org/jira/browse/IGNITE-24331), in 
> this ticket we must just reuse code from the current 
> {{DisasterRecoveryManager#resetPartitions}}, use correct 
> {{localPartitionStates}} method and change 
> {{ZoneRebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}} key rather than 
> {{RebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24331) Support DisasterRecoveryManager#localPartitionStates and globalPartitionStates in Collocation track

2025-01-29 Thread Mirza Aliev (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mirza Aliev updated IGNITE-24331:
-
Description: 
h3. Motivation
This ticket is a part of the implementation track for 
https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised 
DisasterRecoveryManager API in alignment with Colocation track

In this ticket we have to provide a correct behaviour for 
{{DisasterRecoveryManager#localPartitionStates}} and close method 
{{DisasterRecoveryManager#globalPartitionStates}} which uses 
{{localPartitionStates}}.
Currently, those methods return a mapping of {{TablePartitionId}} to the 
mapping between a node name and a partition state. 

To conform Collocation track, those methods must use {{ZonePartitionId}} and 
hence return states of a zone's partition rather than table's.

h3. Definition of Done 

* DisasterRecoveryManager API is enriched with {{localPartitionStates}} and 
{{globalPartitionStates}} methods that are work with zones' based partitions  

  was:
h3. Motivation
This ticket is a part of the implementation track for 
https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised 
DisasterRecoveryManager API in alignment with Colocation track

In this ticket we have to provide a correct behaviour for 
{{DisasterRecoveryManager#localPartitionStates}} and close method 
{{DisasterRecoveryManager#globalPartitionStates}} which uses 
{{localPartitionStates}}.
Currently, those methods return a mapping of {{TablePartitionId}} to the 
mapping between a node name and a partition state. 

To conform Collocation track, those methods must use {{ZonePartitionId}} and 
hence return states of a zone's partition rather than table's.

h3. Definition of Done 

* DisasterRecoveryManager API is enriched with {{localPartitionStates}} and 
{{globalPartitionStates}} that are work with zones' based partitions  


> Support DisasterRecoveryManager#localPartitionStates and 
> globalPartitionStates in Collocation track
> ---
>
> Key: IGNITE-24331
> URL: https://issues.apache.org/jira/browse/IGNITE-24331
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Mirza Aliev
>Priority: Major
>  Labels: ignite-3
>
> h3. Motivation
> This ticket is a part of the implementation track for 
> https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised 
> DisasterRecoveryManager API in alignment with Colocation track
> In this ticket we have to provide a correct behaviour for 
> {{DisasterRecoveryManager#localPartitionStates}} and close method 
> {{DisasterRecoveryManager#globalPartitionStates}} which uses 
> {{localPartitionStates}}.
> Currently, those methods return a mapping of {{TablePartitionId}} to the 
> mapping between a node name and a partition state. 
> To conform Collocation track, those methods must use {{ZonePartitionId}} and 
> hence return states of a zone's partition rather than table's.
> h3. Definition of Done 
> * DisasterRecoveryManager API is enriched with {{localPartitionStates}} and 
> {{globalPartitionStates}} methods that are work with zones' based partitions  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24332) Support DisasterRecoveryManager#resetPartitions in Collocation track

2025-01-29 Thread Mirza Aliev (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mirza Aliev updated IGNITE-24332:
-
Description: 
h3. Motivation
This ticket is a part of the implementation track for 
https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised 
DisasterRecoveryManager API in alignment with Colocation track

In this ticket we have to provide a correct behaviour for 
{{DisasterRecoveryManager#resetPartitions}} for partitions from zones.

h3. Implementation notes
Taking into account that prerequisite with {{localPartitionStates}} is already 
implemented (https://issues.apache.org/jira/browse/IGNITE-24331), in this 
ticket we must just reuse code from the current 
{{DisasterRecoveryManager#resetPartitions}}, use correct 
{{localPartitionStates}} method and change 
{{ZoneRebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}} key rather than 
{{RebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}}

h3. Definition of Done

* DisasterRecoveryManager API is enriched with

  was:
h3. Motivation
This ticket is a part of the implementation track for 
https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised 
DisasterRecoveryManager API in alignment with Colocation track

In this ticket we have to provide a correct behaviour for 
{{DisasterRecoveryManager#resetPartitions}} for partitions from zones.

h3. Implementation notes
Taking into account that prerequisite with {{localPartitionStates}} is already 
implemented (https://issues.apache.org/jira/browse/IGNITE-24331), in this 
ticket we must just reuse code from the current 
{{DisasterRecoveryManager#resetPartitions}}, use correct 
{{localPartitionStates}} method and change 
{{ZoneRebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}} key rather than 
{{RebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}}


> Support DisasterRecoveryManager#resetPartitions in Collocation track
> 
>
> Key: IGNITE-24332
> URL: https://issues.apache.org/jira/browse/IGNITE-24332
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Mirza Aliev
>Priority: Major
>  Labels: ignite-3
>
> h3. Motivation
> This ticket is a part of the implementation track for 
> https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised 
> DisasterRecoveryManager API in alignment with Colocation track
> In this ticket we have to provide a correct behaviour for 
> {{DisasterRecoveryManager#resetPartitions}} for partitions from zones.
> h3. Implementation notes
> Taking into account that prerequisite with {{localPartitionStates}} is 
> already implemented (https://issues.apache.org/jira/browse/IGNITE-24331), in 
> this ticket we must just reuse code from the current 
> {{DisasterRecoveryManager#resetPartitions}}, use correct 
> {{localPartitionStates}} method and change 
> {{ZoneRebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}} key rather than 
> {{RebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}}
> h3. Definition of Done
> * DisasterRecoveryManager API is enriched with



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24331) Support DisasterRecoveryManager#localPartitionStates and globalPartitionStates in Collocation track

2025-01-29 Thread Mirza Aliev (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mirza Aliev updated IGNITE-24331:
-
Description: 
h3. Motivation
This ticket is a part of the implementation track for 
https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised 
DisasterRecoveryManager API in alignment with Colocation track

In this ticket we have to provide a correct behaviour for 
{{DisasterRecoveryManager#localPartitionStates}} and close method 
{{DisasterRecoveryManager#globalPartitionStates}} which uses 
{{localPartitionStates}}.
Currently, those methods return a mapping of {{TablePartitionId}} to the 
mapping between a node name and a partition state. 

To conform Collocation track, those methods must use {{ZonePartitionId}} and 
hence return states of a zone's partition rather than table's.

h3. Definition of Done 

* DisasterRecoveryManager API is enriched with {{localPartitionStates}} and 
{{globalPartitionStates}} methods that work with zones' based partitions  

  was:
h3. Motivation
This ticket is a part of the implementation track for 
https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised 
DisasterRecoveryManager API in alignment with Colocation track

In this ticket we have to provide a correct behaviour for 
{{DisasterRecoveryManager#localPartitionStates}} and close method 
{{DisasterRecoveryManager#globalPartitionStates}} which uses 
{{localPartitionStates}}.
Currently, those methods return a mapping of {{TablePartitionId}} to the 
mapping between a node name and a partition state. 

To conform Collocation track, those methods must use {{ZonePartitionId}} and 
hence return states of a zone's partition rather than table's.

h3. Definition of Done 

* DisasterRecoveryManager API is enriched with {{localPartitionStates}} and 
{{globalPartitionStates}} methods that are work with zones' based partitions  


> Support DisasterRecoveryManager#localPartitionStates and 
> globalPartitionStates in Collocation track
> ---
>
> Key: IGNITE-24331
> URL: https://issues.apache.org/jira/browse/IGNITE-24331
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Mirza Aliev
>Priority: Major
>  Labels: ignite-3
>
> h3. Motivation
> This ticket is a part of the implementation track for 
> https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised 
> DisasterRecoveryManager API in alignment with Colocation track
> In this ticket we have to provide a correct behaviour for 
> {{DisasterRecoveryManager#localPartitionStates}} and close method 
> {{DisasterRecoveryManager#globalPartitionStates}} which uses 
> {{localPartitionStates}}.
> Currently, those methods return a mapping of {{TablePartitionId}} to the 
> mapping between a node name and a partition state. 
> To conform Collocation track, those methods must use {{ZonePartitionId}} and 
> hence return states of a zone's partition rather than table's.
> h3. Definition of Done 
> * DisasterRecoveryManager API is enriched with {{localPartitionStates}} and 
> {{globalPartitionStates}} methods that work with zones' based partitions  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24332) Support DisasterRecoveryManager#resetPartitions in Collocation track

2025-01-29 Thread Mirza Aliev (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mirza Aliev updated IGNITE-24332:
-
Description: 
h3. Motivation
This ticket is a part of the implementation track for 
https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised 
DisasterRecoveryManager API in alignment with Colocation track

In this ticket we have to provide a correct behaviour for 
{{DisasterRecoveryManager#resetPartitions}} for partitions from zones.

h3. Implementation notes
Taking into account that prerequisite with {{localPartitionStates}} is already 
implemented (https://issues.apache.org/jira/browse/IGNITE-24331), in this 
ticket we must just reuse code from the current 
{{DisasterRecoveryManager#resetPartitions}}, use correct 
{{localPartitionStates}} method and change 
{{ZoneRebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}} key rather than 
{{RebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}}

h3. Definition of Done

* DisasterRecoveryManager API is enriched with {{resetPartitions}} method that 
works with zones' based partitions

  was:
h3. Motivation
This ticket is a part of the implementation track for 
https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised 
DisasterRecoveryManager API in alignment with Colocation track

In this ticket we have to provide a correct behaviour for 
{{DisasterRecoveryManager#resetPartitions}} for partitions from zones.

h3. Implementation notes
Taking into account that prerequisite with {{localPartitionStates}} is already 
implemented (https://issues.apache.org/jira/browse/IGNITE-24331), in this 
ticket we must just reuse code from the current 
{{DisasterRecoveryManager#resetPartitions}}, use correct 
{{localPartitionStates}} method and change 
{{ZoneRebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}} key rather than 
{{RebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}}

h3. Definition of Done

* DisasterRecoveryManager API is enriched with


> Support DisasterRecoveryManager#resetPartitions in Collocation track
> 
>
> Key: IGNITE-24332
> URL: https://issues.apache.org/jira/browse/IGNITE-24332
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Mirza Aliev
>Priority: Major
>  Labels: ignite-3
>
> h3. Motivation
> This ticket is a part of the implementation track for 
> https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised 
> DisasterRecoveryManager API in alignment with Colocation track
> In this ticket we have to provide a correct behaviour for 
> {{DisasterRecoveryManager#resetPartitions}} for partitions from zones.
> h3. Implementation notes
> Taking into account that prerequisite with {{localPartitionStates}} is 
> already implemented (https://issues.apache.org/jira/browse/IGNITE-24331), in 
> this ticket we must just reuse code from the current 
> {{DisasterRecoveryManager#resetPartitions}}, use correct 
> {{localPartitionStates}} method and change 
> {{ZoneRebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}} key rather than 
> {{RebalanceUtil#PENDING_ASSIGNMENTS_PREFIX}}
> h3. Definition of Done
> * DisasterRecoveryManager API is enriched with {{resetPartitions}} method 
> that works with zones' based partitions



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24335) Support DisasterRecoveryManager#restartPartition in Collocation track

2025-01-29 Thread Mirza Aliev (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mirza Aliev updated IGNITE-24335:
-
Description: 
h3. Motivation
This ticket is a part of the implementation track for 
https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised 
DisasterRecoveryManager API in alignment with Collocation track

In this ticket we have to provide a correct behaviour for 
{{DisasterRecoveryManager#restartPartition}} for partitions from zones.

h3. Implementation notes

As usual for the such methods from DisasterRecoveryManager, we must switch to 
{{ZonePartitionId}}. Also this mechanism uses under the hood 
{{TableManager#restartPartition}}. Seems that we need to have similar method in 
{{PartitionReplicaLifecycleManager}}, which has all prerequisites like  
{{PartitionReplicaLifecycleManager#weakStopPartition}}, 
{{PartitionReplicaLifecycleManager#waitForMetadataCompleteness}}

h3. Definition of Done

* DisasterRecoveryManager API is enriched with {{restartPartition}} method that 
works with zones' based partitions

  was:
h3. Motivation
This ticket is a part of the implementation track for 
https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised 
DisasterRecoveryManager API in alignment with Collocation track

In this ticket we have to provide a correct behaviour for 
{{DisasterRecoveryManager#restartPartition}} for partitions from zones.

h3. Implementation notes

As usual for the such methods from DisasterRecoveryManager, we must switch to 
{{ZonePartitionId}}. Also this mechanism uses under the hood 
{{TableManager#restartPartition}}. Seems that we need to have similar method in 
{{PartitionReplicaLifecycleManager}}, which has all prerequisites like  
{{PartitionReplicaLifecycleManager#weakStopPartition}}, 
{{PartitionReplicaLifecycleManager#waitForMetadataCompleteness}}



> Support DisasterRecoveryManager#restartPartition in Collocation track
> -
>
> Key: IGNITE-24335
> URL: https://issues.apache.org/jira/browse/IGNITE-24335
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Mirza Aliev
>Priority: Major
>  Labels: ignite-3
>
> h3. Motivation
> This ticket is a part of the implementation track for 
> https://issues.apache.org/jira/browse/IGNITE-24330, where we have revised 
> DisasterRecoveryManager API in alignment with Collocation track
> In this ticket we have to provide a correct behaviour for 
> {{DisasterRecoveryManager#restartPartition}} for partitions from zones.
> h3. Implementation notes
> As usual for the such methods from DisasterRecoveryManager, we must switch to 
> {{ZonePartitionId}}. Also this mechanism uses under the hood 
> {{TableManager#restartPartition}}. Seems that we need to have similar method 
> in {{PartitionReplicaLifecycleManager}}, which has all prerequisites like  
> {{PartitionReplicaLifecycleManager#weakStopPartition}}, 
> {{PartitionReplicaLifecycleManager#waitForMetadataCompleteness}}
> h3. Definition of Done
> * DisasterRecoveryManager API is enriched with {{restartPartition}} method 
> that works with zones' based partitions



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24230) Enrich Disaster Recovery REST API with new methods for the collocation case

2025-01-29 Thread Mirza Aliev (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mirza Aliev updated IGNITE-24230:
-
Summary: Enrich Disaster Recovery REST API with new methods for the 
collocation case  (was: Support Disaster Recovery methods for the collocation 
case)

> Enrich Disaster Recovery REST API with new methods for the collocation case
> ---
>
> Key: IGNITE-24230
> URL: https://issues.apache.org/jira/browse/IGNITE-24230
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Mirza Aliev
>Priority: Major
>  Labels: ignite-3
>
> h3. Motivation 
> Once we revise DisasterRecoveryManager API in alignment with Colocation track 
> https://issues.apache.org/jira/browse/IGNITE-24330 and implement all related 
> tickets, so DisasterRecoveryManager API is improved with new methods that 
> work with zones' partitions, 
> we need to implement REST API for all new methods. See 
> {{DisasterRecoveryController}}
> h3. Definition of done 
> *  REST API of DisasterRecoveryManager is improved with new methods that were 
> implemented for zones' partitions.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (IGNITE-24323) Calcite engine. Query execution inside UDF leads to deadlock

2025-01-29 Thread Aleksey Plekhanov (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-24323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922026#comment-17922026
 ] 

Aleksey Plekhanov commented on IGNITE-24323:


Implementation details:
Regular {{IgniteThreadPoolExecutor}} is used instead of striped pool executor. 
But with custom {{BlockingQueue}}. This blocking queue implemented as wrapper 
over the new data structure {{QueryTasksQueue}}. On {{poll}} from this queue 
query is blocked by ID and no more task with the same query ID can be polled 
until query is unblocked explicitely. Query is unblocked in {{finally}} block 
after the task completes execution. So no tasks can be executed concurrently 
for the same query and we still provide initial tasks ordering in scope of one 
query.   

> Calcite engine. Query execution inside UDF leads to deadlock
> 
>
> Key: IGNITE-24323
> URL: https://issues.apache.org/jira/browse/IGNITE-24323
> Project: Ignite
>  Issue Type: Bug
>Reporter: Aleksey Plekhanov
>Assignee: Aleksey Plekhanov
>Priority: Major
>  Labels: calcite, ise
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> We have striped executor to execute query tasks. Thread for query/fragment on 
> each node is fixed and defined by hash of query ID and fragment ID. If user 
> created user defined function and try to execute some query inside it, it can 
> lead to deadlock, since task execution thread blocked by new synchronous 
> query execution request and the new query can be assigned to the same thread.
> For H2-based SQL engine threads are no binded to the query and there is no 
> such problem.
> For Calcite-based SQL engine we use non thread safe datastructures in tasks. 
> To workaround this problem we can use regular IgniteThreadPoolExecutor, but 
> need to ensure that there are no tasks concurrently executing for the same 
> query/fragment.
> Also, perhaps this approach can affect performance and should be enabled by 
> some property explicitly only for users that require this mode. 
>   



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (IGNITE-24307) Query plan does not use the most optimal table index

2025-01-29 Thread Iurii Gerzhedovich (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-24307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922027#comment-17922027
 ] 

Iurii Gerzhedovich commented on IGNITE-24307:
-

[~vdmitrienko] I can just say that no difference for this case in terms of the 
time of execution. You can try to investigate this and find bounds when it will 
be significant

> Query plan does not use the most optimal table index
> 
>
> Key: IGNITE-24307
> URL: https://issues.apache.org/jira/browse/IGNITE-24307
> Project: Ignite
>  Issue Type: Bug
>  Components: sql
>Affects Versions: 3.0.0-beta1
> Environment: 3 nodes (each node is CMG, each node 
> {color:#067d17}"-Xms4096m"{color}, {color:#067d17}"-Xmx4096m"{color}), each 
> on separate host. Each host vCPU: 4, Memory: 32GB.
>Reporter: Vladimir Dmitrienko
>Priority: Major
>  Labels: ignite-3
>
> *Steps to reproduce:*
> Execute the following SQL script:
> {code:java}
> CREATE TABLE IF NOT EXISTS test_table (id int primary key, field_1 int, 
> field_2 int, field_3 int);
> CREATE INDEX IF NOT EXISTS one_field_index ON test_table (field_1 ASC);
> CREATE INDEX IF NOT EXISTS three_field_index ON test_table (field_1 ASC, 
> field_2 ASC, field_3 ASC);
> INSERT INTO test_table VALUES (1, 2, 3, 4);
> EXPLAIN PLAN FOR SELECT field_1, field_2, field_3 FROM test_table WHERE 
> field_1 = 2 AND field_2 = 3 AND field_3 = 4;{code}
>  
> *Actual result:*
> Execution plan uses the {{ONE_FIELD_INDEX}} even if the more appropriate 
> {{THREE_FIELD_INDEX}} exists;
>  
> {code:java}
> PLAN  
>   
>   
>   
>   
> Exchange(distribution=[single]): rowcount = 1.0, cumulative cost = IgniteCost 
> [rowCount=2.0, cpu=2.0, memory=1.0, io=1.0, network=13.0], id = 362   
>   
>   
>  
> IndexScan(table=[[PUBLIC, TEST_TABLE]], tableId=[35], 
> index=[ONE_FIELD_INDEX], type=[SORTED], searchBounds=[[ExactBounds 
> [bound=2]]], filters=[AND(=($t0, 2), =($t1, 3), =($t2, 4))], 
> requiredColumns=[{1, 2, 3}], collation=[[1]]): rowcount = 1.0, cumulative 
> cost = IgniteCost [rowCount=1.0, cpu=1.0, memory=1.0, io=1.0, network=1.0], 
> id = 361 {code}
>  
> *Expected result:* 
> {{THREE_FIELD_INDEX}} used. 
> *Note:*
> if id column added to the query, {{THREE_FIELD_INDEX}} is used.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-22621) Implement transacation storage for the zone replica listener

2025-01-29 Thread Alexander Lapin (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-22621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Lapin updated IGNITE-22621:
-
Ignite Flags:   (was: Docs Required,Release Notes Required)

> Implement transacation storage for the zone replica listener
> 
>
> Key: IGNITE-22621
> URL: https://issues.apache.org/jira/browse/IGNITE-22621
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Kirill Gusakov
>Priority: Major
>  Labels: ignite-3
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-23749) CancelServiceTask permissions

2025-01-29 Thread Nikita Amelchev (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-23749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikita Amelchev updated IGNITE-23749:
-
Fix Version/s: 2.18

> CancelServiceTask permissions
> -
>
> Key: IGNITE-23749
> URL: https://issues.apache.org/jira/browse/IGNITE-23749
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Egor Fomin
>Assignee: Egor Fomin
>Priority: Minor
> Fix For: 2.18
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-23749) CancelServiceTask permissions

2025-01-29 Thread Nikita Amelchev (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-23749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikita Amelchev updated IGNITE-23749:
-
Ignite Flags: Release Notes Required  (was: Docs Required,Release Notes 
Required)

> CancelServiceTask permissions
> -
>
> Key: IGNITE-23749
> URL: https://issues.apache.org/jira/browse/IGNITE-23749
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Egor Fomin
>Assignee: Egor Fomin
>Priority: Minor
> Fix For: 2.18
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-23749) CancelServiceTask permissions

2025-01-29 Thread Nikita Amelchev (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-23749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikita Amelchev updated IGNITE-23749:
-
Release Note: Removed redundant permission check to cancel services

> CancelServiceTask permissions
> -
>
> Key: IGNITE-23749
> URL: https://issues.apache.org/jira/browse/IGNITE-23749
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Egor Fomin
>Assignee: Egor Fomin
>Priority: Minor
> Fix For: 2.18
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-23749) CancelServiceTask permissions

2025-01-29 Thread Nikita Amelchev (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-23749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikita Amelchev updated IGNITE-23749:
-
Description: Remove redundant ADMIN_OPS permission check to cancel services

> CancelServiceTask permissions
> -
>
> Key: IGNITE-23749
> URL: https://issues.apache.org/jira/browse/IGNITE-23749
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Egor Fomin
>Assignee: Egor Fomin
>Priority: Minor
> Fix For: 2.18
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Remove redundant ADMIN_OPS permission check to cancel services



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (IGNITE-24323) Calcite engine. Query execution inside UDF leads to deadlock

2025-01-29 Thread Aleksey Plekhanov (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-24323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922263#comment-17922263
 ] 

Aleksey Plekhanov commented on IGNITE-24323:


Green TC bot visa: 
https://tcbot2.sbt-ignite-dev.ru/pr.html?serverId=apache&suiteId=IgniteTests24Java8_RunAll&branchForTc=pull/11833/head&action=Latest
Something broken in TC bot and "Comment JIRA" is not working now.

> Calcite engine. Query execution inside UDF leads to deadlock
> 
>
> Key: IGNITE-24323
> URL: https://issues.apache.org/jira/browse/IGNITE-24323
> Project: Ignite
>  Issue Type: Bug
>Reporter: Aleksey Plekhanov
>Assignee: Aleksey Plekhanov
>Priority: Major
>  Labels: calcite, ise
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> We have striped executor to execute query tasks. Thread for query/fragment on 
> each node is fixed and defined by hash of query ID and fragment ID. If user 
> created user defined function and try to execute some query inside it, it can 
> lead to deadlock, since task execution thread blocked by new synchronous 
> query execution request and the new query can be assigned to the same thread.
> For H2-based SQL engine threads are no binded to the query and there is no 
> such problem.
> For Calcite-based SQL engine we use non thread safe datastructures in tasks. 
> To workaround this problem we can use regular IgniteThreadPoolExecutor, but 
> need to ensure that there are no tasks concurrently executing for the same 
> query/fragment.
> Also, perhaps this approach can affect performance and should be enabled by 
> some property explicitly only for users that require this mode. 
>   



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (IGNITE-24139) Update Ignite dependency jetty

2025-01-29 Thread Julia Bakulina (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-24139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922270#comment-17922270
 ] 

Julia Bakulina commented on IGNITE-24139:
-

[TCbot2 green 
visa|https://tcbot2.sbt-ignite-dev.ru/pr.html?serverId=apache&suiteId=IgniteTests24Java8_RunAll&branchForTc=pull/11830/head&action=Latest]
 

> Update Ignite dependency jetty
> --
>
> Key: IGNITE-24139
> URL: https://issues.apache.org/jira/browse/IGNITE-24139
> Project: Ignite
>  Issue Type: Task
>Reporter: Julia Bakulina
>Assignee: Julia Bakulina
>Priority: Minor
>  Labels: ise
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Update Ignite jetty dependency to 11 version



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24344) Rename types related to transaction state storages

2025-01-29 Thread Roman Puchkovskiy (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roman Puchkovskiy updated IGNITE-24344:
---
Description: 
Currently, there is per-table TxStateTableStorage and per-partition (where 
partitions are table partitions) TxStateStorage.

We are going to switch to zones, so 'table' term should go. TxStateTableStorage 
is not specific to tables, it can serve as zone-wide storage, but we need to 
rename it. The idea is that the new name is not specific about tables or zones; 
TxStateStorage seems to be ok in this regard.

This means that TxStateStorage (that now corresponds to one partition) needs to 
be renamed, TxStatePartitionStorage seems to be a good name. It also aligns 
well with MvPartitionStorage which we already have.

{*}Definition of done{*}: TxStateStorage gets renamed to 
TxStatePartitionStorage, and TxStateTableStorage to TxStateStorage.

  was:Currently, there is per-table TxState


> Rename types related to transaction state storages
> --
>
> Key: IGNITE-24344
> URL: https://issues.apache.org/jira/browse/IGNITE-24344
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Roman Puchkovskiy
>Assignee: Roman Puchkovskiy
>Priority: Major
>  Labels: ignite-3
>
> Currently, there is per-table TxStateTableStorage and per-partition (where 
> partitions are table partitions) TxStateStorage.
> We are going to switch to zones, so 'table' term should go. 
> TxStateTableStorage is not specific to tables, it can serve as zone-wide 
> storage, but we need to rename it. The idea is that the new name is not 
> specific about tables or zones; TxStateStorage seems to be ok in this regard.
> This means that TxStateStorage (that now corresponds to one partition) needs 
> to be renamed, TxStatePartitionStorage seems to be a good name. It also 
> aligns well with MvPartitionStorage which we already have.
> {*}Definition of done{*}: TxStateStorage gets renamed to 
> TxStatePartitionStorage, and TxStateTableStorage to TxStateStorage.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24344) Rename types related to transaction state storages

2025-01-29 Thread Roman Puchkovskiy (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roman Puchkovskiy updated IGNITE-24344:
---
Epic Link: IGNITE-22115

> Rename types related to transaction state storages
> --
>
> Key: IGNITE-24344
> URL: https://issues.apache.org/jira/browse/IGNITE-24344
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Roman Puchkovskiy
>Assignee: Roman Puchkovskiy
>Priority: Major
>  Labels: ignite-3
>
> Currently, there is per-table TxStateTableStorage and per-partition (where 
> partitions are table partitions) TxStateStorage.
> We are going to switch to zones, so 'table' term should go. 
> TxStateTableStorage is not specific to tables, it can serve as zone-wide 
> storage, but we need to rename it. The idea is that the new name is not 
> specific about tables or zones; TxStateStorage seems to be ok in this regard.
> This means that TxStateStorage (that now corresponds to one partition) needs 
> to be renamed, TxStatePartitionStorage seems to be a good name. It also 
> aligns well with MvPartitionStorage which we already have.
> {*}Definition of done{*}: TxStateStorage gets renamed to 
> TxStatePartitionStorage, and TxStateTableStorage to TxStateStorage.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (IGNITE-24345) Defer zone destruction after drop

2025-01-29 Thread Roman Puchkovskiy (Jira)
Roman Puchkovskiy created IGNITE-24345:
--

 Summary: Defer zone destruction after drop
 Key: IGNITE-24345
 URL: https://issues.apache.org/jira/browse/IGNITE-24345
 Project: Ignite
  Issue Type: Improvement
Reporter: Roman Puchkovskiy






--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (IGNITE-24346) Do not destroy tx state storage while its content might be needed for write intent resolution

2025-01-29 Thread Roman Puchkovskiy (Jira)
Roman Puchkovskiy created IGNITE-24346:
--

 Summary: Do not destroy tx state storage while its content might 
be needed for write intent resolution
 Key: IGNITE-24346
 URL: https://issues.apache.org/jira/browse/IGNITE-24346
 Project: Ignite
  Issue Type: Improvement
Reporter: Roman Puchkovskiy






--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24346) Do not destroy tx state storage while its content might be needed for write intent resolution

2025-01-29 Thread Roman Puchkovskiy (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roman Puchkovskiy updated IGNITE-24346:
---
Description: 
Tx finish records stored in tx state storage have to be retained until all 
write intents corresponding to the finish record's transaction are cleaned up 
(either converted to normal tuple versions or removed). It is possible that, in 
the current implementation, we might erase such records too early when 
destroying a dropped table.

The suspected scenario is:
 # tx1 is started, it writes to tables A and B, the it gets committed, commit 
partition is a partition of A
 # Cleanup of tx1 is deferred for some reason (like high amount of work the 
cleanuper does)
 # A is dropped
 # After the moment of A's drop sinks under LWM, storages of A's partitions get 
destroyed (including the commit partition of tx1)
 # Cleanup of tx1 is attempted, but the tx state storage is destroyed, so the 
cleanup cannot proceed  -> write intents of tx1 will remain forever unresolved

We need to check whether this is a possible scenario and, if it's possible, 
make sure that we keep tx state storages available for cleanup activities until 
all their transactions get cleaned up, even if this means deferring table 
partition destruction.

For per-zone tx state storages to which we are currently switching (see 
IGNITE-22621), nothing changes: a transaction might span multiple zones and 
then the zone hosting the commit partition might have been dropped and 
destroyed (IGNITE-24345).

> Do not destroy tx state storage while its content might be needed for write 
> intent resolution
> -
>
> Key: IGNITE-24346
> URL: https://issues.apache.org/jira/browse/IGNITE-24346
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Roman Puchkovskiy
>Priority: Major
>  Labels: ignite-3
>
> Tx finish records stored in tx state storage have to be retained until all 
> write intents corresponding to the finish record's transaction are cleaned up 
> (either converted to normal tuple versions or removed). It is possible that, 
> in the current implementation, we might erase such records too early when 
> destroying a dropped table.
> The suspected scenario is:
>  # tx1 is started, it writes to tables A and B, the it gets committed, commit 
> partition is a partition of A
>  # Cleanup of tx1 is deferred for some reason (like high amount of work the 
> cleanuper does)
>  # A is dropped
>  # After the moment of A's drop sinks under LWM, storages of A's partitions 
> get destroyed (including the commit partition of tx1)
>  # Cleanup of tx1 is attempted, but the tx state storage is destroyed, so the 
> cleanup cannot proceed  -> write intents of tx1 will remain forever unresolved
> We need to check whether this is a possible scenario and, if it's possible, 
> make sure that we keep tx state storages available for cleanup activities 
> until all their transactions get cleaned up, even if this means deferring 
> table partition destruction.
> For per-zone tx state storages to which we are currently switching (see 
> IGNITE-22621), nothing changes: a transaction might span multiple zones and 
> then the zone hosting the commit partition might have been dropped and 
> destroyed (IGNITE-24345).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24345) Defer zone destruction after drop

2025-01-29 Thread Roman Puchkovskiy (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roman Puchkovskiy updated IGNITE-24345:
---
Description: 
When a table is dropped, table partition storages are not destroyed 
immediately. Instead, this is deferred until table drop moment sinks below the 
LWM.

As partitions will become per-zone (instead of per-table), we need to implement 
the same logic for zones. That is, zone partitions need to be destroyed only 
after zone drop moment sinks below the LWM.

> Defer zone destruction after drop
> -
>
> Key: IGNITE-24345
> URL: https://issues.apache.org/jira/browse/IGNITE-24345
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Roman Puchkovskiy
>Priority: Major
>  Labels: ignite-3
>
> When a table is dropped, table partition storages are not destroyed 
> immediately. Instead, this is deferred until table drop moment sinks below 
> the LWM.
> As partitions will become per-zone (instead of per-table), we need to 
> implement the same logic for zones. That is, zone partitions need to be 
> destroyed only after zone drop moment sinks below the LWM.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24344) Rename types related to transaction meta storages

2025-01-29 Thread Roman Puchkovskiy (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roman Puchkovskiy updated IGNITE-24344:
---
Description: Currently, there is per-table TxState

> Rename types related to transaction meta storages
> -
>
> Key: IGNITE-24344
> URL: https://issues.apache.org/jira/browse/IGNITE-24344
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Roman Puchkovskiy
>Assignee: Roman Puchkovskiy
>Priority: Major
>  Labels: ignite-3
>
> Currently, there is per-table TxState



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (IGNITE-24344) Rename types related to transaction meta storages

2025-01-29 Thread Roman Puchkovskiy (Jira)
Roman Puchkovskiy created IGNITE-24344:
--

 Summary: Rename types related to transaction meta storages
 Key: IGNITE-24344
 URL: https://issues.apache.org/jira/browse/IGNITE-24344
 Project: Ignite
  Issue Type: Improvement
Reporter: Roman Puchkovskiy
Assignee: Roman Puchkovskiy






--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24344) Rename types related to transaction state storages

2025-01-29 Thread Roman Puchkovskiy (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roman Puchkovskiy updated IGNITE-24344:
---
Summary: Rename types related to transaction state storages  (was: Rename 
types related to transaction meta storages)

> Rename types related to transaction state storages
> --
>
> Key: IGNITE-24344
> URL: https://issues.apache.org/jira/browse/IGNITE-24344
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Roman Puchkovskiy
>Assignee: Roman Puchkovskiy
>Priority: Major
>  Labels: ignite-3
>
> Currently, there is per-table TxState



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (IGNITE-24347) Table drop should not cause zone partitions destruction

2025-01-29 Thread Roman Puchkovskiy (Jira)
Roman Puchkovskiy created IGNITE-24347:
--

 Summary: Table drop should not cause zone partitions destruction
 Key: IGNITE-24347
 URL: https://issues.apache.org/jira/browse/IGNITE-24347
 Project: Ignite
  Issue Type: Improvement
Reporter: Roman Puchkovskiy






--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24345) Defer zone destruction after drop

2025-01-29 Thread Roman Puchkovskiy (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roman Puchkovskiy updated IGNITE-24345:
---
Description: 
When a table is dropped, table partition storages are not destroyed 
immediately. Instead, this is deferred until table drop moment sinks below the 
LWM.

As partitions will become per-zone (instead of per-table), we need to implement 
the same logic for zones. That is, zone partitions need to be destroyed only 
after zone drop moment sinks below the LWM.

IGNITE-24345 defines that table drop should stop causing eventual storage 
destruction.

  was:
When a table is dropped, table partition storages are not destroyed 
immediately. Instead, this is deferred until table drop moment sinks below the 
LWM.

As partitions will become per-zone (instead of per-table), we need to implement 
the same logic for zones. That is, zone partitions need to be destroyed only 
after zone drop moment sinks below the LWM.


> Defer zone destruction after drop
> -
>
> Key: IGNITE-24345
> URL: https://issues.apache.org/jira/browse/IGNITE-24345
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Roman Puchkovskiy
>Priority: Major
>  Labels: ignite-3
>
> When a table is dropped, table partition storages are not destroyed 
> immediately. Instead, this is deferred until table drop moment sinks below 
> the LWM.
> As partitions will become per-zone (instead of per-table), we need to 
> implement the same logic for zones. That is, zone partitions need to be 
> destroyed only after zone drop moment sinks below the LWM.
> IGNITE-24345 defines that table drop should stop causing eventual storage 
> destruction.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (IGNITE-24347) Table drop should not cause zone partitions destruction

2025-01-29 Thread Roman Puchkovskiy (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roman Puchkovskiy resolved IGNITE-24347.

Resolution: Invalid

The issue is invalid: as MV partition storages will remain per-table for now, 
table drop still should cause an eventual destruction of such storages 
belonging to the dropped table.

> Table drop should not cause zone partitions destruction
> ---
>
> Key: IGNITE-24347
> URL: https://issues.apache.org/jira/browse/IGNITE-24347
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Roman Puchkovskiy
>Priority: Major
>  Labels: ignite-3
>
> As in IGNITE-24345 partition destruction will be moved to zone-related 
> events, table drop should not cause partition destruction anymore (even in 
> its deferred variety).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24347) Table drop should not cause zone partitions destruction

2025-01-29 Thread Roman Puchkovskiy (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roman Puchkovskiy updated IGNITE-24347:
---
Description: As in IGNITE-24345 partition destruction will be moved to 
zone-related events, table drop should not cause partition destruction anymore 
(even in its deferred variety).

> Table drop should not cause zone partitions destruction
> ---
>
> Key: IGNITE-24347
> URL: https://issues.apache.org/jira/browse/IGNITE-24347
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Roman Puchkovskiy
>Priority: Major
>  Labels: ignite-3
>
> As in IGNITE-24345 partition destruction will be moved to zone-related 
> events, table drop should not cause partition destruction anymore (even in 
> its deferred variety).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24345) Defer zone destruction after drop

2025-01-29 Thread Roman Puchkovskiy (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roman Puchkovskiy updated IGNITE-24345:
---
Description: 
When a table is dropped, table partition storages are not destroyed 
immediately. Instead, this is deferred until table drop moment sinks below the 
LWM.

As partitions will become per-zone (instead of per-table), we need to implement 
the same logic for zones. That is, zone partitions need to be destroyed only 
after zone drop moment sinks below the LWM.

MV partition storages will remain per-table (at least in this Epic), so they 
will still need to be destroyed eventually as a result of table drop.

  was:
When a table is dropped, table partition storages are not destroyed 
immediately. Instead, this is deferred until table drop moment sinks below the 
LWM.

As partitions will become per-zone (instead of per-table), we need to implement 
the same logic for zones. That is, zone partitions need to be destroyed only 
after zone drop moment sinks below the LWM.

IGNITE-24345 defines that table drop should stop causing eventual storage 
destruction.


> Defer zone destruction after drop
> -
>
> Key: IGNITE-24345
> URL: https://issues.apache.org/jira/browse/IGNITE-24345
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Roman Puchkovskiy
>Priority: Major
>  Labels: ignite-3
>
> When a table is dropped, table partition storages are not destroyed 
> immediately. Instead, this is deferred until table drop moment sinks below 
> the LWM.
> As partitions will become per-zone (instead of per-table), we need to 
> implement the same logic for zones. That is, zone partitions need to be 
> destroyed only after zone drop moment sinks below the LWM.
> MV partition storages will remain per-table (at least in this Epic), so they 
> will still need to be destroyed eventually as a result of table drop.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24336) Calcite. Add support of user-defined table functions.

2025-01-29 Thread Vladimir Steshin (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-24336:
--
Release Note: User-defined SQL table functions are now available with 
Calcite engine.
 Description: 
We should implement user-defined table functions. They already have an 
implementation in _Calcite_, _TableFunctionImpl_. And we also have user-defined 
scalar functions.

Table functions should be described similarly to the existing configuration of 
scalar functions, via _CacheConfiguration#setSqlFunctionClasses()_ and 
annotation _@QuerySqlTableFunction_. Or we might use another annotation. The 
annotation should describe the returned columns and their data types.

*Example:*
{code:java}
public static final class TableFunctions {
@QuerySqlFunction(tableColumnTypes = {Integer.class, String.class}, 
tableColumnNames = {"INT_COL", "STR_COL"})
public static Iterable table_fun(int i) {
return Arrays.asList(
new Object[]{i, "code_" + i},
new Object[]{i * 10, "code_" + (i * 10)}
);
}
}

ignite.getOrCreateCache(new 
CacheConfiguration("test_cache").setSqlFunctionClasses(TableFunctions.class));

sql("select STR_COL from TABLE_FUN(1) where INT_COL>1")
{code}

Table function implementation must return an _Iterable_ representing table 
rows. Each its element is an _Object[]_ or an _Iterable_ representing a table 
row. The number of row elements must match the number of the defined columns. 

*Known limitations:*
Only _Calcite_ engine will support table functions first.

  was:
We should implement user-defined table functions. They already have a 
implementation in _Calcite_, _TableFunctionImpl_. And we also have user-defined 
scalar functions.

Table functions should be described similarly to the existing configuration of 
scalar functions, via _CacheConfiguration#setSqlFunctionClasses()_ and 
annotation _@QuerySqlTableFunction_. Or we might use another annotation. The 
annotation should describe the returned columns and their data types.

*Example:*
{code:java}
public static final class TableFunctions {
@QuerySqlFunction(tableColumnTypes = {Integer.class, String.class}, 
tableColumnNames = {"INT_COL", "STR_COL"})
public static Iterable table_fun(int i) {
return Arrays.asList(
new Object[]{i, "code_" + i},
new Object[]{i * 10, "code_" + (i * 10)}
);
}
}

ignite.getOrCreateCache(new 
CacheConfiguration("test_cache").setSqlFunctionClasses(TableFunctions.class));

sql("select STR_COL from TABLE_FUN(1) where INT_COL>1")
{code}

Table function implementation must return an _Iterable_ representing table 
rows. Each its element is an _Object[]_ or an _Iterable_ representing a table 
row. The number of row elements must match the number of the defined columns. 

*Known limitations:*
Only _Calcite_ engine will support table functions first.


> Calcite. Add support of user-defined table functions.
> -
>
> Key: IGNITE-24336
> URL: https://issues.apache.org/jira/browse/IGNITE-24336
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: calcite, ignite-2, ise, sql
>
> We should implement user-defined table functions. They already have an 
> implementation in _Calcite_, _TableFunctionImpl_. And we also have 
> user-defined scalar functions.
> Table functions should be described similarly to the existing configuration 
> of scalar functions, via _CacheConfiguration#setSqlFunctionClasses()_ and 
> annotation _@QuerySqlTableFunction_. Or we might use another annotation. The 
> annotation should describe the returned columns and their data types.
> *Example:*
> {code:java}
> public static final class TableFunctions {
> @QuerySqlFunction(tableColumnTypes = {Integer.class, String.class}, 
> tableColumnNames = {"INT_COL", "STR_COL"})
> public static Iterable table_fun(int i) {
> return Arrays.asList(
> new Object[]{i, "code_" + i},
> new Object[]{i * 10, "code_" + (i * 10)}
> );
> }
> }
> ignite.getOrCreateCache(new 
> CacheConfiguration("test_cache").setSqlFunctionClasses(TableFunctions.class));
> sql("select STR_COL from TABLE_FUN(1) where INT_COL>1")
> {code}
> Table function implementation must return an _Iterable_ representing table 
> rows. Each its element is an _Object[]_ or an _Iterable_ representing a table 
> row. The number of row elements must match the number of the defined columns. 
> *Known limitations:*
> Only _Calcite_ engine will support table functions first.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24336) Calcite. Add support of user-defined table functions.

2025-01-29 Thread Vladimir Steshin (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-24336:
--
Labels: calcite ignite-2 ise  (was: calcite ignite-2 ise sql)

> Calcite. Add support of user-defined table functions.
> -
>
> Key: IGNITE-24336
> URL: https://issues.apache.org/jira/browse/IGNITE-24336
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: calcite, ignite-2, ise
>
> We should implement user-defined table functions. They already have an 
> implementation in _Calcite_, _TableFunctionImpl_. And we also have 
> user-defined scalar functions.
> Table functions should be described similarly to the existing configuration 
> of scalar functions, via _CacheConfiguration#setSqlFunctionClasses()_ and 
> annotation _@QuerySqlTableFunction_. Or we might use another annotation. The 
> annotation should describe the returned columns and their data types.
> *Example:*
> {code:java}
> public static final class TableFunctions {
> @QuerySqlFunction(tableColumnTypes = {Integer.class, String.class}, 
> tableColumnNames = {"INT_COL", "STR_COL"})
> public static Iterable table_fun(int i) {
> return Arrays.asList(
> new Object[]{i, "code_" + i},
> new Object[]{i * 10, "code_" + (i * 10)}
> );
> }
> }
> ignite.getOrCreateCache(new 
> CacheConfiguration("test_cache").setSqlFunctionClasses(TableFunctions.class));
> sql("select STR_COL from TABLE_FUN(1) where INT_COL>1")
> {code}
> Table function implementation must return an _Iterable_ representing table 
> rows. Each its element is an _Object[]_ or an _Iterable_ representing a table 
> row. The number of row elements must match the number of the defined columns. 
> *Known limitations:*
> Only _Calcite_ engine will support table functions first.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-22621) Implement transacation storage for the zone replica listener

2025-01-29 Thread Alexander Lapin (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-22621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Lapin updated IGNITE-22621:
-
Attachment: Screenshot 2025-01-29 at 10.29.28 AM.png

> Implement transacation storage for the zone replica listener
> 
>
> Key: IGNITE-22621
> URL: https://issues.apache.org/jira/browse/IGNITE-22621
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Kirill Gusakov
>Priority: Major
>  Labels: ignite-3
> Attachments: Screenshot 2025-01-29 at 10.29.28 AM.png
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Assigned] (IGNITE-22621) Implement transacation storage for the zone replica listener

2025-01-29 Thread Alexander Lapin (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-22621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Lapin reassigned IGNITE-22621:


Assignee: Roman Puchkovskiy

> Implement transacation storage for the zone replica listener
> 
>
> Key: IGNITE-22621
> URL: https://issues.apache.org/jira/browse/IGNITE-22621
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Kirill Gusakov
>Assignee: Roman Puchkovskiy
>Priority: Major
>  Labels: ignite-3
> Attachments: Screenshot 2025-01-29 at 10.29.28 AM.png
>
>
> h3. Motivation
> Few words about general context. Tx Finish flow is a crucial step in AI3 Tx 
> Protocol. Generally it consists of two phases along with some prerequisites.
> 0. Prerequisites: schema validation, tx state check.
> 1. Txn state switch phase. Atomic transaction commit/rollback implemented 
> through replicating persistent txn state(COMMITTED/ABORTED) to 
> commitPartition. Once it's done, transaction is considered as finished.
> 2. Cleanup phase. Releasing locks, converting writeIntents to common values 
> or removing them depending on whether it's commit or abort.
> h3. !Screenshot 2025-01-29 at 10.29.28 AM.png! Definition of Done
>  * Move processing TxFinishReplicaRequest from 
> PartitionReplicaListener#processTxFinishAction to 
> ZonePartitionRaftListener#processTxFinishAction by introducing corresponding 
> method. In order to do that, it's required to add some dependencies to 
> ZonePartitionRaftListener:
>  * 
>  ** SchemaCompatibilityValidator
>  ** TxStateStorage // Adding the storage isn't trivial.
>  **  TxManager
>  **  RaftCommandRunner // In order to run TxFinishCommand



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-22621) Implement transacation storage for the zone replica listener

2025-01-29 Thread Alexander Lapin (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-22621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Lapin updated IGNITE-22621:
-
Description: 
h3. Motivation

Few words about general context. Tx Finish flow is a crucial step in AI3 Tx 
Protocol. Generally it consists of two phases along with some prerequisites.
0. Prerequisites: schema validation, tx state check.
1. Txn state switch phase. Atomic transaction commit/rollback implemented 
through replicating persistent txn state(COMMITTED/ABORTED) to commitPartition. 
Once it's done, transaction is considered as finished.
2. Cleanup phase. Releasing locks, converting writeIntents to common values or 
removing them depending on whether it's commit or abort.
h3. !Screenshot 2025-01-29 at 10.29.28 AM.png! Definition of Done
 * Move processing TxFinishReplicaRequest from 
PartitionReplicaListener#processTxFinishAction to 
ZonePartitionRaftListener#processTxFinishAction by introducing corresponding 
method. In order to do that, it's required to add some dependencies to 
ZonePartitionRaftListener:

 * 
 ** SchemaCompatibilityValidator
 ** TxStateStorage // Adding the storage isn't trivial.
 **  TxManager
 **  RaftCommandRunner // In order to run TxFinishCommand

> Implement transacation storage for the zone replica listener
> 
>
> Key: IGNITE-22621
> URL: https://issues.apache.org/jira/browse/IGNITE-22621
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Kirill Gusakov
>Priority: Major
>  Labels: ignite-3
> Attachments: Screenshot 2025-01-29 at 10.29.28 AM.png
>
>
> h3. Motivation
> Few words about general context. Tx Finish flow is a crucial step in AI3 Tx 
> Protocol. Generally it consists of two phases along with some prerequisites.
> 0. Prerequisites: schema validation, tx state check.
> 1. Txn state switch phase. Atomic transaction commit/rollback implemented 
> through replicating persistent txn state(COMMITTED/ABORTED) to 
> commitPartition. Once it's done, transaction is considered as finished.
> 2. Cleanup phase. Releasing locks, converting writeIntents to common values 
> or removing them depending on whether it's commit or abort.
> h3. !Screenshot 2025-01-29 at 10.29.28 AM.png! Definition of Done
>  * Move processing TxFinishReplicaRequest from 
> PartitionReplicaListener#processTxFinishAction to 
> ZonePartitionRaftListener#processTxFinishAction by introducing corresponding 
> method. In order to do that, it's required to add some dependencies to 
> ZonePartitionRaftListener:
>  * 
>  ** SchemaCompatibilityValidator
>  ** TxStateStorage // Adding the storage isn't trivial.
>  **  TxManager
>  **  RaftCommandRunner // In order to run TxFinishCommand



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24339) Flaky test DdlCommandHandlerExceptionHandlingTest#testZoneNotFoundOnDrop2

2025-01-29 Thread Iurii Gerzhedovich (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Iurii Gerzhedovich updated IGNITE-24339:

Description: 
The test 
org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandlerExceptionHandlingTest#testZoneNotFoundOnDrop2
 is flaky with the following stacktrace:
 
{code:java}
java.lang.AssertionError: java.util.concurrent.ExecutionException: 
java.lang.IllegalArgumentException: Time is out of bounds: 0   at 
org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:78)
 at 
org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:35)
 at org.hamcrest.TypeSafeMatcher.matches(TypeSafeMatcher.java:83) at 
org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:10) at 
org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6) at 
org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandlerExceptionHandlingTest.testZoneNotFoundOnDrop2(DdlCommandHandlerExceptionHandlingTest.java:111)
 at java.base/java.lang.reflect.Method.invoke(Method.java:566) at 
java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at 
java.base/java.util.ArrayList.forEach(ArrayList.java:1541) Caused by: 
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: 
Time is out of bounds: 0 at 
java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
 at 
java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2028)
 at 
org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:74)
 ... 8 more Caused by: java.lang.IllegalArgumentException: Time is out of 
bounds: 0 at 
org.apache.ignite.internal.hlc.HybridTimestamp.(HybridTimestamp.java:111) 
at 
org.apache.ignite.internal.hlc.HybridTimestamp.hybridTimestamp(HybridTimestamp.java:131)
 at 
org.apache.ignite.internal.catalog.commands.CatalogUtils.clusterWideEnsuredActivationTimestamp(CatalogUtils.java:537)
 at 
org.apache.ignite.internal.catalog.CatalogManagerImpl.calcClusterWideEnsureActivationTime(CatalogManagerImpl.java:376)
 at 
org.apache.ignite.internal.catalog.CatalogManagerImpl.awaitVersionActivation(CatalogManagerImpl.java:370)
 at 
org.apache.ignite.internal.catalog.CatalogManagerImpl.awaitVersionActivation(CatalogManagerImpl.java:364)
 at 
java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
 at 
java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2241)
 at 
org.apache.ignite.internal.catalog.CatalogManagerImpl.saveUpdateAndWaitForActivation(CatalogManagerImpl.java:314)
 at 
org.apache.ignite.internal.catalog.CatalogManagerImpl.execute(CatalogManagerImpl.java:235)
 at 
org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandler.handle(DdlCommandHandler.java:76)
 ... 4 more{code}
Initial version of Catalog uses {{0L}} as activation time, which is not valid 
value according to HybridTimestamp's constructor:
 
Let's change activationTimestamp for initial version of catalog in 
CatalogManagerImpl from 0 to 1. But we need to rework tests after that

  was:
The test 
org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandlerExceptionHandlingTest#testZoneNotFoundOnDrop2
 is flaky with the following stacktrace:
 
{code:java}
java.lang.AssertionError: java.util.concurrent.ExecutionException: 
java.lang.IllegalArgumentException: Time is out of bounds: 0   at 
org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:78)
 at 
org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:35)
 at org.hamcrest.TypeSafeMatcher.matches(TypeSafeMatcher.java:83) at 
org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:10) at 
org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6) at 
org.apache.ignite.internal.sql.engine.exec.ddl.DdlCommandHandlerExceptionHandlingTest.testZoneNotFoundOnDrop2(DdlCommandHandlerExceptionHandlingTest.java:111)
 at java.base/java.lang.reflect.Method.invoke(Method.java:566) at 
java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at 
java.base/java.util.ArrayList.forEach(ArrayList.java:1541) Caused by: 
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: 
Time is out of bounds: 0 at 
java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
 at 
java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2028)
 at 
org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.matchesSafely(CompletableFutureMatcher.java:74)
 ... 8 more Caused by: java.lang.IllegalArgumentException: Time is out of 
bounds: 0 at 
org.apache.ignite.internal.hlc.HybridTimestamp.(HybridTimestamp.java:111) 
at 
org.apa

[jira] [Updated] (IGNITE-24337) Don't throw NoSuchFileException in IgniteUtils.deleteIfExistsThrowable

2025-01-29 Thread Philipp Shergalis (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philipp Shergalis updated IGNITE-24337:
---
Description: It is definitely not correct that deleteIfExists method throws 
error if the directory doesn't exist. Checking if file exists before calling 
this method is race-prone (it could get deleted by another thread after this 
check)

> Don't throw NoSuchFileException in IgniteUtils.deleteIfExistsThrowable
> --
>
> Key: IGNITE-24337
> URL: https://issues.apache.org/jira/browse/IGNITE-24337
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Philipp Shergalis
>Assignee: Philipp Shergalis
>Priority: Major
>  Labels: ignite-3
>
> It is definitely not correct that deleteIfExists method throws error if the 
> directory doesn't exist. Checking if file exists before calling this method 
> is race-prone (it could get deleted by another thread after this check)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (IGNITE-24340) Queries permanently causing lock table overflow get retried forever

2025-01-29 Thread Roman Puchkovskiy (Jira)
Roman Puchkovskiy created IGNITE-24340:
--

 Summary: Queries permanently causing lock table overflow get 
retried forever
 Key: IGNITE-24340
 URL: https://issues.apache.org/jira/browse/IGNITE-24340
 Project: Ignite
  Issue Type: Bug
Reporter: Roman Puchkovskiy






--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-24343) Use ZonePartitionId as commitPartitionId

2025-01-29 Thread Alexander Lapin (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-24343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Lapin updated IGNITE-24343:
-
Description: 
h3. Motivation

Side note: we are not consistent in what commitPartitionId means: usually we 
assume that it's tableId + partitionIndex however in storage API, e.g. in 
PartitionDataStorage#addWrite commitPartitionId itself means partition index 
along with dedicated 
commitTableId. Below I will assume that commitPartitionId is a combination of 
table/zoneId and partitionIndex.
  * CommitPartition is a partition where where persisted txnState is stored. 
The state is atomically switched within TxFinish transaction phase.
 * CommitPartitionId is used in order to evaluate writeIntents through 
commitPartition path.
 * Write Intent(WI) resolution is a process that allows to define whether WI 
should be considered as applied (either committed or aborted) or ignored 
because corresponding transaction is in PENDING state. Simply, for WI(txId)
 ** Tx(txId).isPending() -> ignore WI (return previous value, or null if 
there's no such value).
 ** Tx(txId).isAborted() -> ignore WI (return previous value, or null if 
there's no such value).
 ** Tx(txId).isCommited() -> consider WI as a common value.
 * There are 3 different pathes of WI Resolution:
 ** Local path. Every node that participates in tx flow has volatile 
txnStateMap, thus it worth checking locally whether we already know the state 
of corresponding transaction.
 ** If local path is not applicable we will use Coordinator Path by requesting 
txn state from cooresponding coordinator.
 ** If Coordinator Path also not applicable we will use Commit Partition Path. 
Hence it's required to know where commit partition is.
 ** For more details please check 
[IEP-91|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211885498#IEP91:Transactionprotocol-ROtransactions.1]
 Worth mentioning that IEP is a bit outdated and both RO and RW transaction may 
see WI and thus do WI Resolution.

 
Seems that we are done with [taxidermy|https://en.wikipedia.org/wiki/Taxonomy].
h3. Definition of Done
 * Within collocation track we do store zoneId + partitionIndex as 
commitPartitionId instead of tablerId + partitionIndex.
 * WI resolution works on top of ZonePartitionId as commitPartitionId.

h3. Implementation Notes
 * As an entry point we may start with PartitionDataStorage#addWrite. Basically 
depending on whether it's collocation track or common one it's possible to 
propagate either zoneId or tableId as int without updating internal storage 
structure.
 * That means that UpdateCommand.tablePartitionId should be adjusted or 
extended with zonePartitionId, that required to adjust 
ReadWriteReplicaRequest.commitPartitionId in a similar manner which on its turn 
requires to adjust InternalTransaction.commitPartition. Long story short, we 
will find ourselves in methods like 
org.apache.ignite.internal.sql.engine.prepare.IgniteRelShuttle#enlist(org.apache.ignite.internal.sql.engine.rel.SourceAwareIgniteRel)
 where we retrieve tableId from relation 
int tableId = rel.getTable().unwrap(IgniteTable.class).id(); Seems that we have 
two options here, either propagate zoneId from the very beginning of the stack 
or add Catalog dependency to ExecutionService and similar components in order 
to convert tableId to zoneId in idempotent manner. I mean that in theory it's 
possible to move tables between zones, thus it's required to be consistent in 
defining zoneId by tableId, e.g. by requesting catalog on tx start time.
 * On the other hand ReadResult has int commitTableId that should be renamed to 
commitZoneId or some generalized option. 
 * Corresponding changes in WI flow should also be implemented.

  was:
h3. Motivation

Side note: we are not consistent in what commitPartitionId means: usually we 
assume that it's tableId + partitionIndex however in storage API, e.g. in 
PartitionDataStorage#addWrite commitPartitionId itself means partition index 
along with dedicated 
commitTableId. Below I will assume that commitPartitionId is a combination of 
table/zoneId and partitionIndex.
  * CommitPartition is a partition where where persisted txnState is stored. 
The state is atomically switched within TxFinish transaction phase.
 * CommitPartitionId is used in order to evaluate writeIntents through 
commitPartition path.
 * Write Intent(WI) resolution is a process that allows to define whether WI 
should be considered as applied (either committed or aborted) or ignored 
because corresponding transaction is in PENDING state. Simply, for WI(txId)
 ** Tx(txId).isPending() -> ignore WI (return previous value, or null if 
there's no such value).
 ** Tx(txId).isAborted() -> ignore WI (return previous value, or null if 
there's no such value).
 ** Tx(txId).isCommited() -> consider WI as a common value.
 * There are 3 different pathes of WI Resolution:
 ** Local pa