[
https://issues.apache.org/jira/browse/CASSANDRA-17401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18044164#comment-18044164
]
Andrei Norka commented on CASSANDRA-17401:
------------------------------------------
Hi, [~paulo] [~ifesdjeen]
It appears that we've encountered the exact issue [[email protected]]
described here. During our production migration involving an {{ALTER TABLE ...
ADD COLUMN}} operation on Cassandra 4.1.3, we experienced a severe reprepare
storm — a situation where prepared statements are repeatedly invalidated and
reprepared across cluster nodes, leading to significant performance degradation.
CASSANDRA-17248, which was mentioned in the discussion, does not fully address
the root cause of this problem (we did not perform a Cassandra version
upgrade). After analyzing the behaviour, I believe the most effective long-term
solution is to stop evicting the prepared statement cache.
Can we revisit the proposed fix that implements this approach?
[https://github.com/apache/cassandra/pull/3059]
> Race condition in QueryProcessor causes just prepared statement not to be in
> the prepared statements cache
> ----------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-17401
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17401
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Messaging/Client
> Reporter: Ivan Senic
> Assignee: Jaydeepkumar Chovatia
> Priority: Normal
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The changes in the
> [QueryProcessor#prepare|https://github.com/apache/cassandra/blame/cassandra-4.0.2/src/java/org/apache/cassandra/cql3/QueryProcessor.java#L575-L638]
> method that were introduced in versions *4.0.2* and *3.11.12* can cause a
> race condition between two threads trying to concurrently prepare the same
> statement. This race condition can cause removing of a prepared statement
> from the cache, after one of the threads has received the result of the
> prepare and eventually uses MD5Digest to call
> [QueryProcessor#getPrepared|https://github.com/apache/cassandra/blame/cassandra-4.0.2/src/java/org/apache/cassandra/cql3/QueryProcessor.java#L212-L215].
> The race condition looks like this:
> * Thread1 enters _prepare_ method and resolves _safeToReturnCached_ as false
> * Thread1 executes eviction of hashes
> * Thread2 enters _prepare_ method and resolves _safeToReturnCached_ as false
> * Thread1 prepares the statement and caches it
> * Thread1 returns the result of the prepare
> * Thread2 executes eviction of hashes
> * Thread1 tries to execute the prepared statement with the received
> MD5Digest, but statement is not in the cache as it was evicted by Thread2
> I tried to reproduce this by using a Java driver, but hitting this case from
> a client side is highly unlikely and I can not simulate the needed race
> condition. However, we can easily reproduce this in Stargate (details
> [here|https://github.com/stargate/stargate/pull/1647]), as it's closer to
> QueryProcessor.
> Reproducing this in a unit test is fairly easy. I am happy to showcase this
> if needed.
> Note that the issue can occur only when safeToReturnCached is resolved as
> false.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
