[ https://issues.apache.org/jira/browse/CASSANDRA-21505 ]
Minal Kyada deleted comment on CASSANDRA-21505:
-----------------------------------------
was (Author: JIRAUSER311749):
RCA -
1. Reproduced the bug with the failing seed = "-570398361159823834L" in PaxosV1
and PaxosV2 both, the failure reproduced deterministically at step 387, this
helped us rule out a Paxos-specific protocol bug and hinted towards Harry
model's condition eval.
2. Traced the full mutation history (INSERT, UPDATE) for the affected partition
and validated that Paxos did the right thing.
3. Identified that s1 evaluates to empty bytes and since it Double, it returns
null in this case. (meaningless emptiness, leftover from thrift)
Debugging discoveries -
BUG 1: the model's extract method normalizes the stored row value to null
because we apply meaningless emptiness but we don't do the same while binding
the variable.
BUG 2: Now we apply meaningless emptiness to everything, but another query
failed which happens to have boolean column type.
Next Steps-
1. Confirm the boolean failure scenario
2. Re-run as a repeat job (multiple seeds) to verify no further issues.
> PaxosV1MultiNodeTableWalkTest Failure - CAS on static column w/empty bytes
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-21505
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21505
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Feature/Lightweight Transactions
> Reporter: C. Scott Andreas
> Priority: Normal
>
> {{PaxosV1MultiNodeTableWalkTest}} fails deterministically at a {{UPDATE ...
> IF <double static column> = <empty bytes>}} CAS. The Harry model predicts the
> CAS does not apply (expecting a failed-CAS result of {{{}[applied], s0,
> s1{}}}), while Paxos v1 applies it (returning {{[applied]}} only). The
> mismatch surfaces as an assertion in the model when it validates the returned
> rows against the expected columns.
> The disagreement is in how {{s1 = <empty bytes>}} is evaluated when {{s1}} is
> a {{double}} static column whose current value is empty bytes. This bug
> covers determining whether Paxos v1's evaluation or the Harry model's
> expectation is correct; and updating either the implementation or test.
>
> *Environment*
> * Apache Cassandra trunk at {{1332e21}}
> * {{CasMultiNodeTableWalkBase}} uses {{ReadRepairStrategy.NONE}} and
> {{paxos_variant = v1}} (this is a pure Paxos v1 path; no read repair).
> * Reproduced locally with JDK 11 (in-JVM dtest). Observed in CI on
> {{{}jvm21-dtests{}}}.
>
> *Reproducing*
> {{org.apache.cassandra.distributed.test.cql3.PaxosV1MultiNodeTableWalkTest}}
> (3-node cluster), pinned to the failing seed via the harness hook:
> // in
> PaxosV1MultiNodeTableWalkTest.preCheck(...):builder.withSeed(-570398361159823834L);
>
> then:
> {{ant test-jvm-dtest-some
> -Dtest.name=org.apache.cassandra.distributed.test.cql3.PaxosV1MultiNodeTableWalkTest
> -Dtest.methods=test}}
> It fails at step 387 (of 400) deterministically ({{{}Seed =
> -570398361159823834, Examples = 10{}}}).
>
> *Failure Observed*
> Schema:
> {{CREATE TABLE ks1.tbl (
> pk0 timeuuid, ck0 inet,
> s0 boolean static, s1 double static,
> v0 uuid, v1 frozen<set<frozen<tuple<float>>>>,
> PRIMARY KEY (pk0, ck0)
> )}}
> Shrunk operation history:
> {{INSERT INTO ks1.tbl (pk0, ck0, s0, s1, v0, v1)
> VALUES (00000000-0000-1200-b500-000000000000, '3484:...:762b', false,
> <empty bytes>, ...); -- s1 set to empty bytes
> ...}}
> {{UPDATE ks1.tbl SET s1 = <double> WHERE pk0 =
> 00000000-0000-1200-b500-000000000000
> IF s0 = false AND s1 = <empty bytes>;}}
>
> Model annotation for the CAS: "would apply false" (the model expects the
> condition to be false, so the CAS should not apply).
> Assertion (Harry model validating the returned rows):
> {{java.lang.AssertionError: actual rows do not match the schema [[applied],
> s0, s1];
> found [java.nio.HeapByteBuffer[pos=0 lim=1 cap=1]]
> at org.apache.cassandra.harry.model.ASTSingleTableModel
> (ASTSingleTableModel.java:1474)}}
>
> The model expected a failed-CAS result with three columns
> ({{{}[applied]=false{}}}, plus the current {{s0}} and {{{}s1{}}}), but the
> actual result had a single 1-byte value — the shape of a CAS that applied
> ({{{}[applied]=true{}}}).
>
> *Analysis*
> * The row's {{s1}} (a {{double}} static column) was written as empty bytes.
> The CAS condition compares {{{}s1 = <empty bytes>{}}}. Since the stored value
> equals the compared value, Paxos v1 evaluates the condition as true and
> applies the update (result {{{}[applied]{}}}, one column).
> * The Harry model evaluated the same condition as false (annotating "would
> apply false") and therefore expected the failed-CAS result shape
> {{{}[applied], s0, s1{}}}.
> * So the two disagree on the outcome of comparing a fixed-length numeric
> column to an empty-bytes value. Empty bytes for a fixed-length type
> ({{{}double{}}} is 8 bytes) is a degenerate value; the disagreement is in
> empty-value comparison semantics.
> Two possibilities:
> # Paxos v1 is correct (empty == empty, CAS applies) and the Harry model's
> condition evaluation for empty bytes on a fixed-length column is wrong (model
> bug in {{{}test/harry{}}}).
> # The Harry model is correct and Paxos v1 mis-evaluates the condition for an
> empty-bytes fixed-length column (product bug in the CAS condition path).
> Determining which requires comparing Cassandra's CQL semantics for {{= <empty
> bytes>}} on a {{{}double{}}}column against the model's evaluation, and
> checking whether the row's {{s1}} is treated as empty vs null.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
