This is an automated email from the ASF dual-hosted git repository. dcapwell pushed a change to branch cep-15-accord in repository https://gitbox.apache.org/repos/asf/cassandra.git
omit 101a6ceafa Accord: Test fixes omit e6585da67d ninja: remove accord_demo.txt and simulator.sh omit c4cddd93cd Cleanup Accord diff to get it ready for merge omit ba459682a9 Accord: switch back to 1g heap to keep CI stable omit 122f530085 Fix RouteIndexTest when run with -latest.yaml omit cf3a7a3343 Improve Journal table key serialization Also improve: - TxnId serialization - StoreParticipants serialization - compareUnsigned Node.Id for consistency with serialized TxnId omit f3040dc418 Fix simulator after rebase omit 118535e499 Accord: Fix issue where 3 tests would crash consistently in CI omit bd1622d5ec Various fixes and improvements Improve: - InMemoryJournal compaction should simulate files to ensure we compact the same cohorts of records (so shadowing applied correctly) - Don't update CFK deps if already known - avoid unnecessary heapification of LogGroupTimers - begin removal of Guava (mostly unused) - consider medium path on fast path delayed - add Seekables.indexOf to support faster serialization - unboxed Invariant.requires variant - AsyncChains.addCall [...] omit bc16318bfc Fix upgrade TTL tests that fail due to improved error detection in cep-15-accord omit e0dc2fc712 Fix CQLSSTableWriterConcurrencyTest omit 37ef88f62d Switch to plain Thread for AccordExecutor, and improve testing for ASYNC mode omit de9925f81a fixing post-rebase checkstyle problems omit 297a424ec4 Accord: test fixes and stability improvements * Fix short accord simulation test (seed 0x6bea128ae851724b), ConcurrentModificationException * Increase wait time during closing to avoid Unterminated threads * Increase timeouts, improve test stability * More descriptive output from CQL test * Shorten max CMS delay * Improve future handling in config service omit 0d30627155 Fix: - Accord Journal purging was disabled - remove unique_id from schema keyspace - avoid String.format in Compactor hot path - avoid string concatenation on hot path; improve segment compactor partition build efficiency - Partial compaction should update records in place to ensure truncation of discontiguous compactions do not lead to an incorrect field version being used - StoreParticipants.touches behaviour for RX was erroneously modified; should touch all no [...] omit 30a458e399 Decouple Accord serializers from C* message version so accord can make changes without impacting C* messaging omit 336404c8e3 Accord repin: turn pruning back on omit 48515fe143 Fix: - Decouple command serialization from TableMetadata version; introduce ColumnMetadata ids; gracefully handle missing TableId - DataInputPlus.readLeastSignificantBytes must truncate high bits - Fix RandomPartitioner accord serialization - Fast path stable commits must not override recovery propose/commit decisions regarding visibility of a transaction - RejectBefore must mergeMax, not merge, to ensure we maintain epoch and hlc increasing independently - Bad c [...] omit 825acec2ff Topology mixup tests should do repairs for host replacement and other required topology changes omit 7e86999d99 Implement Nemesis compactor, improve partial compaction omit f941823cae Epoch/Topology Garbage Collection omit 401009f5d4 When bootstrap fails the retry field is not populated which leads to a NPE omit 0d557af29b InterceptingExecutor.schedule returns a simulator-unsafe Future omit 6850eb19e3 Add initial Accord Overview Doc omit 7b6bc62877 Integrate RX with Cassandra Repair, so that repair safely flushes pending topology and other durability requirements. Also improve: - Introduce DurabilityService - Retire SyncPoint, replace Barrier with Write and RX - MessageType -> enum, restore GetMaxConflict - Standardise backoff logic with WaitStrategy - improve TimeoutStrategy/RetryStrategy specification strings - Forbid KX, remove directKeyDeps - Introduce UniqueTimeService, permitting hlc reservations for [...] omit 4b23a1ddde Accord fixes: - Bad ArrayBuffers recycling logic - RX must ensure dependencies TRANSITIVE_VISIBLE - Permit constructing "antiRange" that spans multiple prefixes - Not computing range CommandSummary IsDep correctly - Truncated commands that aren't shard durable could not repopulate CFK on replay, permitting recovery of another command to make an incorrect decision - NPE on async persist of RX (i.e. supplying no callback) - NPE in Builder.shouldCleanup when durabi [...] omit 6341d983ff Fix: - Only use persisted RedundantBefore for compaction - RouteIndex should index only touches, not Route - Flush RangesForEpoch updates to journal immediately, so we do not rely on the command we are processing succeeding - DurableBefore updates must wait for the epochs to be known locally - Shard.mustWitnessEpoch to support guaranteeing to witness relevant non-topology schema changes - We must propagate RedundantBefore RX shard bounds along with epoch syncs - [...] omit 39c3f786c6 Accord: Serialization Improvements omit 12657d35c3 Improve stability of tests with many epochs omit 1e264a5e52 Fix AccordMigrationTest, preclude possible races in topology propagation omit 66335fff2f (Accord) Fix recovery when one shard is committed/stable and another is preaccepted Also fix: - Topology slicing must declare whether we share/slice node ownership (to assist above) - CFK.visit removes transitive dependencies too eagerly across epoch change - apply cleanup to builder consistently, and construct the same value we would produce by purge (so that replay is idempotent) - Invoke ExecuteTxn.LocalExecute callbacks on originating CommandStore - misc other [...] omit d7d50824fd Fix flakey test org.apache.cassandra.simulator.test.ShortPaxosSimulationTest#casOnAccordSimulationTest omit cbc89c3539 Expose epoch ready state as a vtable for operators to inspect things omit fc5227120a Fix null field accounting omit eaecfb565f Fix topology loading omit 81b3b26f37 Accord: TableParamsTest is flakey due to bad generators and production validation logic missed the argument to String.format causing confusing errors omit 5c8f9cd858 Fixed an issue where table walk and token conflict mutations would use server timestamps which made multinode tests unstable omit fcf941da0b Add @Ignore to failing tests while we triage omit ff9817ed31 Fix: - Even if we can decide autonomously that we took the fast path, we must still wait for earlier transactions to decide themselves - We must update a command that is in CFK.loadingPruned whether or not it is outOfRange - We must visit pruned commands that are transitive dependencies of RX to ensure dependencies are propagated - flagsWithoutDomainAndKind() -> flagsWithoutDomainOrKindOrCardinality() [to fix integration regression] - Don't invoke uniqueNow() twic [...] omit 8b2b043572 Avoid double loading in Cassandra side of journal; make sure to include records appended to journal. omit 03586d793d Accord: jvm-dtest changed semantics of uncaughtExceptions handling which broke tests that depended on the prior semantics omit be616c285e When generating AbstractTypeGenerators.safeTypeGen() for tests composite types limit depth but collections didnt omit 5162426d8f Refactor RedundantStatus to encode vector of states that can be merged independently Also fix: - Truncate command on first access, without participants - Use Ballot.ZERO when invoking CFK.insertOutOfRange where appropriate - Don't supply a command's own route to ProgressLog.waiting to ensure new keys are incorporated - Ensure progress in CommandsForKey by setting vestigial commands to ERASED - Add any missing owned keys to StoreParticipants.route to ensure fetch c [...] omit 4d66a9411f Improve topology fetching omit fdd6d450a5 To improve accord interoperability test coverage, need to extend the harry model domain to handle more possible CQL states omit e13146a662 Follow-up to CASSANDRA-20222: - Implement missing parts of protocol optimisations, refine some particulars and remove MEDIUM_PATH_WAIT_ON_RECOVERY Also fix: - Deps.txnIds -> Deps.txnIdsWithFlags to make clear unsafety, and validate Command isn't created with flags - Save a lossy low/high epoch we're waiting on in WaitingState so that we can reconstruct the same Route on callback - Load any potentially invalidated commands we had in ProgressLog to ensure they are ma [...] omit 7c490c2c76 Fix TxnNamedRead discarding paging state omit 14d6bb9a34 Fix paging with Accord range reads omit 9242bbbb0b Fix ShortAccordSimulationTest omit dd667f01ad Live migration for non-serial reads omit 72d696344f Fix empty LatestDepsSerializer omit 5259330038 Follow-up to CASSANDRA-20228: - Fix waiting state callback computes different route to initiator - Invariants.checkX -> Invariants.requireX (to allow complementary Invariants.expectX as appropriate) omit 914c08f15a Fix testClearSnapshot_RemoveMultiple omit 6420f00259 Fix "modifiers" missing field exception on JDK17 omit 9156bdc43e Agree a distributed uniqueHlc to use for Apply - detect and save whether RX can be used as a GC bound for HLCs - have RX durably record if they witnessed any superseding epoch and use this for HLC GC - rework GC to retain applyAt for writes omit cef93dadd7 Fix race condition in AccordExecutor#cancelExclusive omit 08c06d6976 Fix Simulator Tests omit aeee9a58a6 Fix deps serializers omit ab2a8d4d5d Remove Journal record ownership tags functionality omit 224842c360 Migrate route index from commands table to journal, and drop the commands table omit 91788efcfd Protocol optimisations: - Privileged coordinator. If the coordinator is a replica we can reduce our quorum sizes by including the coordinator's vote. - with deps: if we include coordinator's preaccept deps we can reliably reduce quorum size by 1, at the expense of recovery sometimes requiring additional phases and waiting for future txns - with only vote: if we only include the vote we can avoid any additional recovery phases or waiting for future txns, but can [...] omit 35d49471e3 Fix ForceSnapshotTest omit fc25cbb6f7 Fix CoordinatorReadLatencyMetricTest omit 64bce3701f Fix serialization order for topology updates omit 87cccbbb06 Fix problem with iteration order in AccordJournal omit 4941180c38 FetchRequest should report as unavailable any slice that executes in a later epoch that is not owned by the replicas omit 7fbaf38f71 Fix topology replay during bootstrap and startup, decouple Accord from TCM omit c311dcfc5d Refactor the ast package to enable harry model based testing omit bb28f9f1ba Migrate in memory journal to CommandChange logic shared with AccordJournal omit 05c0927e70 Remove TimestampsForKey omit 4f4478c4e2 Fixes - Fix notifying unmanaged after update redundant before/bootstrap - Do not infer invalid if we have a single round of replies with minKnown not decided and maxKnown erased - in this case store the knowledge for next request. - Fix SyncPoint topology selection - Fix CheckStatusOkFull.with(InvalidIf) - Fix NotifyWaitingOn - ExecuteTxn should only contact latest topology for follow-up requests - DurableBefore.min should not go [...] omit 0a58b04907 Fix Accord SAI tests and Accord double apply omit 6f963657dd Implement field saving/loading in AccordJournal omit 75edf657a6 TCM's Retry.Deadline#retryIndefinitely is dangerous if used with RemoteProcessor as the deadline does not impact message retries omit 6076428d72 Accord simulation test failing with "ClassNotFoundException: WARN [AccordExecutor[1,0]" omit 08612da746 Fix infinite loop, and notify progress log of sync point durability while waiting to apply omit b8caf086ae Semi integrate burn test: omit eb80e7e996 Fix restartWithEpochChanges test; make CMS nodes first gossip with CMS nodes omit c4aaa722c0 Make it easier to reuse generators, make Harry more extensible and accord-proof, refactor Harry's major subsystems omit 885fd71ad9 Accord: Fix unit tests and improve burn test stability omit b9913055ae Key transaction recovery should witness range transactions omit 4c86dd7fb4 Fix AbstractKeys.without (and CoordinateEphemeralRead execution in future epoch) omit 6ed6f2e8d7 Set Accord debug tables partitioner to LocalPartitioner omit c9d4ffb7bc Non-serial reads and range reads without live migration support omit 80854d8657 Implement missing virtual tables for Accord debugging omit ba5b5979f5 Split accord migration into two phases omit a5dd6f9eaf Accord's ConfigService lock is held over large areas which cause deadlocks and performance issues omit b5faeae480 Improvements to queueing, execution and performance omit 724bced6fa Miscellaneous migration test fixes omit f45daf5dd8 Get Harry working on top of Accord and fix various issues found by TopologyMixupTestBase omit 4f83c754c6 Use ExclusiveSyncPoints to join a new topology omit b5542df745 Accord Deps tests have incorrect range semantics omit 8cda9c9728 Check for splittable ranges omit 1ad3651a7d TopologyMixupTestBase does not fix replication factor for Keyspaces after reaching rf=3 omit 771a6f6542 Accord should block currently unsafe operations omit 0cf0aabaf6 Accord metrics are isolated which cause existing coordination metrics to be empty, should also populate there as well omit 3e02b5178a Fix CASTest omit 59dd0b6e81 Shut down scheduler with "now" Fix NPE in MockJournal on null onFlush Fix SavedCommandTest. After the serialization change that serializes "changed" before "is null", null flag can no be written. omit 8226590392 Accord should not block partition restricted index queries omit 6bbf99dfa4 Add bounce to load test omit 91da288387 load historical transactions when loading topology omit f3b7362b1a AccordJournalPurger should use shouldCleanupPartial omit d03ba339db Store historical transactions per epoch update durability scheduling and majority deps fetching do not deserialize deps in CommandsForRangesLoader unless required omit b395be27d3 do not schedule additional durability attempts while some in flight; plus minor performance improvements omit d26867fee3 disable TableId interning omit 2b5d4527c0 increase timeout for CalculateDepsReq omit cde53cdd1a Ninja: fast path now updates after 3600s rather than 60s. omit 09723effd6 Follow-up to: Do not contact faulty replicas, and support reporting slow replies for preaccept/read. Do not wait for stale or left nodes for durability. omit a585835b77 Ninja: fast path now updates after 60s rather than 5s. Added toString to ReconfigureAccordFastPath so the TCM logs/table gives the debug info needed omit 789211b9d5 Fix truncatedApply deserialization omit ca127c1486 Journal diff serialization: validateFlags and WaitingOn size omit 21fa53744a Enable and test purging omit f71aa0a9ba Halve cache memory consumption by not retaining 'original' to diff; dedup RoutingKey tableId; avoid calculating rejectsFastPath in more cases; delay retry of fetchMajorityDeps; fix SetShardDurable marking shards durable omit 20e67bd7f5 ExclusiveSyncPoints should always wait for a simple quorum omit 4db862ac1a Fix condition on where we shut down accord; move scheduled executor shutdown until after MS. omit 6fa27dc5b9 visit journal backwards to save time parsing don't load range commands that are redundant, and load least possible use MISC verb handler for maintenance tasks omit 10c6ba1e2d ninja: increase durability scheduling delays ninja: fix NPE disable ephemeral reads omit f797878aa3 Accord: Share DurableBefore between CommandStores omit 36ccd1db82 Follow-up to CASSANDRA-19967 and CASSANDRA-19869 omit 383fb94a5f ninja: Fix AcceptSerializer omit c5583a1aee ninja: fix CFK serializer omit 540e20dab8 Support Restart node in Accord omit 7172da551e Add purging to Accord Journal table omit 625b0d9e5f Ninja: Repin accord omit 71e9238b68 improve AccordLoadTest to support more keys omit f23cb632de CASSANDRA-19967: Refine epoch quorum intersections, sync points, conflict relations and outcome inference omit c2fbb422bd Implement Journal replay on startup: * reconstruct CFK, TFK, progressLog * migrate CommandStore collection state from Accord table to the log * make memtable writes non-durable; reconstruct memtable state from Writes omit 4bbb9628aa ninja: handle serialization of truncated accept responses omit 3e3018fa31 Change MaxConflicts to use a BTree under the hood omit 0462ca05f2 This commits contains the following two patches in order to reduce the amount of conflicts resolution necessary for future rebasing: omit 13a37f4ea5 CEP-15 (C*) - misc accord perf improvements omit 1f06ebdcac CEP-15 (C*): Read accord repair cfk keys from sstable index. omit 21ba9510ab Rebase fixup: Accord should follow the pattern and use requestTime.computeDeadline like the rest of the code, and Accord timeout MUST be less than user timeout Rebase fixup: when a local keyspace is being open but it isnt present return null so error msg can be provided Rebase fixup: improved metrics error msg when the exception doesnt match what is expected Rebase improvement: when we see a timeout or preempt use the new vtable to show the status cross the cluster Reb [...] omit d3e241dd42 Ninja fix: RandomSource.asJdkRandom did not provide a seed. This should have been fixed in CASSANDRA-19847 as it was fixed on Cassadnra trunk omit ec4c923954 Rebase fixup: SerializationsTest needed to recreate the service.SyncComplete.bin file omit 032cf130a5 CEP-15: C* - Early repair failures hang nodetool omit f2c4e77922 CEP-15 (C*) increase message timeouts for range barrier messages omit 9ad574a34f Journal segment compaction omit f206f58585 Redesign progress mechanisms to be memory efficient, use fewer messages and to resolve dependency chains promptly. omit 6d0a40663a Fix Journal segment allocation/switch race condition omit 0a7c253574 Fix a problem with static segments being opened with an empty offset after switch unless active segments offset file was closed omit d4e2867305 Create a fuzz test that randomizes topology changes, cluster actions, and CQL operations omit 048105dc88 Add a table to inspect the current state of a txn omit aea01e88fe Command to Exclude Replicas from Durability Status Coordination omit c242971e8c Switch to streaming serialization of SavedCommand omit bc93cd344e txns that update a static row when the desired row doesn't exist leads to an error omit 30357e85eb Add size to the segment index for safer journal reads omit e990d258dc Switch to infinite loop executor instead of a while-loop thread. omit 55714b8e60 Revert acccord module to absolute path omit 081669e2ab Add an ability to reconstruct arbitrary epoch state from the log to TCM omit f7aaede996 CommandsForRanges does not support slice which cause over returned data being sent omit b0d3b4b8f2 Accord migration and interop correctness omit 22fa0c1248 CASSANDRA-19825: Fix various bugs and abstraction deficiencies, including: omit 8e4313964d CommandsForKey Improvements incl Pruning omit 66c900749d Accord Journal / Determinism omit 534af5ae2d CEP-15 (Accord): When starting a transaction in a table where Accord is not enabled, should fail fast rather than fail with lack of ranges omit 4497253e16 Bring back Journal simulator (w/o Accord at least for now); add semaphore interceptor. omit 4d93ad6539 CEP-15: (Accord) When nodes are removed from a cluster, need to update topology tracking to avoid being blocked omit c578c824b2 Introduce Periodic mode to Accord Journal omit 5af655b4d7 CEP-15: (Accord) SyncPoint timeouts become a Exhausted rather than a Timeout and doesn’t get retried omit 1f7f3a5e0d ApplyThenWaitUntilApplied supplies wrong epoch for executeAtEpoch omit cbfbde88d1 Don't run incremental repairs for consensus only repairs omit 64c5cb9314 Move burn test read timestamp validation from replica to coordination omit ad00f0cf74 CEP-15: (Accord) Bootstraps LocalOnly txn can not be recreated from SerializerSupport omit 13f4628f16 ninja: fix cqlsh tests omit e65ee4d1ad Accord Journal Determinism: PreAccept replay stability omit c8347eb4fe Accord barrier/inclusive sync point fixes omit 903cda4fd8 Baseline Diagnostic vtables for Accord omit e42936c056 IndexOutOfBoundsException while serializing CommandsForKey omit 980a34c138 Move preaccept expiration logic away from Agent omit f8dd392609 Accord: Need to simulate Cassandra Journal in Accord BurnTest to detect issues earlier before they are seen in Cassandra omit fd37498072 post-rebase fixes, mostly around CASSANDRA-19341 and CASSANDRA-19567 omit ab4ef82b42 Accord: NPE in RangeDeps.forEach omit 9a244593a7 Prohibit counter column access in Accord transactions omit a43306cdd8 CEP-15 (C*) Integrate accord with repair omit ceebe6baa7 When jvm-dtest is shutting down an instance TCM retries block the shutdown causing the test to fail omit c7244541d6 (Accord) Cassandra bootstrap no longer using the range txn and instead uses the sync point empty txn for reads omit 8c7ebf8906 Accord: PreLoadContext must properly and consistently support ranges omit 280aec9145 Ninja for CASSANDRA-19305: Disable EphemeralRead by default to get benchmarks stable omit 56106d7c04 perf improvements omit 8f88bd3332 post-trunk rebase fixes omit f20acef031 fix MultiElementType pack/unpack accessor api omit 4cdefe3948 CEP-15: (C*) per-table transactional configuration omit d8f12802c4 Refactor CommandsForKey for efficiency, and to support transitive dependency elision omit 5376ae43a2 Fast single-partition "Ephemeral Reads" omit 070f6543f6 Add TODO for TxnQuery using wrong cluster metadata omit 723610cbf8 Fix (de)serialization of WaitingOn into cache omit 6147578a71 Get simulator working (again) omit 0437826587 CASSANDRA-18365: Protocol fixes omit 89b72c683e Fix LocalPartitioner duplication in *_for_key Accord system tables omit 58a5f7da58 Pre-requisite changes for CASSANDRA-18888 omit fd0024bade (Accord): Bug fixes from CASSANDRA-18675 to better support adding keyspaces omit b48d5129d8 (Accord) NPE while trying to serialize FoundKnownMap as value is null half the time but unexpected while serializing omit 0e6a6206f3 Schema based accord fast path configuration omit 72a8849bf2 Fix TombstoneCountWarningTest Don't use ImmediateExectuor in JVM dtests to process messages Fix GuardrailTablesTest Instance.receiveMessage should use sync Fix StorageAttachedIndexDDLTest failing due to background Accord compactions Add back enforceInvariants=false to shouldCleanup in AccordCommandsPurger Fix CompactionAccordIteratorsTest Fix empty row check in command/tfk mutation methods omit 810b07485d Fix AccordCommandsPurger universal durability check omit 1be0dd7051 Fix Mutation serializer Fix AccordJournal.Type interop entries omit fcac5d182c Fix additional live migration/interop merge issues omit 53fefba7c5 Reduce command deps omit 9b1e22beae Improve validation and address various discovered faults omit 511074f99b Quick fix for AccordCommandStoreTest#commandLoadSave() omit dc689cf063 Fix Paxos V2 prepare response serialization omit dec690f93a Accord/non-Accord interoperability and support for live migration omit dc914651ff Several simulator fixes not yet merged to cep-15-accord omit a92802357f Use pinned Harry version omit 3c12d13f29 CEP-15: (C*) Implement TopologySorter to prioritise hosts based on DynamicSnitch and/or topology layout omit e4d52d7fb5 CEP-15: Add Accord configuration stub omit 8dc447c262 CEP-15: Accord metrics omit 7919fb8eab ninja-fix: minor post-TCM-rebase cleanup omit 327af9a671 ninja-fix: checkstyle fixes in VirtualTableTest, HistoryValidatorTest, SeedDefiner, RunStartDefiner, and Config omit 9e37f30d70 CEP-15: Minimize transaction state kept in system tables omit 857e9cea6b - Avoid persisting fragments that do not require completion as Updates in TxnWrite, as they can simply be pulled from PartialTxn when needed in Write#apply() - Avoid serializing full TxnData instances to Accord state tables omit 9edce1c29d CEP-15: (C*) accord.messages.BeginRecovery.RecoverNack#supersededBy is nullable but C* serializer doesn't expect null omit e6e9e307ac Fix Accord compaction purger tombstone logic omit a503877d48 Bootstrap/TCM integration test fixes omit a6554fd66a CEP-15 (C*): when loading commands that have empty waiting_on, make sure not to loose the partial deps (#3590) omit a1ac5f5ac9 Allow exceptions to be propagated remotely omit 5a59f45139 CASSANDRA-18774: Fix pre-commit hook omit 532feb2caf CEP-15 (C*) When a host replacement happens don't loose the peer mapping right away (#3575) omit 8e1473590d Support state eviction (i.e. permit the state machine to erase transactions that are known to be applied across the cluster) omit 1b1e290d80 CEP-15: Accord Bootstrap Integration omit 8c21ea5bc1 CEP-15: Convert AccordStateCache cache from write-through to write-back omit 4b739d6635 CEP-15: Extend Accord MessageType with a side effect flag omit 82cfde5839 CEP-15: (C*) Add notion of CommandsForRanges and make this durable in C* omit fc1e2bcd6f - make sure workspace.xml specifies a storagedir - removing unnecessary calls to ServerTestUtils.daemonInitialization() in a handful of tests - minor cleanup in Verb and BTreeSet omit 61d999ea9a (Accord only) Permit nodes to join a cluster without the full transaction history omit 4fcb4b1b94 CEP-15: Store PreAccept, Accept, Commit, and Apply messages in a durable log before processing by CommandStores omit 5f404433a6 CEP-15: (C*) Enhance in-memory FileSystem to work with mmap and support tests to add custom logic omit 8fe40b1c53 CEP-15: (C*) Accord message processing should avoid being passed on to a Stage and run directly in the messageing handler omit 821f239f90 CEP-15 (Accord) Original and recover coordinators may hit a race condition with PreApply where reads and writes are interleaved, causing one of the coordinators to see the writes from the other omit a396e3529e CEP-15 (Accord) Expected reply message with verb ACCORD_INFORM_OF_TXNID_RSP but got ACCORD_SIMPLE_RSP omit 6038c90bf6 Operations.migrateReadRequiredOperations fails due to concurrent access when TransactionStatement is prepared omit a94f812768 fixing remaining (mostly compilation) issues after initial rebase of cep-15-accord on cep-21-tcm omit b3cbb20831 Improve transaction statement validation omit 2411579b97 CEP-15: (Accord) Migrate Accord away from JDK random to a new interface RandomSource omit 2141ad1bf9 Remove git hook for pre-push as it is redundant and causes issues when merging to mainline omit 3ae6a056ba CEP-15/Accord Immutable State Refactor omit 2872b514f3 add AsyncChain implementations and tests omit 724b186593 CEP-15 (C*): Messaging and storage engine integration add 73f0e2e401 Suppress CVE-2025-25193 add 49ec0dc692 Merge branch 'cassandra-4.0' into cassandra-4.1 add 7076623ac7 Merge branch 'cassandra-4.1' into cassandra-5.0 add 9d6c3c43b8 Merge branch 'cassandra-5.0' into trunk add b01274d6f6 CASSANDRA-20496 - Add CQLSSTableWriter tests for Vectors and Constraints add 27a6ef3ea8 Update netty to 4.1.119.Final and netty-tcnative to 2.0.70.Final add 89d153affd Merge branch 'cassandra-5.0' into trunk add 51cf55747b Update OWASP dependency checker to version 12.1.0 add 147a224a88 Merge branch 'cassandra-4.0' into cassandra-4.1 add abb3e8bdbb Merge branch 'cassandra-4.1' into cassandra-5.0 add 6468b2898d Merge branch 'cassandra-5.0' into trunk add a449a4f76b PaxosCleanupLocalCoordinator wait for transaction timeout before repairing add 5aadbc62f0 CASSANDRA-20117: fixed typos in NTR spec and SASI documents add cf60eb8672 Fix mixed mode paxos hang add eaa20275ff Merge branch 'cassandra-4.1' into cassandra-5.0 add 45df7627d5 Merge branch 'cassandra-5.0' into trunk add 50978a0d07 CASSANDRA-20402: Add new reason RequestFailureReason.INDEX_BUILD_IN_PROGRESS and IndexBuildInProgress exception when queries fail during index build add 5bc66043d7 ninja: remove unused import add 12a9f16eec Merge branch 'cassandra-4.1' into cassandra-5.0 add a777cf0392 Merge branch 'cassandra-5.0' into trunk add 7f1503d9c9 Split out truncation record locking to prevent it being blocked by slow interval tree build on removeEndpoint add 38f3afcd42 Several doc fixes, particularly developing/cql/ , developing/data-modeling/ and managing/ add bfea5446d7 Merge branch 'cassandra-5.0' into trunk add b31d15b9b5 Avoid failing queries when epoch changes and replica goes up/down add 95aca49915 Avoid NPE during cms initialization abort add c3089b564a Ignore repetitions of semicolon in CQLSH add 07bbf2cd97 Docs should mention that cassandra.yaml should be secure add 93e77182d5 Merge branch 'cassandra-4.0' into cassandra-4.1 add 0c9d368e09 Merge branch 'cassandra-4.1' into cassandra-5.0 add 5ccb3fd215 Merge branch 'cassandra-5.0' into trunk add b81163b04b Prepare debian changelog for 5.0.4 add b991a68faf Merge branch 'cassandra-5.0' into trunk add 694a88bdbe Upgrade java-driver-core to 3.12.1 and org.apache coords add 54bca21dab Merge branch 'cassandra-5.0' into trunk add f6de28f727 Move generate-snyk-file target dependency from build-project to dependency-check add 09bc381d59 Merge branch 'cassandra-4.0' into cassandra-4.1 add b4f46ec473 Merge branch 'cassandra-4.1' into cassandra-5.0 add 63e0188b2e Merge branch 'cassandra-5.0' into trunk add 270839ddef ninja-fix – cqlversion in docker quickstart doc add 2039ef7658 Merge branch 'cassandra-5.0' into trunk add f33c3450d4 Only prefetch docker image needed in jenkinsfile build stages add c02a6a0605 Merge branch 'cassandra-5.0' into trunk add 96bfbe6250 Dropwizard Meter causes timeouts when infrequently used add ec00426584 Merge branch 'cassandra-4.0' into cassandra-4.1 add 944baa08da Merge branch 'cassandra-4.1' into cassandra-5.0 add 1bcd91f531 Merge branch 'cassandra-5.0' into trunk add 67df6a5bff Add SSTableIntervalTree latency metric add d06e496773 Update AST Harry CAS fuzz tests to validate the CAS response add 4e2e5f3c57 Fixed multiple single-node SAI query bugs relating to static columns - Ensure MemtableIndexWriter calculates min/max properly with indexes on partition key elements - Ensure only rows with live data are indexed - Ensure min cannot be greater than max in intersection statistics with static keys - Correct tracking of last key in the searcher in the presence of static keys add ef0eec07f8 Merge branch 'cassandra-5.0' into trunk add 6472340d93 AST Harrys multi node tests are flakey when multi cell list happens cross instances add c18b1e937f Bump version, prepare CHANGES add 19b4763d1b Merge branch 'cassandra-5.0' into trunk add 040d5e0111 Bump version, prepare CHANGES add 1f9648a7a8 correct CHANGES add 9f4a4b690e Merge branch 'cassandra-5.0' into trunk add bcd5c48307 Fix test failure with negative position in ThreadLocalReadAheadBufferTest add 0e408e6175 Merge branch 'cassandra-5.0' into trunk add 725e4ba3eb Avoid purging deletions in RowFilter when reconciliation is required add 30f1429e6d Merge branch 'cassandra-5.0' into trunk add 7a888149df Support topology-safe changes to Datacenter & Rack for live nodes add 6f79207c34 Improve metadata log catch up with inter-DC mutation forwarding add eb0c77ae1e Fix TreeMap race in CollectionVirtualTableAdapter causing us to lose rows in the virtual table add 2c05f82755 Add nodetool command to dump the contents of the system_views.{cluster_metadata_log, cluster_metadata_directory} tables add 8404d2fd5c Improve performance when getting writePlacementAllSettled from ClusterMetadata in large cluster with many range movements add d3658f0491 Fix JMX initialization problem in injvm-dtest framework add fcea0b6fd8 CASSANDRA-19633 Replaced node is stuck in a loop calculating ranges add 3e2e4c3ea1 Merge branch 'cassandra-4.0' into cassandra-4.1 add 0f9bb246cd Merge branch 'cassandra-4.1' into cassandra-5.0 add f7462446a4 Merge branch 'cassandra-5.0' into trunk add 6b2cdba56b AST fuzz tests can be flakey in multi node clusters due to ephemeral read errors caused by a race condition issue with SAIUtils test class add 78290bed45 Avoid computing prepared statement size for unprepared batches add 8077f276c2 Merge branch 'cassandra-4.0' into cassandra-4.1 add 7d93089a1e Merge branch 'cassandra-4.1' into cassandra-5.0 add 1e7e24893d Merge branch 'cassandra-5.0' into trunk add 4436b84a5a SAI fails queries when multiple columns exist and a non-indexed column is a composite with a map add e1823e0e22 Merge branch 'cassandra-5.0' into trunk new 09c8fa1030 CEP-15 (C*): Messaging and storage engine integration new e6a8c030c1 add AsyncChain implementations and tests new 34bd73e2a7 CEP-15/Accord Immutable State Refactor new 6c62cc4ffa Remove git hook for pre-push as it is redundant and causes issues when merging to mainline new f092461114 CEP-15: (Accord) Migrate Accord away from JDK random to a new interface RandomSource new ea71336278 Improve transaction statement validation new 68135aaf29 fixing remaining (mostly compilation) issues after initial rebase of cep-15-accord on cep-21-tcm new 4e95e3a440 Operations.migrateReadRequiredOperations fails due to concurrent access when TransactionStatement is prepared new 44824a6e45 CEP-15 (Accord) Expected reply message with verb ACCORD_INFORM_OF_TXNID_RSP but got ACCORD_SIMPLE_RSP new 7ad2bf672c CEP-15 (Accord) Original and recover coordinators may hit a race condition with PreApply where reads and writes are interleaved, causing one of the coordinators to see the writes from the other new 8633a301f7 CEP-15: (C*) Accord message processing should avoid being passed on to a Stage and run directly in the messageing handler new da92eed225 CEP-15: (C*) Enhance in-memory FileSystem to work with mmap and support tests to add custom logic new 8dc82a6369 CEP-15: Store PreAccept, Accept, Commit, and Apply messages in a durable log before processing by CommandStores new 537c1f991a (Accord only) Permit nodes to join a cluster without the full transaction history new f54144d1c1 - make sure workspace.xml specifies a storagedir - removing unnecessary calls to ServerTestUtils.daemonInitialization() in a handful of tests - minor cleanup in Verb and BTreeSet new f76f806de1 CEP-15: (C*) Add notion of CommandsForRanges and make this durable in C* new 8f156fc5dc CEP-15: Extend Accord MessageType with a side effect flag new 145c467c69 CEP-15: Convert AccordStateCache cache from write-through to write-back new 00bbf4ec8e CEP-15: Accord Bootstrap Integration new a6bb08d926 Support state eviction (i.e. permit the state machine to erase transactions that are known to be applied across the cluster) new 76994f8196 CEP-15 (C*) When a host replacement happens don't loose the peer mapping right away (#3575) new bfa0e59f7f CASSANDRA-18774: Fix pre-commit hook new 82acd3e950 Allow exceptions to be propagated remotely new 6167e436c3 CEP-15 (C*): when loading commands that have empty waiting_on, make sure not to loose the partial deps (#3590) new ba02bd0bc7 Bootstrap/TCM integration test fixes new 10e065ffe5 Fix Accord compaction purger tombstone logic new cfc63edcb4 CEP-15: (C*) accord.messages.BeginRecovery.RecoverNack#supersededBy is nullable but C* serializer doesn't expect null new 08428a2750 - Avoid persisting fragments that do not require completion as Updates in TxnWrite, as they can simply be pulled from PartialTxn when needed in Write#apply() - Avoid serializing full TxnData instances to Accord state tables new c0a63c9164 CEP-15: Minimize transaction state kept in system tables new 19fcbcab61 ninja-fix: checkstyle fixes in VirtualTableTest, HistoryValidatorTest, SeedDefiner, RunStartDefiner, and Config new cffe1cc61a ninja-fix: minor post-TCM-rebase cleanup new 27a9313970 CEP-15: Accord metrics new dfd1e99fd1 CEP-15: Add Accord configuration stub new 18d3aa47bc CEP-15: (C*) Implement TopologySorter to prioritise hosts based on DynamicSnitch and/or topology layout new 169f790cb1 Use pinned Harry version new 9e766f46a3 Several simulator fixes not yet merged to cep-15-accord new c10c84b9cd Accord/non-Accord interoperability and support for live migration new 4c575a7e48 Fix Paxos V2 prepare response serialization new 0e899532dc Quick fix for AccordCommandStoreTest#commandLoadSave() new 050688228f Improve validation and address various discovered faults new d07a36106d Reduce command deps new a9cb71833a Fix additional live migration/interop merge issues new 021b06639e Fix Mutation serializer Fix AccordJournal.Type interop entries new 1f4675a902 Fix AccordCommandsPurger universal durability check new e812127a8b Fix TombstoneCountWarningTest Don't use ImmediateExectuor in JVM dtests to process messages Fix GuardrailTablesTest Instance.receiveMessage should use sync Fix StorageAttachedIndexDDLTest failing due to background Accord compactions Add back enforceInvariants=false to shouldCleanup in AccordCommandsPurger Fix CompactionAccordIteratorsTest Fix empty row check in command/tfk mutation methods new cb1a05c5d4 Schema based accord fast path configuration new a95f072e35 (Accord) NPE while trying to serialize FoundKnownMap as value is null half the time but unexpected while serializing new 633bbee1a6 (Accord): Bug fixes from CASSANDRA-18675 to better support adding keyspaces new 20af77031a Pre-requisite changes for CASSANDRA-18888 new 5385e9c60a Fix LocalPartitioner duplication in *_for_key Accord system tables new 8de3dfc711 CASSANDRA-18365: Protocol fixes new a324003c59 Get simulator working (again) new 266986b5a2 Fix (de)serialization of WaitingOn into cache new 4088c68d38 Add TODO for TxnQuery using wrong cluster metadata new 763bcf2de5 Fast single-partition "Ephemeral Reads" new d3deef36d3 CEP-15: (C*) per-table transactional configuration new 94c2ec3c96 fix MultiElementType pack/unpack accessor api new 599cd59736 post-trunk rebase fixes new 7dadc080cb perf improvements new 333c748a91 Ninja for CASSANDRA-19305: Disable EphemeralRead by default to get benchmarks stable new 777cf84f64 Accord: PreLoadContext must properly and consistently support ranges new 64da7141f7 (Accord) Cassandra bootstrap no longer using the range txn and instead uses the sync point empty txn for reads new 8f7f9b083e When jvm-dtest is shutting down an instance TCM retries block the shutdown causing the test to fail new 543210ae12 CEP-15 (C*) Integrate accord with repair new 418f8bf3f0 Prohibit counter column access in Accord transactions new 980f1963f3 Accord: NPE in RangeDeps.forEach new 1b3c3f32a4 post-rebase fixes, mostly around CASSANDRA-19341 and CASSANDRA-19567 new 3127c558d3 Move preaccept expiration logic away from Agent new 11ed5309b5 IndexOutOfBoundsException while serializing CommandsForKey new 1c87bb96ed Baseline Diagnostic vtables for Accord new feb8d2e2b2 Accord barrier/inclusive sync point fixes new 128f526f4a Accord Journal Determinism: PreAccept replay stability new c909f2d669 ninja: fix cqlsh tests new 39efba9c6a CEP-15: (Accord) Bootstraps LocalOnly txn can not be recreated from SerializerSupport new 17c186ad64 Move burn test read timestamp validation from replica to coordination new d6bf8ec3d0 Don't run incremental repairs for consensus only repairs new 3dfd80ff78 ApplyThenWaitUntilApplied supplies wrong epoch for executeAtEpoch new d83de7fbb4 CEP-15: (Accord) SyncPoint timeouts become a Exhausted rather than a Timeout and doesn’t get retried new ac58a62c51 Introduce Periodic mode to Accord Journal new 0d4c0961be CEP-15: (Accord) When nodes are removed from a cluster, need to update topology tracking to avoid being blocked new 1439fe8d31 Bring back Journal simulator (w/o Accord at least for now); add semaphore interceptor. new 6d01bc2535 CEP-15 (Accord): When starting a transaction in a table where Accord is not enabled, should fail fast rather than fail with lack of ranges new 6905157b0e CommandsForKey Improvements incl Pruning new c377a066af CASSANDRA-19825: Fix various bugs and abstraction deficiencies, including: new 8ab5003118 Accord migration and interop correctness new a7c2bcafcd CommandsForRanges does not support slice which cause over returned data being sent new c16297f862 Add an ability to reconstruct arbitrary epoch state from the log to TCM new 66c50d796a Revert acccord module to absolute path new 788caeeef3 Switch to infinite loop executor instead of a while-loop thread. new b757cb8730 Add size to the segment index for safer journal reads new 26d517f4ff txns that update a static row when the desired row doesn't exist leads to an error new 325e48ac39 Switch to streaming serialization of SavedCommand new 2b01b5fa79 Command to Exclude Replicas from Durability Status Coordination new a21c0a75aa Add a table to inspect the current state of a txn new f1db115e73 Create a fuzz test that randomizes topology changes, cluster actions, and CQL operations new 00ebcdc9f2 Fix a problem with static segments being opened with an empty offset after switch unless active segments offset file was closed new 1866b8ec73 Fix Journal segment allocation/switch race condition new 9ebce5d6df Redesign progress mechanisms to be memory efficient, use fewer messages and to resolve dependency chains promptly. new ef5f793dab Journal segment compaction new 058640be4e CEP-15 (C*) increase message timeouts for range barrier messages new 0f51ee407b CEP-15: C* - Early repair failures hang nodetool new f87c22cbbd Rebase fixup: SerializationsTest needed to recreate the service.SyncComplete.bin file new 6265ce5187 Ninja fix: RandomSource.asJdkRandom did not provide a seed. This should have been fixed in CASSANDRA-19847 as it was fixed on Cassadnra trunk new 8f7532bc10 Rebase fixup: Accord should follow the pattern and use requestTime.computeDeadline like the rest of the code, and Accord timeout MUST be less than user timeout Rebase fixup: when a local keyspace is being open but it isnt present return null so error msg can be provided Rebase fixup: improved metrics error msg when the exception doesnt match what is expected Rebase improvement: when we see a timeout or preempt use the new vtable to show the status cross the cluster Reb [...] new dd1230f2a2 CEP-15 (C*): Read accord repair cfk keys from sstable index. new fbfb633bda CEP-15 (C*) - misc accord perf improvements new 1ed52038ce This commits contains the following two patches in order to reduce the amount of conflicts resolution necessary for future rebasing: new 3ef74f5948 Change MaxConflicts to use a BTree under the hood new 1c7c311a2d ninja: handle serialization of truncated accept responses new 1c269348a4 Implement Journal replay on startup: * reconstruct CFK, TFK, progressLog * migrate CommandStore collection state from Accord table to the log * make memtable writes non-durable; reconstruct memtable state from Writes new 562894dcb1 improve AccordLoadTest to support more keys new d0a3586bd9 Add purging to Accord Journal table new d034661ae2 Support Restart node in Accord new 282bacb84b ninja: fix CFK serializer new 5693d0e2f4 ninja: Fix AcceptSerializer new 8f7b1c2351 Follow-up to CASSANDRA-19967 and CASSANDRA-19869 new 0e1819bbb3 Accord: Share DurableBefore between CommandStores new ab6ef97d28 ninja: increase durability scheduling delays ninja: fix NPE disable ephemeral reads new d8ccc36e4d visit journal backwards to save time parsing don't load range commands that are redundant, and load least possible use MISC verb handler for maintenance tasks new cc321720e3 Fix condition on where we shut down accord; move scheduled executor shutdown until after MS. new 82ae3adcb2 ExclusiveSyncPoints should always wait for a simple quorum new aa6ebc140f Halve cache memory consumption by not retaining 'original' to diff; dedup RoutingKey tableId; avoid calculating rejectsFastPath in more cases; delay retry of fetchMajorityDeps; fix SetShardDurable marking shards durable new 25b1bd74b8 Enable and test purging new b8f3b74587 Journal diff serialization: validateFlags and WaitingOn size new 1734011032 Fix truncatedApply deserialization new 0769af2c2e Ninja: fast path now updates after 60s rather than 5s. Added toString to ReconfigureAccordFastPath so the TCM logs/table gives the debug info needed new c4e0f3cf8f Follow-up to: Do not contact faulty replicas, and support reporting slow replies for preaccept/read. Do not wait for stale or left nodes for durability. new 9b6467d81e Ninja: fast path now updates after 3600s rather than 60s. new aeaa07b000 increase timeout for CalculateDepsReq new 570d65249f disable TableId interning new 91def31284 do not schedule additional durability attempts while some in flight; plus minor performance improvements new 22df42c7be Store historical transactions per epoch update durability scheduling and majority deps fetching do not deserialize deps in CommandsForRangesLoader unless required new 042a6e9769 Add bounce to load test new a51bed1c77 Accord should not block partition restricted index queries new d1af562705 Shut down scheduler with "now" Fix NPE in MockJournal on null onFlush Fix SavedCommandTest. After the serialization change that serializes "changed" before "is null", null flag can no be written. new 5afe99b7ca Accord metrics are isolated which cause existing coordination metrics to be empty, should also populate there as well new 1f80f99b71 Accord should block currently unsafe operations new e94d86b1e5 TopologyMixupTestBase does not fix replication factor for Keyspaces after reaching rf=3 new 4e9110e881 Check for splittable ranges new 5a12875b3a Use ExclusiveSyncPoints to join a new topology new 9fe1a977b5 Get Harry working on top of Accord and fix various issues found by TopologyMixupTestBase new 6c0ad476ed Miscellaneous migration test fixes new 513509ee2c Accord's ConfigService lock is held over large areas which cause deadlocks and performance issues new b29eac16fe Split accord migration into two phases new bcf1abbd16 Implement missing virtual tables for Accord debugging new 16b5e191f9 Non-serial reads and range reads without live migration support new 304749ebda Set Accord debug tables partitioner to LocalPartitioner new 03cd31fc57 Fix AbstractKeys.without (and CoordinateEphemeralRead execution in future epoch) new f2ea674140 Key transaction recovery should witness range transactions new ec7c7c9fce Accord: Fix unit tests and improve burn test stability new bad9805b7a Make it easier to reuse generators, make Harry more extensible and accord-proof, refactor Harry's major subsystems new 4c0f9198de Fix restartWithEpochChanges test; make CMS nodes first gossip with CMS nodes new 74817bc83e Semi integrate burn test: new a5dc44ad57 Fix infinite loop, and notify progress log of sync point durability while waiting to apply new c114fc46d2 Accord simulation test failing with "ClassNotFoundException: WARN [AccordExecutor[1,0]" new 211249b8a9 Implement field saving/loading in AccordJournal new 21383d2789 Fix Accord SAI tests and Accord double apply new 44cd181438 Fixes - Fix notifying unmanaged after update redundant before/bootstrap - Do not infer invalid if we have a single round of replies with minKnown not decided and maxKnown erased - in this case store the knowledge for next request. - Fix SyncPoint topology selection - Fix CheckStatusOkFull.with(InvalidIf) - Fix NotifyWaitingOn - ExecuteTxn should only contact latest topology for follow-up requests - DurableBefore.min should not go [...] new 52242f23a8 Remove TimestampsForKey new b664d3e616 Migrate in memory journal to CommandChange logic shared with AccordJournal new 17dbba770b Refactor the ast package to enable harry model based testing new dfcf3aff4e Fix topology replay during bootstrap and startup, decouple Accord from TCM new ed2ca6e2fb FetchRequest should report as unavailable any slice that executes in a later epoch that is not owned by the replicas new 5bfd944bcb Fix problem with iteration order in AccordJournal new 21f9f248cd Fix serialization order for topology updates new 72f3828c0b Fix CoordinatorReadLatencyMetricTest new 13625288c6 Fix ForceSnapshotTest new 3e16efa490 Protocol optimisations: - Privileged coordinator. If the coordinator is a replica we can reduce our quorum sizes by including the coordinator's vote. - with deps: if we include coordinator's preaccept deps we can reliably reduce quorum size by 1, at the expense of recovery sometimes requiring additional phases and waiting for future txns - with only vote: if we only include the vote we can avoid any additional recovery phases or waiting for future txns, but can [...] new 3c7cb1e3b3 Migrate route index from commands table to journal, and drop the commands table new b8a3706ef9 Remove Journal record ownership tags functionality new 70382b13cc Fix deps serializers new b50536f1a4 Fix Simulator Tests new f3e61c153f Fix race condition in AccordExecutor#cancelExclusive new 5be214a417 Agree a distributed uniqueHlc to use for Apply - detect and save whether RX can be used as a GC bound for HLCs - have RX durably record if they witnessed any superseding epoch and use this for HLC GC - rework GC to retain applyAt for writes new 894536ab71 Fix "modifiers" missing field exception on JDK17 new 2e850db02d Fix testClearSnapshot_RemoveMultiple new 7f0e5f8e8b Follow-up to CASSANDRA-20228: - Fix waiting state callback computes different route to initiator - Invariants.checkX -> Invariants.requireX (to allow complementary Invariants.expectX as appropriate) new cf9e34df45 Fix empty LatestDepsSerializer new 6522d52b0b Live migration for non-serial reads new d88596b6a4 Fix ShortAccordSimulationTest new f55653ecb0 Fix paging with Accord range reads new 9da41308d3 Fix TxnNamedRead discarding paging state new 113d83417e Follow-up to CASSANDRA-20222: - Implement missing parts of protocol optimisations, refine some particulars and remove MEDIUM_PATH_WAIT_ON_RECOVERY Also fix: - Deps.txnIds -> Deps.txnIdsWithFlags to make clear unsafety, and validate Command isn't created with flags - Save a lossy low/high epoch we're waiting on in WaitingState so that we can reconstruct the same Route on callback - Load any potentially invalidated commands we had in ProgressLog to ensure they are ma [...] new d543cc9107 To improve accord interoperability test coverage, need to extend the harry model domain to handle more possible CQL states new c4a5971af3 Improve topology fetching new af7dfcc327 Refactor RedundantStatus to encode vector of states that can be merged independently Also fix: - Truncate command on first access, without participants - Use Ballot.ZERO when invoking CFK.insertOutOfRange where appropriate - Don't supply a command's own route to ProgressLog.waiting to ensure new keys are incorporated - Ensure progress in CommandsForKey by setting vestigial commands to ERASED - Add any missing owned keys to StoreParticipants.route to ensure fetch c [...] new f345011822 When generating AbstractTypeGenerators.safeTypeGen() for tests composite types limit depth but collections didnt new 4f2e306c26 Accord: jvm-dtest changed semantics of uncaughtExceptions handling which broke tests that depended on the prior semantics new 080d6c633d Avoid double loading in Cassandra side of journal; make sure to include records appended to journal. new d7ed1ad391 Fix: - Even if we can decide autonomously that we took the fast path, we must still wait for earlier transactions to decide themselves - We must update a command that is in CFK.loadingPruned whether or not it is outOfRange - We must visit pruned commands that are transitive dependencies of RX to ensure dependencies are propagated - flagsWithoutDomainAndKind() -> flagsWithoutDomainOrKindOrCardinality() [to fix integration regression] - Don't invoke uniqueNow() twic [...] new b017ae8175 Add @Ignore to failing tests while we triage new 341b493863 Fixed an issue where table walk and token conflict mutations would use server timestamps which made multinode tests unstable new 5e9e2b2b04 Accord: TableParamsTest is flakey due to bad generators and production validation logic missed the argument to String.format causing confusing errors new 3324e23f64 Fix topology loading new 1aaf0bb336 Fix null field accounting new b81d13f994 Expose epoch ready state as a vtable for operators to inspect things new 3fb23e0247 Fix flakey test org.apache.cassandra.simulator.test.ShortPaxosSimulationTest#casOnAccordSimulationTest new 64c4aa88f5 (Accord) Fix recovery when one shard is committed/stable and another is preaccepted Also fix: - Topology slicing must declare whether we share/slice node ownership (to assist above) - CFK.visit removes transitive dependencies too eagerly across epoch change - apply cleanup to builder consistently, and construct the same value we would produce by purge (so that replay is idempotent) - Invoke ExecuteTxn.LocalExecute callbacks on originating CommandStore - misc other [...] new 67e21ec4ba Fix AccordMigrationTest, preclude possible races in topology propagation new 1d5146a22e Improve stability of tests with many epochs new e9a0d8ef8e Accord: Serialization Improvements new 5086846494 Fix: - Only use persisted RedundantBefore for compaction - RouteIndex should index only touches, not Route - Flush RangesForEpoch updates to journal immediately, so we do not rely on the command we are processing succeeding - DurableBefore updates must wait for the epochs to be known locally - Shard.mustWitnessEpoch to support guaranteeing to witness relevant non-topology schema changes - We must propagate RedundantBefore RX shard bounds along with epoch syncs - [...] new a12f0ac9db Accord fixes: - Bad ArrayBuffers recycling logic - RX must ensure dependencies TRANSITIVE_VISIBLE - Permit constructing "antiRange" that spans multiple prefixes - Not computing range CommandSummary IsDep correctly - Truncated commands that aren't shard durable could not repopulate CFK on replay, permitting recovery of another command to make an incorrect decision - NPE on async persist of RX (i.e. supplying no callback) - NPE in Builder.shouldCleanup when durabi [...] new 430c55dbc4 Integrate RX with Cassandra Repair, so that repair safely flushes pending topology and other durability requirements. Also improve: - Introduce DurabilityService - Retire SyncPoint, replace Barrier with Write and RX - MessageType -> enum, restore GetMaxConflict - Standardise backoff logic with WaitStrategy - improve TimeoutStrategy/RetryStrategy specification strings - Forbid KX, remove directKeyDeps - Introduce UniqueTimeService, permitting hlc reservations for [...] new fa1157203f Add initial Accord Overview Doc new ab556ab545 InterceptingExecutor.schedule returns a simulator-unsafe Future new e3982ce0bb When bootstrap fails the retry field is not populated which leads to a NPE new 2999d8f235 Epoch/Topology Garbage Collection new 03053b14bf Implement Nemesis compactor, improve partial compaction new 94f720476e Topology mixup tests should do repairs for host replacement and other required topology changes new 890be9c6e5 Fix: - Decouple command serialization from TableMetadata version; introduce ColumnMetadata ids; gracefully handle missing TableId - DataInputPlus.readLeastSignificantBytes must truncate high bits - Fix RandomPartitioner accord serialization - Fast path stable commits must not override recovery propose/commit decisions regarding visibility of a transaction - RejectBefore must mergeMax, not merge, to ensure we maintain epoch and hlc increasing independently - Bad c [...] new 289dbb7a05 Accord repin: turn pruning back on new fb05d224e7 Fix: - Accord Journal purging was disabled - remove unique_id from schema keyspace - avoid String.format in Compactor hot path - avoid string concatenation on hot path; improve segment compactor partition build efficiency - Partial compaction should update records in place to ensure truncation of discontiguous compactions do not lead to an incorrect field version being used - StoreParticipants.touches behaviour for RX was erroneously modified; should touch all no [...] new 14b6395434 Accord: test fixes and stability improvements * Fix short accord simulation test (seed 0x6bea128ae851724b), ConcurrentModificationException * Increase wait time during closing to avoid Unterminated threads * Increase timeouts, improve test stability * More descriptive output from CQL test * Shorten max CMS delay * Improve future handling in config service new 7f7ac0781b fixing post-rebase checkstyle problems new f519538092 Switch to plain Thread for AccordExecutor, and improve testing for ASYNC mode new 265511f6de Fix CQLSSTableWriterConcurrencyTest new 92212b2255 Fix upgrade TTL tests that fail due to improved error detection in cep-15-accord new e87d5e5020 Various fixes and improvements Improve: - InMemoryJournal compaction should simulate files to ensure we compact the same cohorts of records (so shadowing applied correctly) - Don't update CFK deps if already known - avoid unnecessary heapification of LogGroupTimers - begin removal of Guava (mostly unused) - consider medium path on fast path delayed - add Seekables.indexOf to support faster serialization - unboxed Invariant.requires variant - AsyncChains.addCall [...] new 49740f87d2 Accord: Fix issue where 3 tests would crash consistently in CI new 68aea4b15f Fix simulator after rebase new 19048ef570 Improve Journal table key serialization Also improve: - TxnId serialization - StoreParticipants serialization - compareUnsigned Node.Id for consistency with serialized TxnId new dc841f5481 Fix RouteIndexTest when run with -latest.yaml new 6c6082b5d0 Accord: switch back to 1g heap to keep CI stable new 71aedb2851 Cleanup Accord diff to get it ready for merge new d3439c671a ninja: remove accord_demo.txt and simulator.sh new 4cdbb5cf2a Accord: Test fixes This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (101a6ceafa) \ N -- N -- N refs/heads/cep-15-accord (4cdbb5cf2a) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 227 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .build/build-owasp.xml | 6 +- .build/build-resolver.xml | 4 +- .build/cassandra-deps-template.xml | 2 +- .build/owasp/dependency-check-suppressions.xml | 6 + .build/parent-pom-template.xml | 10 +- .jenkins/Jenkinsfile | 2 +- .snyk | 2 + CHANGES.txt | 22 + build.xml | 2 +- debian/changelog | 55 ++ doc/SASI.md | 14 +- .../examples/BASH/docker-run-cqlsh-quickstart.sh | 2 +- .../examples/CQL/create_ks_trans_repl.cql | 2 +- doc/modules/cassandra/examples/CQL/no_revoke.cql | 10 +- .../CQL/vector-search/vector-search-cycling.cql | 4 +- .../cassandra/examples/RESULTS/2i/2i-check.result | 8 +- .../cassandra/pages/developing/cql/SASI.adoc | 34 +- .../cassandra/pages/developing/cql/changes.adoc | 10 +- .../cassandra/pages/developing/cql/ddl.adoc | 2 +- .../cassandra/pages/developing/cql/functions.adoc | 2 +- .../cassandra/pages/developing/cql/mvs.adoc | 4 +- .../cassandra/pages/developing/cql/security.adoc | 6 +- .../data-modeling/data-modeling_conceptual.adoc | 2 +- .../data-modeling/data-modeling_logical.adoc | 6 +- .../data-modeling/data-modeling_physical.adoc | 6 +- .../data-modeling/data-modeling_queries.adoc | 2 +- .../data-modeling/data-modeling_rdbms.adoc | 2 +- .../data-modeling/data-modeling_refining.adoc | 2 +- .../pages/getting-started/configuring.adoc | 2 +- .../configuration/cass_jvm_options_file.adoc | 2 +- .../cassandra/pages/managing/operating/cdc.adoc | 4 +- .../managing/operating/compaction/tombstones.adoc | 2 +- .../pages/managing/operating/topo_changes.adoc | 2 +- .../managing/tools/sstable/sstableverify.adoc | 4 +- .../pages/reference/cql-commands/create-index.adoc | 4 +- .../cql-commands/create-table-examples.adoc | 2 +- doc/modules/cassandra/pages/reference/static.adoc | 2 +- .../cassandra/pages/troubleshooting/use_tools.adoc | 4 +- .../pages/vector-search/data-modeling.adoc | 4 +- .../cassandra/partials/cql-syntax-legend.adoc | 4 +- .../partials/table-column-definitions.adoc | 2 +- .../cassandra/partials/table-properties.adoc | 4 +- doc/native_protocol_v3.spec | 44 +- doc/native_protocol_v4.spec | 58 +- doc/native_protocol_v5.spec | 96 ++-- pylib/cqlshlib/cqlhandling.py | 29 +- pylib/cqlshlib/test/test_cql_parsing.py | 11 + pylib/cqlshlib/util.py | 21 - .../config/CassandraRelevantProperties.java | 7 + src/java/org/apache/cassandra/config/Config.java | 2 + .../cassandra/config/DatabaseDescriptor.java | 20 +- .../org/apache/cassandra/cql3/QueryProcessor.java | 10 +- .../cassandra/cql3/statements/CQL3CasRequest.java | 21 +- .../cassandra/db/AbstractMutationVerbHandler.java | 28 +- .../org/apache/cassandra/db/ColumnFamilyStore.java | 2 +- .../org/apache/cassandra/db/SystemKeyspace.java | 54 +- .../org/apache/cassandra/db/filter/RowFilter.java | 5 +- .../db/lifecycle/LifecycleTransaction.java | 4 +- .../org/apache/cassandra/db/lifecycle/Tracker.java | 22 +- .../org/apache/cassandra/db/lifecycle/View.java | 21 +- .../db/virtual/ClusterMetadataDirectoryTable.java | 71 ++- .../db/virtual/ClusterMetadataLogTable.java | 34 +- .../db/virtual/CollectionVirtualTableAdapter.java | 19 +- .../org/apache/cassandra/dht/RangeStreamer.java | 9 +- .../cassandra/exceptions/RequestFailureReason.java | 80 ++- ...ion.java => IndexBuildInProgressException.java} | 12 +- .../apache/cassandra/index/IndexStatusManager.java | 12 +- .../cassandra/index/SecondaryIndexManager.java | 15 +- .../index/sai/StorageAttachedIndexGroup.java | 8 +- .../index/sai/disk/StorageAttachedIndexWriter.java | 10 +- .../index/sai/disk/v1/MemtableIndexWriter.java | 6 +- .../iterators/KeyRangeIntersectionIterator.java | 4 + .../sai/plan/StorageAttachedIndexSearcher.java | 4 +- .../cassandra/index/sai/utils/IndexTermType.java | 3 +- .../cassandra/index/sai/utils/PrimaryKey.java | 18 +- .../apache/cassandra/locator/ReplicaLayout.java | 22 +- .../org/apache/cassandra/locator/ReplicaPlan.java | 38 +- .../org/apache/cassandra/locator/ReplicaPlans.java | 57 +- .../apache/cassandra/metrics/KeyspaceMetrics.java | 4 + .../org/apache/cassandra/metrics/TableMetrics.java | 5 + src/java/org/apache/cassandra/net/InboundSink.java | 14 +- .../org/apache/cassandra/service/StorageProxy.java | 2 +- .../apache/cassandra/service/StorageService.java | 32 +- .../cassandra/service/StorageServiceMBean.java | 6 + .../org/apache/cassandra/service/paxos/Paxos.java | 6 + .../cleanup/PaxosCleanupLocalCoordinator.java | 34 +- .../service/paxos/v1/PrepareCallback.java | 28 +- .../org/apache/cassandra/tcm/CMSOperations.java | 27 + .../apache/cassandra/tcm/CMSOperationsMBean.java | 3 + .../org/apache/cassandra/tcm/ClusterMetadata.java | 19 +- src/java/org/apache/cassandra/tcm/Startup.java | 2 +- .../org/apache/cassandra/tcm/Transformation.java | 16 +- .../tcm/listeners/LegacyStateListener.java | 7 +- .../tcm/listeners/PlacementsChangeListener.java | 2 +- .../apache/cassandra/tcm/membership/Directory.java | 31 +- .../apache/cassandra/tcm/membership/Location.java | 12 + .../tcm/migration/CMSInitializationRequest.java | 14 +- .../apache/cassandra/tcm/migration/Election.java | 12 +- .../cassandra/tcm/ownership/DataPlacement.java | 5 + .../cassandra/tcm/ownership/DataPlacements.java | 18 + .../cassandra/tcm/ownership/ReplicaGroups.java | 15 + .../apache/cassandra/tcm/ownership/TokenMap.java | 4 +- .../tcm/ownership/VersionedEndpoints.java | 7 +- .../tcm/sequences/CancelCMSReconfiguration.java | 2 +- .../cassandra/tcm/serialization/Version.java | 3 +- .../cassandra/tcm/transformations/AlterSchema.java | 2 +- .../tcm/transformations/AlterTopology.java | 193 +++++++ src/java/org/apache/cassandra/tools/NodeTool.java | 3 + .../nodetool/{Refresh.java => AlterTopology.java} | 26 +- .../apache/cassandra/tools/nodetool/CMSAdmin.java | 50 ++ .../cassandra/transport/messages/BatchMessage.java | 2 +- .../cassandra/distributed/impl/Instance.java | 9 +- .../cassandra/distributed/impl/IsolatedJmx.java | 14 +- .../distributed/test/RepairDigestTrackingTest.java | 6 +- .../test/TableLevelIncrementalBackupsTest.java | 6 - .../cassandra/distributed/test/TestBaseImpl.java | 6 + .../cql3/AccordInteropMultiNodeTableWalkBase.java | 53 +- .../AccordInteropMultiNodeTokenConflictBase.java | 2 +- .../cql3/AccordInteropSingleNodeTableWalkBase.java | 86 --- .../AccordInteropSingleNodeTokenConflictBase.java | 65 --- .../test/cql3/CasMultiNodeTableWalkBase.java | 3 +- .../FullAccordInteropMultiNodeTableWalkTest.java | 2 - .../FullAccordInteropSingleNodeTableWalkTest.java | 41 -- ...llAccordInteropSingleNodeTokenConflictTest.java | 41 -- ...dReadsAccordInteropSingleNodeTableWalkTest.java | 41 -- ...dsAccordInteropSingleNodeTokenConflictTest.java | 41 -- .../test/cql3/MultiNodeTableWalkBase.java | 74 ++- .../test/cql3/MultiNodeTableWalkTest.java | 116 ---- .../test/cql3/MultiNodeTokenConflictTest.java | 19 +- .../test/cql3/SingleNodeTableWalkTest.java | 51 +- .../test/cql3/SingleNodeTokenConflictTest.java | 9 +- .../distributed/test/cql3/StatefulASTBase.java | 57 +- .../distributed/test/jmx/JMXTestsUtil.java | 21 +- .../distributed/test/log/AlterTopologyTest.java | 267 +++++++++ .../test/log/BounceIndexRebuildTest.java | 10 +- .../test/log/ClusterMetadataDumpTest.java | 116 ++++ .../test/log/FetchLogFromPeersDCTest.java | 60 ++ .../test/log/MetadataChangeSimulationTest.java | 4 +- .../cassandra/distributed/test/log/ModelState.java | 14 + .../distributed/test/log/ReconfigureCMSTest.java | 4 +- .../distributed/test/log/SimulatedOperation.java | 21 + .../test/sai/IndexAvailabilityTest.java | 80 ++- .../test/sai/ReplicaFilteringWithStaticsTest.java | 40 ++ .../cassandra/distributed/test/sai/SAIUtil.java | 22 +- .../test/tcm/FailureDetectorRecomputeTest.java | 113 ++++ .../distributed/upgrade/MixedModePaxosTTLTest.java | 94 ++++ .../upgrade/MixedModePaxosTestBase.java | 55 +- .../accord/AccordJournalCompactionTest.java | 2 +- .../apache/cassandra/harry/gen/BijectionCache.java | 6 +- .../cassandra/harry/model/ASTSingleTableModel.java | 618 ++++++++++++++++----- .../harry/model/ASTSingleTableModelTest.java | 87 ++- .../cassandra/harry/model/BytesPartitionState.java | 153 +++-- .../cassandra/harry/model/PartitionState.java | 6 +- .../cassandra/harry/model/TokenPlacementModel.java | 17 + .../apache/cassandra/harry/util/StringUtils.java | 2 +- .../constraints/ConstraintsSatisfiabilityTest.java | 4 +- .../cassandra/constraints/JsonConstraintTest.java | 2 +- .../constraints/RegexpConstraintTest.java | 2 +- .../cassandra/constraints/TimeConstraintsTest.java | 2 +- .../unit/org/apache/cassandra/cql3/KnownIssue.java | 6 + .../cassandra/cql3/ast/AssignmentOperator.java | 6 + .../apache/cassandra/cql3/ast/CasCondition.java | 11 + .../org/apache/cassandra/cql3/ast/Conditional.java | 5 + .../org/apache/cassandra/cql3/ast/Elements.java | 9 + .../org/apache/cassandra/cql3/ast/Expression.java | 5 + .../org/apache/cassandra/cql3/ast/Literal.java | 6 + .../org/apache/cassandra/cql3/ast/Mutation.java | 147 ++++- .../org/apache/cassandra/cql3/ast/Operator.java | 6 + .../unit/org/apache/cassandra/cql3/ast/Select.java | 15 + .../cassandra/cql3/ast/StandardVisitors.java | 10 + .../org/apache/cassandra/cql3/ast/Statement.java | 7 +- test/unit/org/apache/cassandra/cql3/ast/Value.java | 5 + .../validation/entities/SecondaryIndexTest.java | 13 +- .../apache/cassandra/db/lifecycle/ViewTest.java | 8 +- .../org/apache/cassandra/dht/BootStrapperTest.java | 126 +++-- .../exceptions/RequestFailureReasonTest.java | 47 +- .../cassandra/index/IndexStatusManagerTest.java | 2 +- .../index/sai/cql/AllowFilteringTest.java | 32 ++ .../cassandra/index/sai/cql/ComplexQueryTest.java | 44 ++ .../sai/cql/CompositePartitionKeyIndexTest.java | 99 +++- .../index/sai/cql/StaticColumnIndexTest.java | 48 ++ .../cassandra/io/sstable/CQLSSTableWriterTest.java | 82 +++ .../io/util/ThreadLocalReadAheadBufferTest.java | 16 +- .../SchemaChangeDuringRangeMovementTest.java | 48 +- .../service/AlterTopologyArgParsingTest.java | 149 +++++ .../cassandra/service/reads/DataResolverTest.java | 2 +- .../service/reads/DigestResolverTest.java | 2 +- .../cassandra/service/reads/ReadExecutorTest.java | 2 +- .../reads/repair/AbstractReadRepairTest.java | 1 + .../cassandra/tcm/membership/DirectoryTest.java | 126 +++++ .../InProgressSequenceCancellationTest.java | 36 +- .../cassandra/tcm/sequences/SequencesUtils.java | 72 +++ .../cassandra/tools/nodetool/ClientStatsTest.java | 58 +- .../org/apache/cassandra/utils/ASTGenerators.java | 97 +++- .../cassandra/utils/ImmutableUniqueList.java | 54 +- 195 files changed, 4286 insertions(+), 1434 deletions(-) copy src/java/org/apache/cassandra/index/{IndexNotAvailableException.java => IndexBuildInProgressException.java} (63%) create mode 100644 src/java/org/apache/cassandra/tcm/transformations/AlterTopology.java copy src/java/org/apache/cassandra/tools/nodetool/{Refresh.java => AlterTopology.java} (66%) delete mode 100644 test/distributed/org/apache/cassandra/distributed/test/cql3/AccordInteropSingleNodeTableWalkBase.java delete mode 100644 test/distributed/org/apache/cassandra/distributed/test/cql3/AccordInteropSingleNodeTokenConflictBase.java delete mode 100644 test/distributed/org/apache/cassandra/distributed/test/cql3/FullAccordInteropSingleNodeTableWalkTest.java delete mode 100644 test/distributed/org/apache/cassandra/distributed/test/cql3/FullAccordInteropSingleNodeTokenConflictTest.java delete mode 100644 test/distributed/org/apache/cassandra/distributed/test/cql3/MixedReadsAccordInteropSingleNodeTableWalkTest.java delete mode 100644 test/distributed/org/apache/cassandra/distributed/test/cql3/MixedReadsAccordInteropSingleNodeTokenConflictTest.java delete mode 100644 test/distributed/org/apache/cassandra/distributed/test/cql3/MultiNodeTableWalkTest.java create mode 100644 test/distributed/org/apache/cassandra/distributed/test/log/AlterTopologyTest.java create mode 100644 test/distributed/org/apache/cassandra/distributed/test/log/ClusterMetadataDumpTest.java create mode 100644 test/distributed/org/apache/cassandra/distributed/test/log/FetchLogFromPeersDCTest.java create mode 100644 test/distributed/org/apache/cassandra/distributed/test/tcm/FailureDetectorRecomputeTest.java create mode 100644 test/distributed/org/apache/cassandra/distributed/upgrade/MixedModePaxosTTLTest.java create mode 100644 test/unit/org/apache/cassandra/service/AlterTopologyArgParsingTest.java create mode 100644 test/unit/org/apache/cassandra/tcm/membership/DirectoryTest.java --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org