This is an automated email from the ASF dual-hosted git repository. konstantinov pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit a93f0819e322c4c5569e431779d6b71fdbc26fe9 Merge: df24dacc0c ae82efc013 Author: Dmitry Konstantinov <netud...@gmail.com> AuthorDate: Thu Apr 24 12:02:57 2025 +0100 Merge branch 'cassandra-5.0' into trunk * cassandra-5.0: Add LittleEndianMemoryUtil and NativeEndianMemoryUtil, switch memtable-related off-heap objects and Memory to use them and have Little Endian now. Add BE offsets detection on Summary loading. Add test SSTables in an old format with BE offsets in Summary component to LegacySSTableTest. CHANGES.txt | 1 + .../org/apache/cassandra/db/NativeClustering.java | 25 +-- .../apache/cassandra/db/NativeDecoratedKey.java | 9 +- .../org/apache/cassandra/db/rows/NativeCell.java | 33 ++-- .../io/sstable/indexsummary/IndexSummary.java | 12 ++ src/java/org/apache/cassandra/io/util/Memory.java | 144 ++------------ .../utils/memory/LittleEndianMemoryUtil.java | 146 ++++++++++++++ .../apache/cassandra/utils/memory/MemoryUtil.java | 173 +---------------- .../utils/memory/NativeEndianMemoryUtil.java | 214 +++++++++++++++++++++ .../da-500-bti-CompressionInfo.db | Bin 0 -> 207 bytes .../da-500-bti-Data.db | Bin 0 -> 8602 bytes .../da-500-bti-Digest.crc32 | 1 + .../da-500-bti-Filter.db | Bin 0 -> 24 bytes .../da-500-bti-Partitions.db | Bin 0 -> 62 bytes .../da-500-bti-Rows.db | Bin 0 -> 508 bytes .../da-500-bti-Statistics.db | Bin 0 -> 7312 bytes .../da-500-bti-TOC.txt | 8 + .../ma-306-big-CompressionInfo.db | Bin 0 -> 83 bytes .../ma-306-big-Data.db | Bin 0 -> 5221 bytes .../ma-306-big-Digest.crc32 | 1 + .../ma-306-big-Filter.db | Bin 0 -> 24 bytes .../ma-306-big-Index.db | Bin 0 -> 157553 bytes .../ma-306-big-Statistics.db | Bin 0 -> 7046 bytes .../ma-306-big-Summary.db | Bin 0 -> 47 bytes .../ma-306-big-TOC.txt | 8 + .../mb-307-big-CompressionInfo.db | Bin 0 -> 83 bytes .../mb-307-big-Data.db | Bin 0 -> 5232 bytes .../mb-307-big-Digest.crc32 | 1 + .../mb-307-big-Filter.db | Bin 0 -> 24 bytes .../mb-307-big-Index.db | Bin 0 -> 157553 bytes .../mb-307-big-Statistics.db | Bin 0 -> 7058 bytes .../mb-307-big-Summary.db | Bin 0 -> 47 bytes .../mb-307-big-TOC.txt | 8 + .../mc-3113-big-CompressionInfo.db | Bin 0 -> 83 bytes .../mc-3113-big-Data.db | Bin 0 -> 5215 bytes .../mc-3113-big-Digest.crc32 | 1 + .../mc-3113-big-Filter.db | Bin 0 -> 24 bytes .../mc-3113-big-Index.db | Bin 0 -> 157553 bytes .../mc-3113-big-Statistics.db | Bin 0 -> 7070 bytes .../mc-3113-big-Summary.db | Bin 0 -> 47 bytes .../mc-3113-big-TOC.txt | 8 + .../md-31110-big-CompressionInfo.db | Bin 0 -> 83 bytes .../md-31110-big-Data.db | Bin 0 -> 5206 bytes .../md-31110-big-Digest.crc32 | 1 + .../md-31110-big-Filter.db | Bin 0 -> 24 bytes .../md-31110-big-Index.db | Bin 0 -> 157553 bytes .../md-31110-big-Statistics.db | Bin 0 -> 7134 bytes .../md-31110-big-Summary.db | Bin 0 -> 47 bytes .../md-31110-big-TOC.txt | 8 + .../me-31111-big-CompressionInfo.db | Bin 0 -> 83 bytes .../me-31111-big-Data.db | Bin 0 -> 5207 bytes .../me-31111-big-Digest.crc32 | 1 + .../me-31111-big-Filter.db | Bin 0 -> 24 bytes .../me-31111-big-Index.db | Bin 0 -> 157553 bytes .../me-31111-big-Statistics.db | Bin 0 -> 7151 bytes .../me-31111-big-Summary.db | Bin 0 -> 47 bytes .../me-31111-big-TOC.txt | 8 + .../na-40-big-CompressionInfo.db | Bin 0 -> 207 bytes .../na-40-big-Data.db | Bin 0 -> 8587 bytes .../na-40-big-Digest.crc32 | 1 + .../na-40-big-Filter.db | Bin 0 -> 24 bytes .../na-40-big-Index.db | Bin 0 -> 157553 bytes .../na-40-big-Statistics.db | Bin 0 -> 7160 bytes .../na-40-big-Summary.db | Bin 0 -> 47 bytes .../na-40-big-TOC.txt | 8 + .../nb-400-big-CompressionInfo.db | Bin 0 -> 207 bytes .../nb-400-big-Data.db | Bin 0 -> 8620 bytes .../nb-400-big-Digest.crc32 | 1 + .../nb-400-big-Filter.db | Bin 0 -> 24 bytes .../nb-400-big-Index.db | Bin 0 -> 157553 bytes .../nb-400-big-Statistics.db | Bin 0 -> 7177 bytes .../nb-400-big-Summary.db | Bin 0 -> 47 bytes .../nb-400-big-TOC.txt | 8 + .../oa-500-big-CompressionInfo.db | Bin 0 -> 207 bytes .../oa-500-big-Data.db | Bin 0 -> 8513 bytes .../oa-500-big-Digest.crc32 | 1 + .../oa-500-big-Filter.db | Bin 0 -> 24 bytes .../oa-500-big-Index.db | Bin 0 -> 157498 bytes .../oa-500-big-Statistics.db | Bin 0 -> 7312 bytes .../oa-500-big-Summary.db | Bin 0 -> 47 bytes .../oa-500-big-TOC.txt | 8 + .../db/compaction/CompactionsCQLTest.java | 2 +- .../cassandra/io/sstable/LegacySSTableTest.java | 79 +++++--- .../utils/memory/LittleEndianMemoryUtilTest.java | 148 ++++++++++++++ .../utils/memory/NativeEndianMemoryUtilTest.java | 148 ++++++++++++++ 85 files changed, 852 insertions(+), 363 deletions(-) diff --cc CHANGES.txt index 45ea128630,545ba01dab..f076f8a5d1 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,187 -1,10 +1,188 @@@ -5.0.5 +5.1 + * Switch memtable-related off-heap objects to Native Endian and Memory to Little Endian (CASSANDRA-20190) - * SAI marks an index as non-empty when a partial partition/row modifications is flushed due to repair (CASSANDRA-20567) - * SAI fails queries when multiple columns exist and a non-indexed column is a composite with a map (CASSANDRA-19891) + * Change SSTableSimpleScanner to use SSTableReader#openDataReaderForScan (CASSANDRA-20538) + * Automated Repair Inside Cassandra [CEP-37] (CASSANDRA-19918) * Grant permission on keyspaces system_views and system_virtual_schema not possible (CASSANDRA-20171) + * General Purpose Transactions (Accord) [CEP-15] (CASSANDRA-17092) + * Improve performance when getting writePlacementsAllSettled from ClusterMetadata (CASSANDRA-20526) + * Add nodetool command to dump the contents of the system_views.{cluster_metadata_log, cluster_metadata_directory} tables (CASSANDRA-20525) + * Fix TreeMap race in CollectionVirtualTableAdapter causing us to lose rows in the virtual table (CASSANDRA-20524) + * Improve metadata log catch up with inter-DC mutation forwarding (CASSANDRA-20523) + * Support topology-safe changes to Datacenter & Rack for live nodes (CASSANDRA-20528) + * Add SSTableIntervalTree latency metric (CASSANDRA-20502) + * Ignore repetitions of semicolon in CQLSH (CASSANDRA-19956) + * Avoid NPE during cms initialization abort (CASSANDRA-20527) + * Avoid failing queries when epoch changes and replica goes up/down (CASSANDRA-20489) + * Split out truncation record lock (CASSANDRA-20480) + * Throw new IndexBuildInProgressException when queries fail during index build, instead of IndexNotAvailableException (CASSANDRA-20402) + * Fix Paxos repair interrupts running transactions (CASSANDRA-20469) + * Various fixes in constraint framework (CASSANDRA-20481) + * Add support in CAS for -= on numeric types, and fixed improper handling of empty bytes which lead to NPE (CASSANDRA-20477) + * Do not fail to start a node with materialized views after they are turned off in config (CASSANDRA-20452) + * Fix nodetool gcstats output, support human-readable units and more output formats (CASSANDRA-19022) + * Various gossip to TCM upgrade fixes (CASSANDRA-20483) + * Add nodetool command to abort failed nodetool cms initialize (CASSANDRA-20482) + * Repair Paxos for the distributed metadata log when CMS membership changes (CASSANDRA-20467) + * Reintroduce CASSANDRA-17411 in trunk (CASSANDRA-19346) + * Add min/max/mean/percentiles to timer metrics vtable (CASSANDRA-20466) + * Add support for time, date, timestamp types in scalar constraint (CASSANDRA-20274) + * Add regular expression constraint (CASSANDRA-20275) + * Improve constraints autocompletion (CASSANDRA-20341) + * Add JVM version and Cassandra build date to nodetool version -v (CASSANDRA-19721) + * Move all disk error logic to DiskErrorsHandler to enable pluggability (CASSANDRA-20363) * Fix marking an SSTable as suspected and BufferPool leakage in case of a corrupted SSTable read during a compaction (CASSANDRA-20396) + * Add missed documentation for CREATE TABLE LIKE (CASSANDRA-20401) + * Add OCTET_LENGTH constraint (CASSANDRA-20340) + * Reduce memory allocations in miscellaneous places along the hot write path (CASSANDRA-20167) + * Provide keystore_password_file and truststore_password_file options to read credentials from a file (CASSANDRA-13428) + * Unregistering a node should also remove it from tokenMap if it is there and recalculate the placements (CASSANDRA-20346) + * Fix PartitionUpdate.isEmpty deserialization issue to avoid potential EOFException (CASSANDRA-20345) + * Avoid adding LEFT nodes to tokenMap on upgrade from gossip (CASSANDRA-20344) + * Allow empty placements when deserializing cluster metadata (CASSANDRA-20343) + * Reduce heap pressure when initializing CMS (CASSANDRA-20267) + * Paxos Repair: NoSuchElementException on DistributedSchema.getKeyspaceMetadata (CASSANDRA-20320) + * Improve performance of DistributedSchema.validate for large schemas (CASSANDRA-20360) + * Add JSON constraint (CASSANDRA-20273) + * Prevent invalid constraint combinations (CASSANDRA-20330) + * Support CREATE TABLE LIKE WITH INDEXES (CASSANDRA-19965) + * Invalidate relevant prepared statements on every change to TableMetadata (CASSANDRA-20318) + * Add per type max size guardrails (CASSANDRA-19677) + * Make it possible to abort all kinds of multi step operations (CASSANDRA-20217) + * Do not leak non-Java exceptions when calling snapshot operations via JMX (CASSANDRA-20335) + * Implement NOT_NULL constraint (CASSANDRA-20276) + * Improve error messages for constraints (CASSANDRA-20266) + * Add system_views.partition_key_statistics for querying SSTable metadata (CASSANDRA-20161) + * CEP-42 - Add Constraints Framework (CASSANDRA-19947) + * Add table metric PurgeableTombstoneScannedHistogram and a tracing event for scanned purgeable tombstones (CASSANDRA-20132) + * Make sure we can parse the expanded CQL before writing it to the log or sending it to replicas (CASSANDRA-20218) + * Add format_bytes and format_time functions (CASSANDRA-19546) + * Fix error when trying to assign a tuple to target type not being a tuple (CASSANDRA-20237) + * Fail CREATE TABLE LIKE statement if UDTs in target keyspace do not exist or they have different structure from ones in source keyspace (CASSANDRA-19966) + * Support octet_length and length functions (CASSANDRA-20102) + * Make JsonUtils serialize Instant always with the same format (CASSANDRA-20209) + * Port Harry v2 to trunk (CASSANDRA-20200) + * Enable filtering of snapshots on keyspace, table and snapshot name in nodetool listsnapshots (CASSANDRA-20151) + * Create manifest upon loading where it does not exist or enrich it (CASSANDRA-20150) + * Propagate true size of snapshot in SnapshotDetailsTabularData to not call JMX twice in nodetool listsnapshots (CASSANDRA-20149) + * Implementation of CEP-43 - copying a table via CQL by CREATE TABLE LIKE (CASSANDRA-19964) + * Periodically disconnect roles that are revoked or have LOGIN=FALSE set (CASSANDRA-19385) + * AST library for CQL-based fuzz tests (CASSANDRA-20198) + * Support audit logging for JMX operations (CASSANDRA-20128) + * Enable sorting of nodetool status output (CASSANDRA-20104) + * Support downgrading after CMS is initialized (CASSANDRA-20145) + * Deprecate IEndpointSnitch (CASSANDRA-19488) + * Check presence of a snapshot in a case-insensitive manner on macOS platform to prevent hardlinking failures (CASSANDRA-20146) + * Enable JMX server configuration to be in cassandra.yaml (CASSANDRA-11695) + * Parallelized UCS compactions (CASSANDRA-18802) + * Avoid prepared statement invalidation race when committing schema changes (CASSANDRA-20116) + * Restore optimization in MultiCBuilder around building one clustering (CASSANDRA-20129) + * Consolidate all snapshot management to SnapshotManager and introduce SnapshotManagerMBean (CASSANDRA-18111) + * Fix RequestFailureReason constants codes (CASSANDRA-20126) * Introduce SSTableSimpleScanner for compaction (CASSANDRA-20092) + * Include column drop timestamp in alter table transformation (CASSANDRA-18961) + * Make JMX SSL configurable in cassandra.yaml (CASSANDRA-18508) + * Fix cqlsh CAPTURE command to save query results without trace details when TRACING is ON (CASSANDRA-19105) + * Optionally prevent tombstone purging during repair (CASSANDRA-20071) + * Add post-filtering support for the IN operator in SAI queries (CASSANDRA-20025) + * Don’t finish ongoing decommission and move operations during startup (CASSANDRA-20040) + * Nodetool reconfigure cms has correct return code when streaming fails (CASSANDRA-19972) + * Reintroduce RestrictionSet#iterator() optimization around multi-column restrictions (CASSANDRA-20034) + * Explicitly localize strings to Locale.US for internal implementation (CASSANDRA-19953) + * Add -H option for human-friendly output in nodetool compactionhistory (CASSANDRA-20015) + * Fix type check for referenced duration type for nested types (CASSANDRA-19890) + * In simulation tests, correctly set the tokens of replacement nodes (CASSANDRA-19997) + * During TCM upgrade, retain all properties of existing system tables (CASSANDRA-19992) + * Properly cancel in-flight futures and reject requests in EpochAwareDebounce during shutdown (CASSANDRA-19848) + * Provide clearer exception message on failing commitlog_disk_access_mode combinations (CASSANDRA-19812) + * Add total space used for a keyspace to nodetool tablestats (CASSANDRA-19671) + * Ensure Relation#toRestriction() handles ReversedType properly (CASSANDRA-19950) + * Add JSON and YAML output option to nodetool gcstats (CASSANDRA-19771) + * Introduce metadata serialization version V4 (CASSANDRA-19970) + * Allow CMS reconfiguration to work around DOWN nodes (CASSANDRA-19943) + * Make TableParams.Serializer set allowAutoSnapshots and incrementalBackups (CASSANDRA-19954) + * Make sstabledump possible to show tombstones only (CASSANDRA-19939) + * Ensure that RFP queries potentially stale replicas even with only key columns in the row filter (CASSANDRA-19938) + * Allow nodes to change IP address while upgrading to TCM (CASSANDRA-19921) + * Retain existing keyspace params on system tables after upgrade (CASSANDRA-19916) + * Deprecate use of gossip state for paxos electorate verification (CASSANDRA-19904) + * Update dtest-api to 0.0.17 to fix jvm17 crash in jvm-dtests (CASSANDRA-19239) + * Add resource leak test and Update Netty to 4.1.113.Final to fix leak (CASSANDRA-19783) + * Fix incorrect nodetool suggestion when gossip mode is running (CASSANDRA-19905) + * SAI support for BETWEEN operator (CASSANDRA-19688) + * Fix BETWEEN filtering for reversed clustering columns (CASSANDRA-19878) + * Retry if node leaves CMS while committing a transformation (CASSANDRA-19872) + * Add support for NOT operators in WHERE clauses. Fixed Three Valued Logic (CASSANDRA-18584) + * Allow getendpoints for system tables and make sure getNaturalReplicas work for MetaStrategy (CASSANDRA-19846) + * On upgrade, handle pre-existing tables with unexpected table ids (CASSANDRA-19845) + * Reconfigure CMS before assassinate (CASSANDRA-19768) + * Warn about unqualified prepared statement only if it is select or modification statement (CASSANDRA-18322) + * Update legacy peers tables during node replacement (CASSANDRA-19782) + * Refactor ColumnCondition (CASSANDRA-19620) + * Allow configuring log format for Audit Logs (CASSANDRA-19792) + * Support for noboolean rpm (centos7 compatible) packages removed (CASSANDRA-19787) + * Allow threads waiting for the metadata log follower to be interrupted (CASSANDRA-19761) + * Support dictionary lookup for CassandraPasswordValidator (CASSANDRA-19762) + * Disallow denylisting keys in system_cluster_metadata (CASSANDRA-19713) + * Fix gossip status after replacement (CASSANDRA-19712) + * Ignore repair requests for system_cluster_metadata (CASSANDRA-19711) + * Avoid ClassCastException when verifying tables with reversed partitioner (CASSANDRA-19710) + * Always repair the full range when repairing system_cluster_metadata (CASSANDRA-19709) + * Use table-specific partitioners during Paxos repair (CASSANDRA-19714) + * Expose current compaction throughput in nodetool (CASSANDRA-13890) + * CEP-24 Password validation / generation (CASSANDRA-17457) + * Reconfigure CMS after replacement, bootstrap and move operations (CASSANDRA-19705) + * Support querying LocalStrategy tables with any partitioner (CASSANDRA-19692) + * Relax slow_query_log_timeout for MultiNodeSAITest (CASSANDRA-19693) + * Audit Log entries are missing identity for mTLS connections (CASSANDRA-19669) + * Add support for the BETWEEN operator in WHERE clauses (CASSANDRA-19604) + * Replace Stream iteration with for-loop for SimpleRestriction::bindAndGetClusteringElements (CASSANDRA-19679) + * Consolidate logging on trace level (CASSANDRA-19632) + * Expand DDL statements on coordinator before submission to the CMS (CASSANDRA-19592) + * Fix number of arguments of String.format() in various classes (CASSANDRA-19645) + * Remove unused fields from config (CASSANDRA-19599) + * Refactor Relation and Restriction hierarchies (CASSANDRA-19341) + * Raise priority of TCM internode messages during critical operations (CASSANDRA-19517) + * Add nodetool command to unregister LEFT nodes (CASSANDRA-19581) + * Add cluster metadata id to gossip syn messages (CASSANDRA-19613) + * Reduce heap usage occupied by the metrics (CASSANDRA-19567) + * Improve handling of transient replicas during range movements (CASSANDRA-19344) + * Enable debounced internode log requests to be cancelled at shutdown (CASSANDRA-19514) + * Correctly set last modified epoch when combining multistep operations into a single step (CASSANDRA-19538) + * Add new TriggersPolicy configuration to allow operators to disable triggers (CASSANDRA-19532) + * Use Transformation.Kind.id in local and distributed log tables (CASSANDRA-19516) + * Remove period field from ClusterMetadata and metadata log tables (CASSANDRA-19482) + * Enrich system_views.pending_hints vtable with hints sizes (CASSANDRA-19486) + * Expose all dropwizard metrics in virtual tables (CASSANDRA-14572) + * Ensured that PropertyFileSnitchTest do not overwrite cassandra-toploogy.properties (CASSANDRA-19502) + * Add option for MutualTlsAuthenticator to restrict the certificate validity period (CASSANDRA-18951) + * Fix StorageService::constructRangeToEndpointMap for non-distributed keyspaces (CASSANDRA-19255) + * Group nodetool cms commands into single command group (CASSANDRA-19393) + * Register the measurements of the bootstrap process as Dropwizard metrics (CASSANDRA-19447) + * Add LIST SUPERUSERS CQL statement (CASSANDRA-19417) + * Modernize CQLSH datetime conversions (CASSANDRA-18879) + * Harry model and in-JVM tests for partition-restricted 2i queries (CASSANDRA-18275) + * Refactor cqlshmain global constants (CASSANDRA-19201) + * Remove native_transport_port_ssl (CASSANDRA-19397) + * Make nodetool reconfigurecms sync by default and add --cancel to be able to cancel ongoing reconfigurations (CASSANDRA-19216) + * Expose auth mode in system_views.clients, nodetool clientstats, metrics (CASSANDRA-19366) + * Remove sealed_periods and last_sealed_period tables (CASSANDRA-19189) + * Improve setup and initialisation of LocalLog/LogSpec (CASSANDRA-19271) + * Refactor structure of caching metrics and expose auth cache metrics via JMX (CASSANDRA-17062) + * Allow CQL client certificate authentication to work without sending an AUTHENTICATE request (CASSANDRA-18857) + * Extend nodetool tpstats and system_views.thread_pools with detailed pool parameters (CASSANDRA-19289) + * Remove dependency on Sigar in favor of OSHI (CASSANDRA-16565) + * Simplify the bind marker and Term logic (CASSANDRA-18813) + * Limit cassandra startup to supported JDKs, allow higher JDKs by setting CASSANDRA_JDK_UNSUPPORTED (CASSANDRA-18688) + * Standardize nodetool tablestats formatting of data units (CASSANDRA-19104) + * Make nodetool tablestats use number of significant digits for time and average values consistently (CASSANDRA-19015) + * Upgrade jackson to 2.15.3 and snakeyaml to 2.1 (CASSANDRA-18875) + * Transactional Cluster Metadata [CEP-21] (CASSANDRA-18330) + * Add ELAPSED command to cqlsh (CASSANDRA-18861) + * Add the ability to disable bulk loading of SSTables (CASSANDRA-18781) + * Clean up obsolete functions and simplify cql_version handling in cqlsh (CASSANDRA-18787) +Merged from 5.0: + * SAI marks an index as non-empty when a partial partition/row modifications is flushed due to repair (CASSANDRA-20567) + * SAI fails queries when multiple columns exist and a non-indexed column is a composite with a map (CASSANDRA-19891) * Avoid purging deletions in RowFilter when reconciliation is required (CASSANDRA-20541) * Fixed multiple single-node SAI query bugs relating to static columns (CASSANDRA-20338) * Upgrade com.datastax.cassandra:cassandra-driver-core:3.11.5 to org.apache.cassandra:cassandra-driver-core:3.12.1 (CASSANDRA-17231) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org