[
https://issues.apache.org/jira/browse/CASSANDRA-21664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18116444#comment-18116444
]
Jon Haddad commented on CASSANDRA-21664:
----------------------------------------
Awesome. I'm + .999, just a few things to guard against future regressions w/
some tests.
1. The delayed-save feature can quietly turn itself off
The code sets a flag that means "a save is already scheduled." Then it
schedules the save. If that scheduling step fails, for example while the node
is shutting down, the flag stays stuck on. After that, the node thinks a save
is always pending, so it never schedules another one. The feature stops working
for the rest of the node's life.
Your data is still safe, because the backup log still catches everything. But
the speed feature is dead, and there is no log line or counter to tell you. An
operator cannot tell a healthy node from a broken one. The fix is small: clear
the flag if scheduling fails, and add a log line and a counter so the behavior
is visible.
2. A new setting is missing from one of the two config files.
Cassandra ships two example config files. The new
schema_flush_coalescing_window setting was added to one but not the other.
3. The most important new warning is never tested.
The patch warns you when the schema reaches 75% of its size limit. The tests
check a tiny schema (no warning) and a giant one (fails completely). They never
check the middle zone, which is the exact case the warning was built for. So if
someone later broke that warning, every test would still pass. The fix is to
add a test with a schema in that middle zone and confirm the warning fires.
> Make schema changes cost O(1) in the number of tables and fail loudly at
> schema limits
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21664
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21664
> Project: Apache Cassandra
> Issue Type: Improvement
> Components: Cluster/Schema
> Reporter: Patrick McFadin
> Assignee: Patrick McFadin
> Priority: Normal
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h2. What
> Creating 10,000 tables takes 28 minutes (ManyTablesScalingTest, single node
> in-JVM). Per CREATE TABLE at small N: ~121 ms, of which 92 ms is a
> synchronous flush of every system_schema table on the log follower
> (SchemaKeyspace.applyChanges -> flush) and 22ms a blocking TCM log flush. On
> top of that, every schema change diffs the whole schema and rebuilds
> whole-keyspace maps, so cost grows with the number of existing tables: 112 ->
> 234 ms per statement from 0 to 10,000 tables.
> Past ~31,000 minimal tables the serialised cluster metadata exceeds
> max_mutation_size and the TCM snapshot silently stops being written (WARN
> only). Table-count guardrails ship disabled.
> h2. Change
> * Schema diffs and keyspace/table map updates touch only what changed:
> Keyspaces.diff, Tables (persistent map), DistributedSchema table map,
> Keyspaces.withAddedOrUpdated, DistributedSchema.validate, ThreadLocalMeter
> array growth.
> * system_schema flush is coalesced behind a new yaml setting
> schema_flush_coalescing_window (default 1000ms; 0ms restores the synchronous
> flush). system_schema has durable_writes and is rebuilt from the cluster
> metadata log on startup, so delaying the flush cannot lose schema. Drain
> still flushes synchronously.
> * Snapshot store failure is logged at ERROR with a metric
> (TCM.SnapshotStoreFailures), and a WARN fires once serialised metadata passes
> 75% of max_mutation_size.
> * tables_warn_threshold defaults to 1000; tested envelope and per-table cost
> documented.
> * Build: the in-JVM dtest targets' -Xmx8G was overridden by junit's maxmemory
> default of 1024m.
> h2. Result
> Same test, same machine; before = synchronous flush, after = this patch.
> ||N||before||after||per statement, first -> last bucket (after)||
> |1,000|120.9 s|29.9 s|32 -> 31 ms|
> |5,000|675.1 s|204.5 s|32 -> 55 ms|
> |10,000|28.0 min|9.7 min|32 -> 95 ms|
> The fixed per-statement floor is gone. A residual ~6 ms per 1,000 existing
> tables remains (TablesDiff full scans); follow-up.
> h2. Tests
> Scaling assertions at N vs 8N on every touched path; SchemaFlushCoalesceTest;
> SchemaFlushRestartTest (30 tables, stop without drain, restart, all present);
> MetadataSnapshotSizeWarningTest; schema, tcm, guardrail and config suites;
> checkstyle clean.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
