Our foreignid is unique idetifier and we did check for wide partitions; cfhistorgrams show all partitions are evenly sized:
Percentile SSTables Write Latency Read Latency Partition Size Cell Count (micros) (micros) (bytes) 50% 0.00 29.52 0.00 1916 12 75% 0.00 42.51 0.00 2299 12 95% 0.00 61.21 0.00 2759 14 98% 0.00 73.46 0.00 2759 17 99% 0.00 88.15 0.00 2759 17 Min 0.00 9.89 0.00 150 2 Max 0.00 88.15 0.00 7007506 42510 any thing else that we can check? On Wednesday, July 18, 2018, 10:44:29 PM CDT, wxn...@zjqunshuo.com <wxn...@zjqunshuo.com> wrote: #yiv9106249046 body {line-height:1.5;}#yiv9106249046 blockquote {margin-top:0px;margin-bottom:0px;margin-left:0.5em;}#yiv9106249046 p {margin-top:0px;margin-bottom:0px;}#yiv9106249046 div.yiv9106249046foxdiv20180719114102755131 {}#yiv9106249046 body {font-size:10.5pt;color:rgb(0, 0, 0);line-height:1.5;}Your partition key is foreignid. You may have a large partition. Why not use foreignid+timebucket as partition key? From: learner dbaDate: 2018-07-19 01:48To: User cassandra.apache.orgSubject: Timeout for only one keyspace in clusterHi, We have a cluster with multiple keyspaces. All queries are performing good but write operation on few tables in one specific keyspace gets write timeout. Table has counter column and counter update query times out always. Any idea? CREATE TABLE x.y ( foreignid uuid, timebucket text, key text, timevalue int, value counter, PRIMARY KEY (foreignid, timebucket, key, timevalue) ) WITH CLUSTERING ORDER BY (timebucket ASC, key ASC, timevalue ASC) AND bloom_filter_fp_chance = 0.01 AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} AND comment = '' AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'} AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'} AND crc_check_chance = 1.0 AND dclocal_read_repair_chance = 0.1 AND default_time_to_live = 0 AND gc_grace_seconds = 864000 AND max_index_interval = 2048 AND memtable_flush_period_in_ms = 0 AND min_index_interval = 128 AND read_repair_chance = 0.0 AND speculative_retry = '99PERCENTILE'; Query and Error: UPDATE x.y SET value = value + 1 where foreignid = ? AND timebucket = ? AND key = ? AND timevalue = ?, err = &errors.errorString{s:\"gocql: no response received from cassandra within timeout period I verified CL=local_serialWe had been working on this issue for many days; any help will be much appreciated.