[
https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
graham sanderson updated CASSANDRA-7546:
----------------------------------------
Attachment: 7546.20_5.txt
I've add 7646.20_5.txt which is the same as 7546_20.4.txt but with a minor
change that allows it to function correctly to close to the full 32 bits of
time range vs 31 bits.
# Any thoughts on metrics?
I'm thinking a simple CF (and rolled up KS) metric which simply counts number
of highly contented rows over time. Note, we do know when a row was partially
contented, but I don't know that we can assign a meaningful value between 0 &
1. Note, we could do a ratio of good vs bad rows on flush, but I think the raw
count is more interesting
# Note, I plan to move the static {} block at the top to a test case for sanity
checking - it doesn't belong mixed in the code... Once we're all set I'll
submit an actual patch for 2.0.x and 2.1.x - should we patch this in 1.1/1.2
also?
# Any other thoughts? I'd like to start testing this (but don't want to do so
if it you want to make major changes). I'll test on top of 2.0.10 in beta with
our code and cassandra stress (hopefully some scenarios you have in 2.1 both
with a node down for hinting and not), and maybe after that with the
tracking/metric on but the synchronized off in production just to check that it
exactly detects our hint storms and nothing else in production (we have no
application tables that should be heavily contented on the partition level).
I'll make and test a patch on 2.1 also, however I'll have to finish testing on
2.0.x before I can upgrade a (fast h/w) cluster to 2.1
> AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-7546
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7546
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: graham sanderson
> Assignee: graham sanderson
> Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt,
> 7546.20_4.txt, 7546.20_5.txt, 7546.20_alt.txt, suggestion1.txt,
> suggestion1_21.txt
>
>
> In order to preserve atomicity, this code attempts to read, clone/update,
> then CAS the state of the partition.
> Under heavy contention for updating a single partition this can cause some
> fairly staggering memory growth (the more cores on your machine the worst it
> gets).
> Whilst many usage patterns don't do highly concurrent updates to the same
> partition, hinting today, does, and in this case wild (order(s) of magnitude
> more than expected) memory allocation rates can be seen (especially when the
> updates being hinted are small updates to different partitions which can
> happen very fast on their own) - see CASSANDRA-7545
> It would be best to eliminate/reduce/limit the spinning memory allocation
> whilst not slowing down the very common un-contended case.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
