[
https://issues.apache.org/jira/browse/CASSANDRA-1991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271030#comment-13271030
]
Jonathan Ellis commented on CASSANDRA-1991:
-------------------------------------------
Let me see if I can summarize the problem here:
We grab the writelock to (1) make sure only one thread flushes a given memtable
and (2) make sure that we can mark the replay position in the sstable we flush
accurately, that is, that we won't replay mutations on restart that are already
in an sstable, which as Sylvain notes is critical for counters.
The mechanism behind (2) is submitting a getContext task to the commitlog,
while our writelock blocks out any other writes from happening -- Table.apply
grabs the readlock before attempting to submit a mutation to the commitlog.
Basically, getContext within the writelock "drains" the commitlog queue
temporarily.
The reason we need to do this is, Table.apply throws the mutation at the
commitlog and then continues on to update memtables, indexes, etc. This
decoupling is normally a Good Thing; otherwise (if you wait for the commitlog
append before continuing), you serialize all your writer threads on the
commitlog one. But it's a problem if the commitlog gets behind and then you
have to wait for it to drain -- for all the mutations entered in your memtable
prior to the flush, to actually get appended to the commitlog -- before
continuing.
I can think of a few ways to ameliorate this:
# Change switchLock to a "fair" locking policy. Otherwise readers can continue
to grab the lock while the flushing thread is trying to grab the write, letting
the commitlog queue continue to grow. (We tried a "fair" policy early on in
Cassandra and found the performance hit too great; maybe we've gained enough in
other areas to make up for this.)
# Wait for the commitlog write to succeed, before continuing on with the rest
of Table.apply, so there is never any large backlog to worry about. Basically,
some form of CASSANDRA-3578.
# Don't block for the context result while holding the writeLock. We need the
context when we actually write out the sstables, and when we discard/recycle
completed segments post-flush, but the actual memtable switch doesn't care. I
think we could just replace the context, with a Future. So the flush thread
will wait for everything it is writing to be appended to the commitlog, but
that wouldn't need to block new commitlog activity since it has a "handle" to
the CL progress in the future we've created.
#3 is pretty non-invasive and should be performant. Anyone see any problems
with that?
> CFS.maybeSwitchMemtable() calls CommitLog.instance.getContext(), which may
> block, under flusher lock write lock
> ---------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-1991
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1991
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Peter Schuller
> Assignee: Peter Schuller
> Attachments: 1991-checkpointing-flush.txt, 1991-logchanges.txt,
> 1991-trunk-v2.txt, 1991-trunk.txt, 1991-v3.txt, 1991-v4.txt, 1991-v5.txt,
> 1991-v6.txt, 1991-v7.txt, 1991-v8.txt, 1991-v9.txt, trigger.py
>
>
> While investigate CASSANDRA-1955 I realized I was seeing very poor latencies
> for reasons that had nothing to do with flush_writers, even when using
> periodic commit log mode (and flush writers set ridiculously high, 500).
> It turns out writes blocked were slow because Table.apply() was spending lots
> of time (I can easily trigger seconds on moderate work-load) trying to
> acquire a flusher lock read lock ("flush lock millis" log printout in the
> logging patch I'll attach).
> That in turns is caused by CFS.maybeSwitchMemtable() which acquires the
> flusher lock write lock.
> Bisecting further revealed that the offending line of code that blocked was:
> final CommitLogSegment.CommitLogContext ctx =
> writeCommitLog ? CommitLog.instance.getContext() : null;
> Indeed, CommitLog.getContext() simply returns currentSegment().getContext(),
> but does so by submitting a callable on the service executor. So
> independently of flush writers, this can block all (global, for all cf:s)
> writes very easily, and does.
> I'll attach a file that is an independent Python script that triggers it on
> my macos laptop (with an intel SSD, which is why I was particularly
> surprised) (it assumes CPython, out-of-the-box-or-almost Cassandra on
> localhost that isn't in a cluster, and it will drop/recreate a keyspace
> called '1955').
> I'm also attaching, just FYI, the patch with log entries that I used while
> tracking it down.
> Finally, I'll attach a patch with a suggested solution of keeping track of
> the latest commit log with an AtomicReference (as an alternative to
> synchronizing all access to segments). With that patch applied, latencies are
> not affected by my trigger case like they were before. There are some
> sub-optimal > 100 ms cases on my test machine, but for other reasons. I'm no
> longer able to trigger the extremes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
