[ 
https://issues.apache.org/jira/browse/KUDU-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18147727#comment-18147727
 ] 

ASF subversion and git services commented on KUDU-2370:
-------------------------------------------------------

Commit 32653a5767ec1769d6cf48021b649ec1061f93fc in kudu's branch 
refs/heads/master from Alexey Serbin
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=32653a576 ]

[consensus] async cmeta-flushing in RaftConsensus

With this changelist, RequestVote, UpdateReplica, and other
RaftConsensus methods don't hold lock_ and update_lock_ while flushing
Raft consensus metadata to the backing storage. This should help during
Raft election storms and other situations described in [1][2][3][4][5].

The WAL segments are still being flushed/synced while holding
update_lock_ in RaftConsensus::Update() (but lock_ isn't held, though),
and this aspect should be taken care in a separate changelist.

Existing Raft consensus tests pass.  Also, several new tests scenarios
added to cover corner cases introduced by the new functionality:

  * a tablet server crashes after the WAL is synced but before the
    asynchronous cmeta flush completes in RaftConsensus::Update(),
    so the term persisted in the cmeta file is behind the term
    persisted in the latest WAL segments:
      ** RaftConsensusITest.FollowerCrashBeforeCmetaFlushRecovers

  * when RaftConsensus::Shutdown() is called while a cmeta flush is still
    pending, the voting information is correctly persisted to disk before
    Shutdown() returns:
      ** RaftConsensusQuorumTest.PendingVoteInfoPersistedOnShutdown
      ** RaftConsensusQuorumTest.TombstonedVotePersistsOnShutdown

  * when a duplicate vote request arrives from a peer that's been
    granted 'yes' vote already, the response is sent only after the
    information on the granted 'yes' vote is persisted in the
    replica's consensus metadata file on disk:
      ** RaftConsensusQuorumTest.DuplicateRequestVoteWhileCmetaFlushPending

  * even with asynchronous cmeta flushing, the metadata is flushed
    exactly once per Raft term advance, and repeated RaftConsensus::Update()
    do not trigger additional flushes:
      ** RaftConsensusQuorumTest.FollowerCmetaFlushedExactlyOncePerTermAdvance

I ran modified same_tablet_concurrent_writes-itest before and after the
update under perf:

  * perf record -e raw_syscalls:* -- \
      ./bin/same_tablet_concurrent_writes-itest \
      --gtest_filter=SameTabletConcurrentWritesTest.InsertsOnly
  * perf trace --input perf.data --summary > perf-trace.out 2>&1

The collected data came through post-processing using awk to find
the amount of time spent by Raft and RPC worker threads on futex(),
fsync(), and fdatasync() syscalls.  The results show significant
reduction in time RPC and Raft worker threads spent waiting on
futex (e.g. waiting on simple_spinlock), and that was even without
injection of extra delays into fsync()/fdatasync() paths.

NOTE: it's expected that Raft workers now spend a bit more time
      on fsync()/fdatasync() calls, where the delta is transferred
      from RPC workers to Raft workers

      Time spent on futex() syscall (ms)
------------------------------------------------
Thread Type            |   Before  |   After
------------------------------------------------
raft worker, total sum |   95733.6 |   58460.8
raft worker, total max |   13859.7 |    5443.3
raft worker, max max   |     500.8 |     500.5
                       |           |
rpc worker, total sum  | 2720960.0 | 1028690.0
rpc worker, total max  |   28678.1 |    5879.7
rpc worker, max max    |   24346.2 |    5141.0

      Time spent on fdatasync() syscall (ms)
------------------------------------------------
Thread Type            |   Before  |   After
------------------------------------------------
raft worker, total sum |       7.7 |      17.1
raft worker, total max |       3.4 |       3.3
raft worker, max max   |       3.4 |       3.3
                       |           |
rpc worker, total sum  |      40.5 |      30.4
rpc worker, total max  |       5.3 |       5.6
rpc worker, max max    |       2.8 |       3.1

      Time spent on fsync() syscall (ms)
------------------------------------------------
Thread Type            |   Before  |   After
------------------------------------------------
raft worker, total sum |       2.6 |      12.2
raft worker, total max |       1.1 |       1.9
raft worker, max max   |       1.1 |       1.9
                       |           |
rpc worker, total sum  |      25.3 |      22.0
rpc worker, total max  |       3.6 |       3.7
rpc worker, max max    |       1.8 |       1.9

[1] https://issues.apache.org/jira/browse/KUDU-1064
[2] https://issues.apache.org/jira/browse/KUDU-2175
[3] https://issues.apache.org/jira/browse/KUDU-2244
[4] https://issues.apache.org/jira/browse/KUDU-2370
[5] https://issues.apache.org/jira/browse/KUDU-2708

Change-Id: Ia11b326a96da40735f87decb04b2db3229e0a59b
Assisted-by: Cursor <[email protected]> (Claude Sonnet 4.6)
Reviewed-on: https://gerrit.cloudera.org/25160
Tested-by: Alexey Serbin <[email protected]>
Reviewed-by: Marton Greber <[email protected]>
Reviewed-by: Zoltan Martonka <[email protected]>


> Allow accessing consensus metadata during flush/sync
> ----------------------------------------------------
>
>                 Key: KUDU-2370
>                 URL: https://issues.apache.org/jira/browse/KUDU-2370
>             Project: Kudu
>          Issue Type: Improvement
>          Components: consensus, perf
>    Affects Versions: 1.8.0
>            Reporter: Todd Lipcon
>            Priority: Major
>
> In some cases when disks are overloaded or starting to go bad, flushing 
> consensus metadata can take a significant amount of time. Currently, we hold 
> the RaftConsensus::lock_ for the duration of things like voting or changing 
> term, which blocks other requests such as writes or UpdateConsensus calls. 
> There are certainly some cases where exposing "dirty" (non-durable) cmeta is 
> illegal from a Raft perspectives, but there are other cases where it is safe. 
> For example:
> - assume we receive a Write request, and we see that cmeta is currently busy 
> flushing a change that marks the local replica as a FOLLOWER. In that case, 
> if we wait on the lock, when we eventually acquire it, we'll just reject the 
> request anyway. We might as well reject it immediately.
> - Assume we receive a Write request, and we see that cmeta is currently 
> flushing a change that will mark the local replica as a LEADER in the next 
> term. CheckLeadershipAndBindTerm can safely bind to the upcoming term rather 
> than blocking until the flush completes.
> - Assume we recieve an UpdateConsensus or Vote request for term N, and we see 
> that we're currently flushing a change to term M > N. I think it's safe to 
> reject the request even though the new term isn't yet durable.
> Probably a few other cases here where it's safe to act on not-yet-durable 
> info.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to