[ 
https://issues.apache.org/jira/browse/KUDU-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18147726#comment-18147726
 ] 

ASF subversion and git services commented on KUDU-2244:
-------------------------------------------------------

Commit 32653a5767ec1769d6cf48021b649ec1061f93fc in kudu's branch 
refs/heads/master from Alexey Serbin
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=32653a576 ]

[consensus] async cmeta-flushing in RaftConsensus

With this changelist, RequestVote, UpdateReplica, and other
RaftConsensus methods don't hold lock_ and update_lock_ while flushing
Raft consensus metadata to the backing storage. This should help during
Raft election storms and other situations described in [1][2][3][4][5].

The WAL segments are still being flushed/synced while holding
update_lock_ in RaftConsensus::Update() (but lock_ isn't held, though),
and this aspect should be taken care in a separate changelist.

Existing Raft consensus tests pass.  Also, several new tests scenarios
added to cover corner cases introduced by the new functionality:

  * a tablet server crashes after the WAL is synced but before the
    asynchronous cmeta flush completes in RaftConsensus::Update(),
    so the term persisted in the cmeta file is behind the term
    persisted in the latest WAL segments:
      ** RaftConsensusITest.FollowerCrashBeforeCmetaFlushRecovers

  * when RaftConsensus::Shutdown() is called while a cmeta flush is still
    pending, the voting information is correctly persisted to disk before
    Shutdown() returns:
      ** RaftConsensusQuorumTest.PendingVoteInfoPersistedOnShutdown
      ** RaftConsensusQuorumTest.TombstonedVotePersistsOnShutdown

  * when a duplicate vote request arrives from a peer that's been
    granted 'yes' vote already, the response is sent only after the
    information on the granted 'yes' vote is persisted in the
    replica's consensus metadata file on disk:
      ** RaftConsensusQuorumTest.DuplicateRequestVoteWhileCmetaFlushPending

  * even with asynchronous cmeta flushing, the metadata is flushed
    exactly once per Raft term advance, and repeated RaftConsensus::Update()
    do not trigger additional flushes:
      ** RaftConsensusQuorumTest.FollowerCmetaFlushedExactlyOncePerTermAdvance

I ran modified same_tablet_concurrent_writes-itest before and after the
update under perf:

  * perf record -e raw_syscalls:* -- \
      ./bin/same_tablet_concurrent_writes-itest \
      --gtest_filter=SameTabletConcurrentWritesTest.InsertsOnly
  * perf trace --input perf.data --summary > perf-trace.out 2>&1

The collected data came through post-processing using awk to find
the amount of time spent by Raft and RPC worker threads on futex(),
fsync(), and fdatasync() syscalls.  The results show significant
reduction in time RPC and Raft worker threads spent waiting on
futex (e.g. waiting on simple_spinlock), and that was even without
injection of extra delays into fsync()/fdatasync() paths.

NOTE: it's expected that Raft workers now spend a bit more time
      on fsync()/fdatasync() calls, where the delta is transferred
      from RPC workers to Raft workers

      Time spent on futex() syscall (ms)
------------------------------------------------
Thread Type            |   Before  |   After
------------------------------------------------
raft worker, total sum |   95733.6 |   58460.8
raft worker, total max |   13859.7 |    5443.3
raft worker, max max   |     500.8 |     500.5
                       |           |
rpc worker, total sum  | 2720960.0 | 1028690.0
rpc worker, total max  |   28678.1 |    5879.7
rpc worker, max max    |   24346.2 |    5141.0

      Time spent on fdatasync() syscall (ms)
------------------------------------------------
Thread Type            |   Before  |   After
------------------------------------------------
raft worker, total sum |       7.7 |      17.1
raft worker, total max |       3.4 |       3.3
raft worker, max max   |       3.4 |       3.3
                       |           |
rpc worker, total sum  |      40.5 |      30.4
rpc worker, total max  |       5.3 |       5.6
rpc worker, max max    |       2.8 |       3.1

      Time spent on fsync() syscall (ms)
------------------------------------------------
Thread Type            |   Before  |   After
------------------------------------------------
raft worker, total sum |       2.6 |      12.2
raft worker, total max |       1.1 |       1.9
raft worker, max max   |       1.1 |       1.9
                       |           |
rpc worker, total sum  |      25.3 |      22.0
rpc worker, total max  |       3.6 |       3.7
rpc worker, max max    |       1.8 |       1.9

[1] https://issues.apache.org/jira/browse/KUDU-1064
[2] https://issues.apache.org/jira/browse/KUDU-2175
[3] https://issues.apache.org/jira/browse/KUDU-2244
[4] https://issues.apache.org/jira/browse/KUDU-2370
[5] https://issues.apache.org/jira/browse/KUDU-2708

Change-Id: Ia11b326a96da40735f87decb04b2db3229e0a59b
Assisted-by: Cursor <[email protected]> (Claude Sonnet 4.6)
Reviewed-on: https://gerrit.cloudera.org/25160
Tested-by: Alexey Serbin <[email protected]>
Reviewed-by: Marton Greber <[email protected]>
Reviewed-by: Zoltan Martonka <[email protected]>


> spinlock contention in raft_consensus
> -------------------------------------
>
>                 Key: KUDU-2244
>                 URL: https://issues.apache.org/jira/browse/KUDU-2244
>             Project: Kudu
>          Issue Type: Improvement
>          Components: consensus
>            Reporter: Andrew Wong
>            Priority: Major
>              Labels: performance
>
> I was going through the logs of a cluster that was seeing a bunch of 
> kernel_stack_watchdog traces, and the slowness seemed to be caused by a lot 
> of activity in consensus requests. E.g.
> W1214 18:57:29.514219 36138 kernel_stack_watchdog.cc:145] Thread 36317 stuck 
> at 
> /data/jenkins/workspace/generic-package-centos64-7-0-impala/topdir/BUILD/kudu-1.3.0-cdh5.11.0/src/kudu/rpc/outbound_call.cc:192
>  for 123ms:
> Kernel stack:
> [<ffffffff810b7ef5>] sys_sched_yield+0x65/0xd0
> [<ffffffff81645909>] system_call_fastpath+0x16/0x1b
> [<ffffffffffffffff>] 0xffffffffffffffff
> User stack:
>     @     0x7f72fab92057  __GI___sched_yield
>     @          0x19498bf  kudu::Thread::StartThread()
>     @          0x1952e7d  kudu::ThreadPool::CreateThreadUnlocked()
>     @          0x19534d3  kudu::ThreadPool::Submit()
>     @          0x1953a27  kudu::ThreadPool::SubmitFunc()
>     @          0x1953ecb  kudu::ThreadPool::SubmitClosure()
>     @           0x9c94ec  kudu::consensus::RaftConsensus::ElectionCallback()
>     @           0x9e6032  kudu::consensus::LeaderElection::CheckForDecision()
>     @           0x9e78c3  
> kudu::consensus::LeaderElection::VoteResponseRpcCallback()
>     @           0xa8b137  kudu::rpc::OutboundCall::CallCallback()
>     @           0xa8c2bc  kudu::rpc::OutboundCall::SetResponse()
>     @           0xa822c0  kudu::rpc::Connection::HandleCallResponse()
>     @           0xa83ffc  ev::base<>::method_thunk<>()
>     @          0x198a07f  ev_invoke_pending
>     @          0x198af71  ev_run
>     @           0xa5e049  kudu::rpc::ReactorThread::RunThread()
> So it seemed to be cause by some slowness in getting threads. Upon perusing 
> the logs a bit more, there were a sizable number of spinlock profiling traces:
> W1214 18:54:27.897955 36379 rpcz_store.cc:238] Trace:
> 1214 18:54:26.766922 (+     0us) service_pool.cc:143] Inserting onto call 
> queue
> 1214 18:54:26.771135 (+  4213us) service_pool.cc:202] Handling call
> 1214 18:54:26.771138 (+     3us) raft_consensus.cc:1126] Updating replica for 
> 0 ops
> 1214 18:54:27.897699 (+1126561us) raft_consensus.cc:1165] Early marking 
> committed up to index 0
> 1214 18:54:27.897700 (+     1us) raft_consensus.cc:1170] Triggering prepare 
> for 0 ops
> 1214 18:54:27.897701 (+     1us) raft_consensus.cc:1282] Marking committed up 
> to 1766
> 1214 18:54:27.897702 (+     1us) raft_consensus.cc:1332] Filling consensus 
> response to leader.
> 1214 18:54:27.897736 (+    34us) spinlock_profiling.cc:255] Waited 991 ms on 
> lock 0x120b3540. stack: 00000000019406c5 00000000009c60d7 00000000009c75f7 
> 00000000007dc628 0000000000a7adfc 0000000000a7b9cd 000000000194d059 
> 00007f72fbcc2dc4 00007f72fabad1cc ffffffffffffffff
> 1214 18:54:27.897737 (+     1us) raft_consensus.cc:1327] UpdateReplicas() 
> finished
> 1214 18:54:27.897741 (+     4us) inbound_call.cc:130] Queueing success 
> response
> Metrics: {"spinlock_wait_cycles":2478395136}
> Each of the traces noted on the order of 500-1000ms of waiting on spinlocks. 
> Upon looking at raft_consensus.cc, it seems we're holding a spinlock 
> (update_lock_) while we call RaftConsensus::UpdateReplica(), which according 
> to its header, "won't return until all operations have been stored in the log 
> and all Prepares() have been completed". While locking may be necessary, it's 
> worth considering using a different kind of lock here.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to