[
https://issues.apache.org/jira/browse/KUDU-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18147724#comment-18147724
]
ASF subversion and git services commented on KUDU-1064:
-------------------------------------------------------
Commit 32653a5767ec1769d6cf48021b649ec1061f93fc in kudu's branch
refs/heads/master from Alexey Serbin
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=32653a576 ]
[consensus] async cmeta-flushing in RaftConsensus
With this changelist, RequestVote, UpdateReplica, and other
RaftConsensus methods don't hold lock_ and update_lock_ while flushing
Raft consensus metadata to the backing storage. This should help during
Raft election storms and other situations described in [1][2][3][4][5].
The WAL segments are still being flushed/synced while holding
update_lock_ in RaftConsensus::Update() (but lock_ isn't held, though),
and this aspect should be taken care in a separate changelist.
Existing Raft consensus tests pass. Also, several new tests scenarios
added to cover corner cases introduced by the new functionality:
* a tablet server crashes after the WAL is synced but before the
asynchronous cmeta flush completes in RaftConsensus::Update(),
so the term persisted in the cmeta file is behind the term
persisted in the latest WAL segments:
** RaftConsensusITest.FollowerCrashBeforeCmetaFlushRecovers
* when RaftConsensus::Shutdown() is called while a cmeta flush is still
pending, the voting information is correctly persisted to disk before
Shutdown() returns:
** RaftConsensusQuorumTest.PendingVoteInfoPersistedOnShutdown
** RaftConsensusQuorumTest.TombstonedVotePersistsOnShutdown
* when a duplicate vote request arrives from a peer that's been
granted 'yes' vote already, the response is sent only after the
information on the granted 'yes' vote is persisted in the
replica's consensus metadata file on disk:
** RaftConsensusQuorumTest.DuplicateRequestVoteWhileCmetaFlushPending
* even with asynchronous cmeta flushing, the metadata is flushed
exactly once per Raft term advance, and repeated RaftConsensus::Update()
do not trigger additional flushes:
** RaftConsensusQuorumTest.FollowerCmetaFlushedExactlyOncePerTermAdvance
I ran modified same_tablet_concurrent_writes-itest before and after the
update under perf:
* perf record -e raw_syscalls:* -- \
./bin/same_tablet_concurrent_writes-itest \
--gtest_filter=SameTabletConcurrentWritesTest.InsertsOnly
* perf trace --input perf.data --summary > perf-trace.out 2>&1
The collected data came through post-processing using awk to find
the amount of time spent by Raft and RPC worker threads on futex(),
fsync(), and fdatasync() syscalls. The results show significant
reduction in time RPC and Raft worker threads spent waiting on
futex (e.g. waiting on simple_spinlock), and that was even without
injection of extra delays into fsync()/fdatasync() paths.
NOTE: it's expected that Raft workers now spend a bit more time
on fsync()/fdatasync() calls, where the delta is transferred
from RPC workers to Raft workers
Time spent on futex() syscall (ms)
------------------------------------------------
Thread Type | Before | After
------------------------------------------------
raft worker, total sum | 95733.6 | 58460.8
raft worker, total max | 13859.7 | 5443.3
raft worker, max max | 500.8 | 500.5
| |
rpc worker, total sum | 2720960.0 | 1028690.0
rpc worker, total max | 28678.1 | 5879.7
rpc worker, max max | 24346.2 | 5141.0
Time spent on fdatasync() syscall (ms)
------------------------------------------------
Thread Type | Before | After
------------------------------------------------
raft worker, total sum | 7.7 | 17.1
raft worker, total max | 3.4 | 3.3
raft worker, max max | 3.4 | 3.3
| |
rpc worker, total sum | 40.5 | 30.4
rpc worker, total max | 5.3 | 5.6
rpc worker, max max | 2.8 | 3.1
Time spent on fsync() syscall (ms)
------------------------------------------------
Thread Type | Before | After
------------------------------------------------
raft worker, total sum | 2.6 | 12.2
raft worker, total max | 1.1 | 1.9
raft worker, max max | 1.1 | 1.9
| |
rpc worker, total sum | 25.3 | 22.0
rpc worker, total max | 3.6 | 3.7
rpc worker, max max | 1.8 | 1.9
[1] https://issues.apache.org/jira/browse/KUDU-1064
[2] https://issues.apache.org/jira/browse/KUDU-2175
[3] https://issues.apache.org/jira/browse/KUDU-2244
[4] https://issues.apache.org/jira/browse/KUDU-2370
[5] https://issues.apache.org/jira/browse/KUDU-2708
Change-Id: Ia11b326a96da40735f87decb04b2db3229e0a59b
Assisted-by: Cursor <[email protected]> (Claude Sonnet 4.6)
Reviewed-on: https://gerrit.cloudera.org/25160
Tested-by: Alexey Serbin <[email protected]>
Reviewed-by: Marton Greber <[email protected]>
Reviewed-by: Zoltan Martonka <[email protected]>
> /tablets web page can block during leader elections
> ---------------------------------------------------
>
> Key: KUDU-1064
> URL: https://issues.apache.org/jira/browse/KUDU-1064
> Project: Kudu
> Issue Type: Bug
> Components: tserver
> Affects Versions: Private Beta
> Reporter: Todd Lipcon
> Assignee: Alexey Serbin
> Priority: Major
>
> During startup of a server with a lot of tablets, I tried to load /tablets,
> and it took tens of seconds. Finally got a pstack, and the web handler is
> stuck at:
> {code}
> Thread 5 (Thread 0x7f5d0ee82700 (LWP 18625)):
> #0 0x000000000077007c in sys_futex ()
> #1 0x00000000007701bf in base::internal::SpinLockDelay(int volatile*, int,
> int) ()
> #2 0x00000000012b85f0 in base::SpinLock::SlowLock() ()
> #3 0x00000000008e19a2 in
> kudu::consensus::ReplicaState::LockForRead(kudu::unique_lock<kudu::simple_spinlock>*)
> const ()
> #4 0x00000000008d5db5 in
> kudu::consensus::RaftConsensus::CommittedConsensusState() const ()
> #5 0x00000000007225fb in
> kudu::tserver::TabletServerPathHandlers::HandleTabletsPage(kudu::WebCallbackRegistry::WebRequest
> const&, std::basic_stringstream<char, std::char_traits<char>,
> std::allocator<char> >*) ()
> {code}
> I believe the thread holding the lock is blocked on IO to write down votes
> (many threads are hung in IO right now).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)