[
https://issues.apache.org/jira/browse/KUDU-3638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17913863#comment-17913863
]
ASF subversion and git services commented on KUDU-3638:
-------------------------------------------------------
Commit f41a8bd17e7196a4e9bdcf06d7034d7995fe411d in kudu's branch
refs/heads/master from Alexey Serbin
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=f41a8bd17 ]
[tserver] KUDU-3638 follow-up
This patch fixes a typo/mistake in prior changelist [1].
The Heartbeater::Thread::last_tombstoned_report_time_ field has to be
updated only after sending out reports on tombstoned tablet replicas,
otherwise the logic of sending such reports each
--tserver_send_tombstoned_tablets_report_inteval_secs interval isn't
working.
This is a follow-up to [1].
[1] https://github.com/apache/kudu/commit/0ddaac556
Change-Id: I9c5993952c284ab45ff20f88284033a0866281af
Reviewed-on: http://gerrit.cloudera.org:8080/22354
Tested-by: Alexey Serbin <[email protected]>
Reviewed-by: Abhishek Chennaka <[email protected]>
> Deficiencies in cleaning up tombstoned tablets lead to flooding logs and high
> CPU usage at Kudu master nodes
> ------------------------------------------------------------------------------------------------------------
>
> Key: KUDU-3638
> URL: https://issues.apache.org/jira/browse/KUDU-3638
> Project: Kudu
> Issue Type: Bug
> Components: master, tserver
> Affects Versions: 1.17.1
> Reporter: Alexey Serbin
> Priority: Major
>
> In the scope of implementing
> [KUDU-3486|https://issues.apache.org/jira/browse/KUDU-3486], a few
> deficiencies have been introduced that manifest themselves at least as the
> following:
> * Tablet servers that host tombstoned replicas of tablets that are part of
> still existing tables send reports on all of them with every incremental
> heartbeat to leader master after about 30 minutes after start or as
> customized by the {{\-\-tserver_send_tombstoned_tablets_report_inteval_secs}}
> flag
> * Leader master would flood its INFO log with messages like below, adding
> same records again and again upon processing every incremental heartbeat from
> tablet servers like mentioned in the item above {noformat}
> ... catalog_manager.cc:5516] TS <ts_UUID> (<ta_node_name>:7050) does not have
> the latest schema for tablet <tablet_UUID> (table <table_name>
> [id=<table_UUID>]). Expected version A got B
> {noformat}
> As a temporary workaround for the issue, set
> {{\-\-tserver_send_tombstoned_tablets_report_inteval_sec=-1}} for tablet
> servers (NOTE: since the flag is runtime by its nature, it's possible to
> address the issue without restarting tablet servers by using the {{kudu
> tserver set_flag}} CLI tool).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
