[
https://issues.apache.org/jira/browse/KUDU-3638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17913098#comment-17913098
]
ASF subversion and git services commented on KUDU-3638:
-------------------------------------------------------
Commit 0ddaac556f7bc7aeb47db740300921d10eabd856 in kudu's branch
refs/heads/master from Alexey Serbin
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=0ddaac556 ]
[tserver] disable KUDU-3486 behavior by default
This is a quick-and-dirty fix to mitigate KUDU-3638.
This patch isn't focusing on properly addressing the issues that
KUDU-3486 has introduced apart from fixing the obvious bug of missing
updates of the Heartbeater::Thread::last_tombstoned_report_time_ field.
Also, with this patch, the functionality introduced with KUDU-3486
is now disabled by default. To re-enable it back, customize the setting
for the --tserver_send_tombstoned_tablets_report_inteval_secs flag,
if needed.
Properly implementing the functionality that KUDU-3486 attempted to add
would be a much more involved patch because there are several items
to address from both the design and implementation standpoints.
Change-Id: I8e32aafab99c74f0ead3ba65aea58ce91d40297c
Reviewed-on: http://gerrit.cloudera.org:8080/22341
Reviewed-by: Abhishek Chennaka <achenn...@cloudera.com>
Tested-by: Alexey Serbin <ale...@apache.org>
> Deficiences in cleaning up tombstoned tablets lead to flooding logs and high
> CPU usage at Kudu master nodes
> -----------------------------------------------------------------------------------------------------------
>
> Key: KUDU-3638
> URL: https://issues.apache.org/jira/browse/KUDU-3638
> Project: Kudu
> Issue Type: Bug
> Components: master, tserver
> Affects Versions: 1.17.1
> Reporter: Alexey Serbin
> Priority: Major
>
> In the scope of implementing
> [KUDU-3486|https://issues.apache.org/jira/browse/KUDU-3486], a few
> deficiencies have been introduced that manifest themselves at least as the
> following:
> * Tablet servers that host tombstoned replicas of tablets that are part of
> still existing tables send reports on all of them with every incremental
> heartbeat to leader master after about 30 minutes after start or as
> customized by the {{\-\-tserver_send_tombstoned_tablets_report_inteval_secs}}
> flag
> * Leader master would flood its INFO log with messages like below, adding
> same records again and again upon processing every incremental heartbeat from
> tablet servers like mentioned in the item above {noformat}
> ... catalog_manager.cc:5516] TS <ts_UUID> (<ta_node_name>:7050) does not have
> the latest schema for tablet <tablet_UUID> (table <table_name>
> [id=<table_UUID>]). Expected version A got B
> {noformat}
> As a temporary workaround for the issue, set
> {{\-\-tserver_send_tombstoned_tablets_report_inteval_sec=-1}} for tablet
> servers (NOTE: since the flag is runtime by its nature, it's possible to
> address the issue without restarting tablet servers by using the {{kudu
> tserver set_flag}} CLI tool).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
