[
https://issues.apache.org/jira/browse/KUDU-3452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17703001#comment-17703001
]
Xixu Wang commented on KUDU-3452:
---------------------------------
[~aserbin] Thanks for your comments.
{color:#172b4d}1. In my case, the failed node keep unavailable temporarily or a
new tablet server will join the cluster. We should always keep the Kudu cluster
high availability. When find the Kudu cluster has some problems, we will repair
it. But before bringing the Kudu cluster back to high availability (at least 3
tablet servers), there are maybe a period of time that creating table service
will be unavailable. For an online service based on Kudu, that is
intolerable.{color}
{color:#c1c7d0}that allows to keep Kudu data highly available while operators
can bring the failed node back.{color}
2. The online service based on Kudu will create a table with RF=3 periodically.
It always supposes the Kudu cluster available. It is not much possible to
create a table with RF=1 when one of three replicas failed, after that the
failed node is brought back, then alter the table replica factor to 3.
{color:#c1c7d0}Why to create tables with RF=3 then if just a single node
failure renders your data unavailable? Since it doesn't provide a benefit of
high availability, you could create tables with RF=1 and have just one tablet
server in your cluster.{color}
{color:#172b4d}3. In my opinion, create a table with RF=3 when only 2 of 3
tablet servers are alive may be a high risk, but the risk is the same for an
already created table with RF=3, it is better than can not creating tables. We
should keep the Kudu cluster providing service as long as possible even if some
of nodes failed before we have enough time to repair the problem. The service
of reading and writing is available when only 2 of 3 tablet servers alive, the
service of creating table could be available as the same. {color}
{color:#c1c7d0}Allowing to create a table with RF=3 when only two tablet
servers are alive is asking for a disaster, because the table's data will
become non-available when just one more node goes down.{color}
> Support creating three-replicas table or partition when only 2 tservers
> healthy
> -------------------------------------------------------------------------------
>
> Key: KUDU-3452
> URL: https://issues.apache.org/jira/browse/KUDU-3452
> Project: Kudu
> Issue Type: Improvement
> Reporter: Xixu Wang
> Priority: Major
>
> h1. Background
> In my case, every day a new Kudu table (called: history_data_table) will be
> created to store history data and a new partition for another table (called:
> business_data_table) to be ready to store today's data. These tables and
> partitions all require 3 replicas. This business logic was implemented by
> some Python scripts. My Kudu cluster contains 3 masters and 3 tservers. Flag:
> --catalog_manager_check_ts_count_for_create_table is false.
> Sometimes, one tserver maybe become unavailable. Table creating task will
> retry continuously and always fail until the tserver become healthy again.
> See the error:
> {color:#ff8b00}E0222 11:10:32.767140 3321 catalog_manager.cc:672] Error
> processing pending assignments: Invalid argument: error selecting replicas
> for tablet 41dffa9783f14f36a5b6c35e89075c1a, state:0: Not enough tablet
> servers are online for table 'test_table'. Need at least 3 replicas, but only
> 2 tablet servers are available{color}
> {color:#172b4d}As there are no enough replicas, a tablet will never be
> created. The state of this tablet is not running. Therefore, read or write
> this tablet will fail even if there are 2 tservers can be used to create 2
> replicas.{color}
>
> An already created tablet can still be on service even if one of its 3
> replicas become unavailable. Why can not create a three-replicas table when
> only 2 tservers healthy?
>
> h1. Design
> A new flag: --support_create_tablet_without_enough_healthy_tservers is added.
> The original logic keeps the same. When this flag is set true, a
> three-replicas tablet can be created successfully and its status is losing
> one replica. This tablet can be be read and write normally.
>
> There are 3 things need to do:
> # A tool to cancel the table creating task.
> # A tool to show the running table creating task.
> # A method to create table without enough healthy tservers
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
