[
https://issues.apache.org/jira/browse/HIVE-21886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mahesh kumar behera updated HIVE-21886:
---------------------------------------
Description:
If some rename events are found to be dumped and replayed while replace policy
is getting executed, it needs to take care of the policy inclusion in both the
policy for each table name.
Rename handling during replace policy
# Old name not matching old policy – The old table will not be there at the
target cluster. The table will not be returned by get-all-table.
# Old name is not matching new policy
# New name not matching old policy
# New name not matching new policy
# Ignore the event, no need to do anything.
# New name matching new policy
# The table will be returned by get-all-table. Replace policy handler will
bootstrap this table as its matching new policy and not matching old policy.
# All the future events will be ignored as part of check added by replace
policy handling.
# All the event with old table name will anyways be ignored as the old name is
not matching the new policy.
# New name matching old policy
# New name not matching new policy
# As the new name is not matching the new policy, the table need not be
replicated.
# As the old name is not matching the new policy, the rename events will be
ignored.
# So nothing to be done for this scenario.
# New name matching new policy
# As the new name is matching both old and new policy, replace handler will
not bootstrap the table.
# Add the table to the list of tables to be bootstrapped.
# Ignore all the events with new name.
# If there is a drop event for the table (with new name), then remove the
table from the the list of table to be bootstrapped.
# In case of rename event (double rename)
# If the new name satisfies the table pattern, then add the new name to the
list of tables to be bootstrapped and remove the old name from the list of
tables to be bootstrapped.
# If the new name does not satisfies then just removed the table name from the
list of tables to be bootstrapped.
# Old name is matching new policy – As per replace policy handler, which
checks based on old table, the table should be bootstrapped and event should be
ignored. But rename handler should decide based on new name.The old table name
will not be returned by get-all-table, so replace handler will not d anything
for the old table.
# New name not matching old policy
# New name not matching new policy
# As the old table is not there at target and new name is not matching new
policy. Ignore the event.
# No need to add the table to the list of tables to be bootstrapped.
# All the subsequent events will be ignored as the new name is not matching
the new policy.
# New name matching new policy
# As the new name is not matching old policy but matching new policy, the
table will be bootstrapped by replace policy handler. So rename event need not
add this table to list of table to be bootstrapped.
# All the future events will be ignored by replace policy handler.
# For rename event (double rename)
# If there is a rename, the table (with intermittent new name) will not be
present and thus replace handler will not bootstrap the table.
# So if the new name (the latest one) is matching the new policy, then add it
to the list of table to be bootstrapped.
# And If the new name (the latest one) is not matching the new policy, then
just ignore the event as the intermittent new name would not have added to the
list of table to be bootstrapped.
# New name matching old policy
# New name not matching new policy
# Dump the event. The table will be dropped by repl load at the target.
# New name matching new policy
# Replace handler will not bootstrap this table as the new name is matching
both policies.
# As old name is not matching the old policy, the table will not be there at
target. The rename event should add the new name to the list of table to be
bootstrapped.
# Subsequent events with new table name should be ignored.
# Drop events should not be ignored as if the table is present during
bootstrapped, then its a new table and thus should be doropped.
# In case of rename event (double rename)
# If the new name satisfies the table pattern, then add the new name to the
list of tables to be bootstrapped and remove the old name from the list of
tables to be bootstrapped.
# If the new name does not satisfies then just removed the table name from the
list of tables to be bootstrapped.
# Old name is matching old policy – The old table will be there at the target
cluster. The table will not be returned by get-all-table. Repl load should
delete the old table as it is not matching the new policy.
# Old name is not matching new policy
# New name not matching old policy
# New name not matching new policy
# Nothing to be done. Ignore the event.
# New name matching new policy
# As the new name is not matching old policy but matching new policy, the
table will be bootstrapped by replace policy handler. So rename event need not
add this table to list of table to be bootstrapped.
# All the future events will be ignored by replace policy handler.
# For rename event (double rename)
# If there is a rename, the table (with intermittent new name) will not be
present and thus replace handler will not bootstrap the table.
# So if the new name (the latest one) is matching the new policy, then add it
to the list of table to be bootstrapped.
# And If the new name (the latest one) is not matching the new policy, then
just ignore the event as the intermittent new name would not have added to the
list of table to be bootstrapped.
# New name matching old policy
# New name not matching new policy
# Table with new name will be dropped by repl load
# Along with other event, ignore the rename event also.
# New name matching new policy
# As the new name is matching both old and new policy, replace handler will
not bootstrap the table.
# Add the table to the list of tables to be bootstrapped.
# Ignore all the events with new name.
# If there is a drop event for the table (with new name), then remove the
table from the the list of table to be bootstrapped.
# In case of rename event (double rename)
# If the new name satisfies the table pattern, then add the new name to the
list of tables to be bootstrapped and remove the old name from the list of
tables to be bootstrapped.
# If the new name does not satisfies then just removed the table name from the
list of tables to be bootstrapped.
# Old name is matching new policy
# New name not matching old policy
# New name not matching new policy
# The old table needs to be dropped at target. Ignore this event, as the old
table is not matching the new policy, it will be dropped by repl load.
# New name matching new policy
# Allow the event to dump and replayed at target.
# Allow further events to be handled as usual.
# In case of rename event (double rename)
# If the latest new name is matching the new policy, then keep it as is it.
Let rename event replayed at target.
# If the latest new name is not matching the new policy, then change the
rename event to drop event.
# New name matching old policy
# New name not matching new policy
# New name matching new policy
> REPL - With table list - Handle rename events during replace policy
> -------------------------------------------------------------------
>
> Key: HIVE-21886
> URL: https://issues.apache.org/jira/browse/HIVE-21886
> Project: Hive
> Issue Type: Sub-task
> Components: repl
> Reporter: mahesh kumar behera
> Assignee: mahesh kumar behera
> Priority: Major
> Labels: DR, Replication
>
> If some rename events are found to be dumped and replayed while replace
> policy is getting executed, it needs to take care of the policy inclusion in
> both the policy for each table name.
>
> Rename handling during replace policy
> # Old name not matching old policy – The old table will not be there at the
> target cluster. The table will not be returned by get-all-table.
> # Old name is not matching new policy
> # New name not matching old policy
> # New name not matching new policy
> # Ignore the event, no need to do anything.
> # New name matching new policy
> # The table will be returned by get-all-table. Replace policy handler will
> bootstrap this table as its matching new policy and not matching old policy.
> # All the future events will be ignored as part of check added by replace
> policy handling.
> # All the event with old table name will anyways be ignored as the old name
> is not matching the new policy.
> # New name matching old policy
> # New name not matching new policy
> # As the new name is not matching the new policy, the table need not be
> replicated.
> # As the old name is not matching the new policy, the rename events will be
> ignored.
> # So nothing to be done for this scenario.
> # New name matching new policy
> # As the new name is matching both old and new policy, replace handler will
> not bootstrap the table.
> # Add the table to the list of tables to be bootstrapped.
> # Ignore all the events with new name.
> # If there is a drop event for the table (with new name), then remove the
> table from the the list of table to be bootstrapped.
> # In case of rename event (double rename)
> # If the new name satisfies the table pattern, then add the new name to the
> list of tables to be bootstrapped and remove the old name from the list of
> tables to be bootstrapped.
> # If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # Old name is matching new policy – As per replace policy handler, which
> checks based on old table, the table should be bootstrapped and event should
> be ignored. But rename handler should decide based on new name.The old table
> name will not be returned by get-all-table, so replace handler will not d
> anything for the old table.
> # New name not matching old policy
> # New name not matching new policy
> # As the old table is not there at target and new name is not matching new
> policy. Ignore the event.
> # No need to add the table to the list of tables to be bootstrapped.
> # All the subsequent events will be ignored as the new name is not matching
> the new policy.
> # New name matching new policy
> # As the new name is not matching old policy but matching new policy, the
> table will be bootstrapped by replace policy handler. So rename event need
> not add this table to list of table to be bootstrapped.
> # All the future events will be ignored by replace policy handler.
> # For rename event (double rename)
> # If there is a rename, the table (with intermittent new name) will not be
> present and thus replace handler will not bootstrap the table.
> # So if the new name (the latest one) is matching the new policy, then add
> it to the list of table to be bootstrapped.
> # And If the new name (the latest one) is not matching the new policy, then
> just ignore the event as the intermittent new name would not have added to
> the list of table to be bootstrapped.
> # New name matching old policy
> # New name not matching new policy
> # Dump the event. The table will be dropped by repl load at the target.
> # New name matching new policy
> # Replace handler will not bootstrap this table as the new name is matching
> both policies.
> # As old name is not matching the old policy, the table will not be there at
> target. The rename event should add the new name to the list of table to be
> bootstrapped.
> # Subsequent events with new table name should be ignored.
> # Drop events should not be ignored as if the table is present during
> bootstrapped, then its a new table and thus should be doropped.
> # In case of rename event (double rename)
> # If the new name satisfies the table pattern, then add the new name to the
> list of tables to be bootstrapped and remove the old name from the list of
> tables to be bootstrapped.
> # If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # Old name is matching old policy – The old table will be there at the
> target cluster. The table will not be returned by get-all-table. Repl load
> should delete the old table as it is not matching the new policy.
> # Old name is not matching new policy
> # New name not matching old policy
> # New name not matching new policy
> # Nothing to be done. Ignore the event.
> # New name matching new policy
> # As the new name is not matching old policy but matching new policy, the
> table will be bootstrapped by replace policy handler. So rename event need
> not add this table to list of table to be bootstrapped.
> # All the future events will be ignored by replace policy handler.
> # For rename event (double rename)
> # If there is a rename, the table (with intermittent new name) will not be
> present and thus replace handler will not bootstrap the table.
> # So if the new name (the latest one) is matching the new policy, then add
> it to the list of table to be bootstrapped.
> # And If the new name (the latest one) is not matching the new policy, then
> just ignore the event as the intermittent new name would not have added to
> the list of table to be bootstrapped.
> # New name matching old policy
> # New name not matching new policy
> # Table with new name will be dropped by repl load
> # Along with other event, ignore the rename event also.
> # New name matching new policy
> # As the new name is matching both old and new policy, replace handler will
> not bootstrap the table.
> # Add the table to the list of tables to be bootstrapped.
> # Ignore all the events with new name.
> # If there is a drop event for the table (with new name), then remove the
> table from the the list of table to be bootstrapped.
> # In case of rename event (double rename)
> # If the new name satisfies the table pattern, then add the new name to the
> list of tables to be bootstrapped and remove the old name from the list of
> tables to be bootstrapped.
> # If the new name does not satisfies then just removed the table name from
> the list of tables to be bootstrapped.
> # Old name is matching new policy
> # New name not matching old policy
> # New name not matching new policy
> # The old table needs to be dropped at target. Ignore this event, as the old
> table is not matching the new policy, it will be dropped by repl load.
> # New name matching new policy
> # Allow the event to dump and replayed at target.
> # Allow further events to be handled as usual.
> # In case of rename event (double rename)
> # If the latest new name is matching the new policy, then keep it as is it.
> Let rename event replayed at target.
> # If the latest new name is not matching the new policy, then change the
> rename event to drop event.
> # New name matching old policy
> # New name not matching new policy
> # New name matching new policy
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
