[
https://issues.apache.org/jira/browse/HIVE-24912?focusedWorklogId=574622&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-574622
]
ASF GitHub Bot logged work on HIVE-24912:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 31/Mar/21 06:34
Start Date: 31/Mar/21 06:34
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on a change in pull request #2092:
URL: https://github.com/apache/hive/pull/2092#discussion_r604619364
##########
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/incremental/IncrementalLoadTasksBuilder.java
##########
@@ -163,6 +163,17 @@ public IncrementalLoadTasksBuilder(String dbName, String
loadPath,
taskChainTail.addDependentTask(updateIncPendTask);
taskChainTail = updateIncPendTask;
+ Database dbToLoadIn = hive.getDatabase(dbName);
+ if (dbToLoadIn != null && !ReplUtils.isTargetOfReplication(dbToLoadIn)) {
+ Map<String, String> props = new HashMap<>();
+ props.put(ReplUtils.TARGET_OF_REPLICATION, "true");
Review comment:
Can you please check if this is not overriding an existing property?
##########
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/incremental/IncrementalLoadTasksBuilder.java
##########
@@ -163,6 +163,17 @@ public IncrementalLoadTasksBuilder(String dbName, String
loadPath,
taskChainTail.addDependentTask(updateIncPendTask);
taskChainTail = updateIncPendTask;
+ Database dbToLoadIn = hive.getDatabase(dbName);
+ if (dbToLoadIn != null && !ReplUtils.isTargetOfReplication(dbToLoadIn)) {
Review comment:
When would dbToLoadIn be null?
##########
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenarios.java
##########
@@ -4287,6 +4288,37 @@ public void testMoveOptimizationIncremental() throws
IOException {
verifyRun("SELECT count(*) from " + replDbName + ".unptned_late ", "3",
driverMirror);
}
+ @Test
+ public void testReplTargetSetInIncremental() throws Exception {
+ String name = testName.getMethodName();
+ String dbName = createDB(name, driver);
+ String replDbName = dbName + "_dupe";
+
+ run("CREATE TABLE " + dbName + ".unptned(a string) STORED AS TEXTFILE",
driver);
+ Tuple bootstrapDump = bootstrapLoadAndVerify(dbName, replDbName);
+
+ // Remove TARGET_OF_REPLICATION property.
+ run("ALTER DATABASE " + replDbName + " Set DBPROPERTIES ( '" +
TARGET_OF_REPLICATION + "' = '')", driverMirror);
+
Review comment:
Can you verify that the property in target DB indeed doesn't exist.
##########
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenarios.java
##########
@@ -4287,6 +4288,37 @@ public void testMoveOptimizationIncremental() throws
IOException {
verifyRun("SELECT count(*) from " + replDbName + ".unptned_late ", "3",
driverMirror);
}
+ @Test
+ public void testReplTargetSetInIncremental() throws Exception {
+ String name = testName.getMethodName();
+ String dbName = createDB(name, driver);
Review comment:
Set some db level property and make sure after incremental the property
is present
##########
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenarios.java
##########
@@ -4287,6 +4288,37 @@ public void testMoveOptimizationIncremental() throws
IOException {
verifyRun("SELECT count(*) from " + replDbName + ".unptned_late ", "3",
driverMirror);
}
+ @Test
+ public void testReplTargetSetInIncremental() throws Exception {
+ String name = testName.getMethodName();
+ String dbName = createDB(name, driver);
+ String replDbName = dbName + "_dupe";
+
+ run("CREATE TABLE " + dbName + ".unptned(a string) STORED AS TEXTFILE",
driver);
+ Tuple bootstrapDump = bootstrapLoadAndVerify(dbName, replDbName);
+
Review comment:
Add an assertion for the properties('repl.target.for' as well as extra
property you set initially) being present on target db.
##########
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcrossInstances.java
##########
@@ -1114,13 +1115,18 @@ public void
testIfCkptAndSourceOfReplPropsIgnoredByReplDump() throws Throwable {
.dump(primaryDbName, Collections.emptyList());
// Incremental Repl A -> B with alters on db/table/partition
- WarehouseInstance.Tuple tupleReplicaInc = replica.load(replicatedDbName,
primaryDbName)
- .run("repl status " + replicatedDbName)
+ replica.load(replicatedDbName, primaryDbName);
Review comment:
- Perform bootstrap dump and load - should pass
- Perform dump on target db - should fail
- Alter target db to remove repl.target.for and perform dump on target db -
should pass
- perform incremental dump and load (src->target). - should pass
- Perform dump on targetdb - should fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 574622)
Time Spent: 20m (was: 10m)
> Support to add repl.target.for property during incremental run
> --------------------------------------------------------------
>
> Key: HIVE-24912
> URL: https://issues.apache.org/jira/browse/HIVE-24912
> Project: Hive
> Issue Type: Bug
> Reporter: Haymant Mangla
> Assignee: Haymant Mangla
> Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
