[
https://issues.apache.org/jira/browse/HIVE-24675?focusedWorklogId=541693&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-541693
]
ASF GitHub Bot logged work on HIVE-24675:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 26/Jan/21 04:10
Start Date: 26/Jan/21 04:10
Worklog Time Spent: 10m
Work Description: ArkoSharma commented on a change in pull request #1898:
URL: https://github.com/apache/hive/pull/1898#discussion_r563493498
##########
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcrossInstances.java
##########
@@ -2122,4 +2221,21 @@ private void setupUDFJarOnHDFS(Path
identityUdfLocalPath, Path identityUdfHdfsPa
+ NS_REMOTE + "'");
return withClause;
}
+
+ /*
+ * Method used from TestReplicationScenariosExclusiveReplica
+ */
+ private void assertExternalFileInfo(List<String> expected, String
dumplocation, boolean isIncremental,
+ WarehouseInstance warehouseInstance)
+ throws IOException {
+ Path hivePath = new Path(dumplocation, ReplUtils.REPL_HIVE_BASE_DIR);
+ Path metadataPath = new Path(hivePath, EximUtil.METADATA_PATH_NAME);
+ Path externalTableInfoFile;
+ if (isIncremental) {
+ externalTableInfoFile = new Path(hivePath, FILE_NAME);
Review comment:
This is the deprecated one used in existing external table tests.
Changing these tests with the new file is done as part a separate issue which
cleans up the deprecated file.
##########
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplExternalTables.java
##########
@@ -183,6 +184,22 @@ private void dirLocationToCopy(FileList fileList, Path
sourcePath, HiveConf conf
throws HiveException {
Path basePath = getExternalTableBaseDir(conf);
Path targetPath = externalTableDataPath(conf, basePath, sourcePath);
+ //Here, when src and target are HA clusters with same NS, then
sourcePath would have the correct host
+ //whereas the targetPath would have an host that refers to the target
cluster. This is fine for
+ //data-copy running during dump as the correct logical locations would
be used. But if data-copy runs during
+ //load, then the remote location needs to point to the src cluster from
where the data would be copied and
+ //the common original NS would suffice for targetPath.
+
if(hiveConf.getBoolVar(HiveConf.ConfVars.REPL_HA_DATAPATH_REPLACE_REMOTE_NAMESERVICE)
&&
+
hiveConf.getBoolVar(HiveConf.ConfVars.REPL_RUN_DATA_COPY_TASKS_ON_TARGET)) {
+ String remoteNS =
hiveConf.get(HiveConf.ConfVars.REPL_HA_DATAPATH_REPLACE_REMOTE_NAMESERVICE_NAME.varname);
+ if (StringUtils.isEmpty(remoteNS)) {
+ throw new SemanticException(ErrorMsg.REPL_INVALID_CONFIG_FOR_SERVICE
Review comment:
Yes, it is a non recoverable error if the remote NS config is not
specified.There needs to be a valid remote NS name and the same name should be
used in both the dump and load commands.
The same exception is being thrown for managed tables.
##########
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcrossInstances.java
##########
@@ -1707,6 +1738,50 @@ public void testHdfsNameserviceLazyCopyIncr() throws
Throwable {
}
}
+ @Test
+ public void testHdfsNSLazyCopyIncrExtTbls() throws Throwable {
Review comment:
Bootstrap load is expected to fail in the earlier test, so incremental
load failure cannot be tested in the same test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 541693)
Time Spent: 1h 20m (was: 1h 10m)
> Handle external table replication for HA with same NS and lazy copy.
> --------------------------------------------------------------------
>
> Key: HIVE-24675
> URL: https://issues.apache.org/jira/browse/HIVE-24675
> Project: Hive
> Issue Type: Bug
> Reporter: Arko Sharma
> Assignee: Arko Sharma
> Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-24675.01.patch, HIVE-24675.02.patch,
> HIVE-24675.03.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
