sankarh commented on a change in pull request #569: HIVE-21446 : Hive Server
going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266298085
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##########
@@ -72,33 +72,40 @@ public CopyUtils(String distCpDoAsUser, HiveConf hiveConf)
{
public void copyAndVerify(FileSystem destinationFs, Path destRoot,
List<ReplChangeManager.FileInfo> srcFiles) throws
IOException, LoginException, HiveFatalException {
Map<FileSystem, Map< Path, List<ReplChangeManager.FileInfo>>> map =
fsToFileMap(srcFiles, destRoot);
- for (Map.Entry<FileSystem, Map<Path, List<ReplChangeManager.FileInfo>>>
entry : map.entrySet()) {
- FileSystem sourceFs = entry.getKey();
- Map<Path, List<ReplChangeManager.FileInfo>> destMap = entry.getValue();
- for (Map.Entry<Path, List<ReplChangeManager.FileInfo>> destMapEntry :
destMap.entrySet()) {
- Path destination = destMapEntry.getKey();
- List<ReplChangeManager.FileInfo> fileInfoList =
destMapEntry.getValue();
- boolean useRegularCopy = regularCopy(destinationFs, sourceFs,
fileInfoList);
-
- if (!destinationFs.exists(destination)
- && !FileUtils.mkdir(destinationFs, destination, hiveConf)) {
- LOG.error("Failed to create destination directory: " + destination);
- throw new IOException("Destination directory creation failed");
- }
+ UserGroupInformation proxyUser = getProxyUser();
+ try {
+ for (Map.Entry<FileSystem, Map<Path, List<ReplChangeManager.FileInfo>>>
entry : map.entrySet()) {
+ FileSystem sourceFs = entry.getKey();
Review comment:
The sourceFs should be re-get from HDFS if previous copy closed all file
systems object for proxy user.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
