[
https://issues.apache.org/jira/browse/HIVE-24900?focusedWorklogId=568092&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-568092
]
ASF GitHub Bot logged work on HIVE-24900:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 18/Mar/21 04:54
Start Date: 18/Mar/21 04:54
Worklog Time Spent: 10m
Work Description: t3rmin4t0r commented on a change in pull request #2086:
URL: https://github.com/apache/hive/pull/2086#discussion_r596550656
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java
##########
@@ -523,6 +523,8 @@ protected Boolean findNextCompactionAndExecute(boolean
computeStats) throws Inte
final StatsUpdater su = computeStats ? StatsUpdater.init(ci,
msc.findColumnsWithStats(
CompactionInfo.compactionInfoToStruct(ci)), conf,
runJobAsSelf(ci.runAs) ? ci.runAs : t.getOwner()) : null;
+ Path resultDir = QueryCompactor.Util.getCompactionResultDir(sd,
tblValidWriteIds,
Review comment:
make it final.
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java
##########
@@ -523,6 +523,8 @@ protected Boolean findNextCompactionAndExecute(boolean
computeStats) throws Inte
final StatsUpdater su = computeStats ? StatsUpdater.init(ci,
msc.findColumnsWithStats(
CompactionInfo.compactionInfoToStruct(ci)), conf,
runJobAsSelf(ci.runAs) ? ci.runAs : t.getOwner()) : null;
+ Path resultDir = QueryCompactor.Util.getCompactionResultDir(sd,
tblValidWriteIds,
Review comment:
Add a comment - that this is the path which the compactor is supposed to
write to, including an example path + the fact that it has a _v<txn> in it
##########
File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
##########
@@ -2133,6 +2133,7 @@ private static void populateLlapDaemonVarsSet(Set<String>
llapDaemonVarsSetLocal
HIVETESTCURRENTTIMESTAMP("hive.test.currenttimestamp", null, "current
timestamp for test", false),
HIVETESTMODEROLLBACKTXN("hive.test.rollbacktxn", false, "For testing only.
Will mark every ACID transaction aborted", false),
HIVETESTMODEFAILCOMPACTION("hive.test.fail.compaction", false, "For
testing only. Will cause CompactorMR to fail.", false),
+ HIVETESTMODEFAILAFTERCOMPACTION("hive.test.fail.after.compaction", false,
"For testing only. Will cause CompactorMR to fail all compacting.", false),
Review comment:
Mention this fails after open txn + creating output files
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java
##########
@@ -555,6 +559,11 @@ protected Boolean findNextCompactionAndExecute(boolean
computeStats) throws Inte
LOG.error("Caught exception while trying to compact " + ci +
". Marking failed to avoid repeated failures", e);
markFailed(ci, e);
+ // remove the new directories created by compaction
+ if (resultDir != null) {
+ FileSystem fs = resultDir.getFileSystem(conf);
+ fs.delete(resultDir, true);
Review comment:
see if there needs to be exists check
##########
File path: ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java
##########
@@ -463,6 +463,44 @@ public void testCompactionAbort() throws Exception {
runCleaner(hiveConf);
}
+
+ @Test
+ public void testCompactionAbortLeftoverFiles() throws Exception {
+ MetastoreConf.setBoolVar(hiveConf,
MetastoreConf.ConfVars.CREATE_TABLES_AS_ACID, true);
+ dropTable(new String[] {"T"});
+ //note: transaction names T1, T2, etc below, are logical, the actual txnid
will be different
+ runStatementOnDriver("create table T (a int, b int) stored as orc");
+ runStatementOnDriver("insert into T values(0,2)");//makes delta_1_1 in T1
+ runStatementOnDriver("insert into T values(1,4)");//makes delta_2_2 in T2
+
+ //create failed compaction attempt so that compactor txn is aborted
+ HiveConf.setBoolVar(hiveConf,
HiveConf.ConfVars.HIVETESTMODEFAILAFTERCOMPACTION, true);
+ runStatementOnDriver("alter table T compact 'major'");
+ runWorker(hiveConf);
+
+ TxnStore txnHandler = TxnUtils.getTxnStore(hiveConf);
+ ShowCompactResponse resp = txnHandler.showCompact(new
ShowCompactRequest());
+ Assert.assertEquals("Unexpected number of compactions in history",
+ 1, resp.getCompactsSize());
+ Assert.assertEquals("Unexpected 0th compaction state",
+ TxnStore.FAILED_RESPONSE, resp.getCompacts().get(0).getState());
+ GetOpenTxnsResponse openResp = txnHandler.getOpenTxns();
+ Assert.assertEquals(openResp.toString(), 1, openResp.getOpen_txnsSize());
+ //check that the compactor txn is aborted
+ Assert.assertTrue(openResp.toString(),
BitSet.valueOf(openResp.getAbortedBits()).get(0));
+
+ FileSystem fs = FileSystem.get(hiveConf);
+ Path warehousePath = new Path(getWarehouseDir());
+ FileStatus[] actualList = fs.listStatus(new Path(warehousePath + "/t"),
+ FileUtils.HIDDEN_FILES_PATH_FILTER);
+
+ String[] expectedList = new String[] {
+ "/t/delta_0000001_0000001_0000",
+ "/t/delta_0000002_0000002_0000",
+ };
Review comment:
Add comment saying that we should not see a path which has
"/t/base_00002_v*", so that nobody adds that here to "make a test pass"
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java
##########
@@ -555,6 +559,11 @@ protected Boolean findNextCompactionAndExecute(boolean
computeStats) throws Inte
LOG.error("Caught exception while trying to compact " + ci +
". Marking failed to avoid repeated failures", e);
markFailed(ci, e);
+ // remove the new directories created by compaction
+ if (resultDir != null) {
+ FileSystem fs = resultDir.getFileSystem(conf);
+ fs.delete(resultDir, true);
Review comment:
Log.error above needs modification to say "going to delete output-path"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 568092)
Time Spent: 0.5h (was: 20m)
> Failed compaction does not cleanup the directories
> --------------------------------------------------
>
> Key: HIVE-24900
> URL: https://issues.apache.org/jira/browse/HIVE-24900
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Reporter: Ramesh Kumar Thangarajan
> Assignee: Ramesh Kumar Thangarajan
> Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Failed compaction does not cleanup the directories
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
