[
https://issues.apache.org/jira/browse/HIVE-24625?focusedWorklogId=547785&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-547785
]
ASF GitHub Bot logged work on HIVE-24625:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 04/Feb/21 19:05
Start Date: 04/Feb/21 19:05
Worklog Time Spent: 10m
Work Description: nrg4878 commented on a change in pull request #1856:
URL: https://github.com/apache/hive/pull/1856#discussion_r555819727
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
##########
@@ -13272,6 +13274,7 @@ private void updateDefaultTblProps(Map<String, String>
source, Map<String, Strin
retValue = convertToAcidByDefault(storageFormat, qualifiedTableName,
sortCols, retValue);
}
}
+ retValue.put(TABLE_IS_CTAS, Boolean.toString(isCTAS));
Review comment:
I think it is better to set this only when it is true, so we dont mud
the rest of the tables with non-essential properties.
##########
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDefaultTransformer.java
##########
@@ -583,29 +584,34 @@ public Table transformCreateTable(Table table,
List<String> processorCapabilitie
throw new MetaException("Database " + dbName + " for table " +
table.getTableName() + " could not be found");
}
- if (TableType.MANAGED_TABLE.name().equals(tableType)) {
+ if (TableType.MANAGED_TABLE.name().equals(tableType)) {
LOG.debug("Table is a MANAGED_TABLE");
txnal = params.get(TABLE_IS_TRANSACTIONAL);
txn_properties = params.get(TABLE_TRANSACTIONAL_PROPERTIES);
+ boolean ctas = Boolean.valueOf(params.getOrDefault(TABLE_IS_CTAS,
"false"));
isInsertAcid = (txn_properties != null &&
txn_properties.equalsIgnoreCase("insert_only"));
if ((txnal == null || txnal.equalsIgnoreCase("FALSE")) && !isInsertAcid)
{ // non-ACID MANAGED TABLE
- LOG.info("Converting " + newTable.getTableName() + " to EXTERNAL
tableType for " + processorId);
- newTable.setTableType(TableType.EXTERNAL_TABLE.toString());
- params.remove(TABLE_IS_TRANSACTIONAL);
- params.remove(TABLE_TRANSACTIONAL_PROPERTIES);
- params.put("EXTERNAL", "TRUE");
- params.put(EXTERNAL_TABLE_PURGE, "TRUE");
- params.put("TRANSLATED_TO_EXTERNAL", "TRUE");
- newTable.setParameters(params);
- LOG.info("Modified table params are:" + params.toString());
-
- if (!table.isSetSd() || table.getSd().getLocation() == null) {
- try {
- Path newPath = hmsHandler.getWh().getDefaultTablePath(db,
table.getTableName(), true);
- newTable.getSd().setLocation(newPath.toString());
- LOG.info("Modified location from null to " + newPath);
- } catch (Exception e) {
- LOG.warn("Exception determining external table location:" +
e.getMessage());
+ if (ctas) {
Review comment:
Shouldn't we be removing the property "TABLE_IS_CTAS" here from the
table params if set. This way we have a net zero properties in the metadata?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 547785)
Time Spent: 20m (was: 10m)
> CTAS with TBLPROPERTIES ('transactional'='false') loads data into incorrect
> directory
> -------------------------------------------------------------------------------------
>
> Key: HIVE-24625
> URL: https://issues.apache.org/jira/browse/HIVE-24625
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Metastore
> Reporter: Attila Magyar
> Assignee: Attila Magyar
> Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> MetastoreDefaultTransformer in HMS converts a managed non transactional table
> to external table. MoveTask still uses the managed path when loading the
> data, resulting an always empty table.
> {code:java}
> create table tbl1 TBLPROPERTIES ('transactional'='false') as select * from
> other;{code}
> After the conversion the table location points to an external directory:
> Location: |
> hdfs://c670-node2.coelab.cloudera.com:8020/warehouse/tablespace/external/hive/tbl1
> Move task uses the managed location"
> {code:java}
> INFO : Moving data to directory
> hdfs://...:8020/warehouse/tablespace/managed/hive/tbl1 from
> hdfs://...:8020/warehouse/tablespace/managed/hive/.hive-staging_hive_2021-01-05_16-10-39_973_41005081081760609-4/-ext-1000
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
