[
https://issues.apache.org/jira/browse/HIVE-15367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727401#comment-15727401
]
Sahil Takiar commented on HIVE-15367:
-------------------------------------
Posted a new patch with the follow changes:
* Modified the logic that was introduced in HIVE-11319 - instead of always
throwing an exception if the {{LOCATION}} in the {{CREATE TABLE}} clause is
not-empty, this patch allows files that start with {{.hive-staging}} to exist
inside the target table location; this is necessary because the staging
directory is created inside the target table location before this check runs.
So unless we add logic to ignore {{.hive-staging}} files then this patch won't
work.
* Added CTAS qtests for hive-blobstore
* Added CTAS with LOCATION tests for regular qtests
* Updated the diff for encrypted_ctas.q because the .q file was specifying a
target table location, so the .out file needed to be updated
* Added a NPE fix for the materialized view exceptions that were being thrown
> CTAS with LOCATION should write temp data under location directory rather
> than database location
> ------------------------------------------------------------------------------------------------
>
> Key: HIVE-15367
> URL: https://issues.apache.org/jira/browse/HIVE-15367
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Attachments: HIVE-15367.1.patch, HIVE-15367.2.patch,
> HIVE-15367.3.patch
>
>
> For regular CTAS queries, temp data from a SELECT query will be written to to
> a staging directory under the database location. The code to control this is
> in {{SemanticAnalyzer.java}}
> {code}
> // allocate a temporary output dir on the location of the table
> String tableName = getUnescapedName((ASTNode) ast.getChild(0));
> String[] names = Utilities.getDbTableName(tableName);
> Path location;
> try {
> Warehouse wh = new Warehouse(conf);
> //Use destination table's db location.
> String destTableDb = qb.getTableDesc() != null?
> qb.getTableDesc().getDatabaseName(): null;
> if (destTableDb == null) {
> destTableDb = names[0];
> }
> location = wh.getDatabasePath(db.getDatabase(destTableDb));
> } catch (MetaException e) {
> throw new SemanticException(e);
> }
> {code}
> However, CTAS queries allow specifying a {{LOCATION}} for the new table. Its
> possible for this location to be on a different filesystem than the database
> location. If this happens temp data will be written to the database
> filesystem and will be copied to the table filesystem in {{MoveTask}}.
> This extra copying of data can drastically affect performance. Rather than
> always use the database location as the staging dir for CTAS queries, Hive
> should first check if there is an explicit {{LOCATION}} specified in the CTAS
> query. If there is, staging data should be stored under the {{LOCATION}}
> directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
