Wenchen Fan created SPARK-20236:
-----------------------------------
Summary: Overwrite a partitioned table should only overwrite
related partitions
Key: SPARK-20236
URL: https://issues.apache.org/jira/browse/SPARK-20236
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 2.2.0
Reporter: Wenchen Fan
When we overwrite a partitioned table, currently Spark will truncate the entire
table to write new data, or truncate a bunch of partitions according to the
given static partitions.
For example, {{INSERT OVERWRITE tbl ...}} will truncate the entire table,
{{INSERT OVERWRITE tbl PARTITION (a=1, b)}} will truncate all the partitions
that starts with {{a=1}}.
This behavior is kind of reasonable as we can know which partitions will be
overwritten before runtime. However, hive has a different behavior that it only
overwrites related partitions, e.g. {{INSERT OVERWRITE tbl SELECT 1,2,3}} will
only overwrite partition {{a=2, b=3}}, assuming {{tbl}} has only one data
column and is partitioned by {{a}} and {{b}}.
It seems better if we can follow hive's behavior.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
