Yibo Dong created SPARK-60127:
---------------------------------
Summary: ADD IF NOT EXISTS PARTITION changes the location of an
existing partition
Key: SPARK-60127
URL: https://issues.apache.org/jira/browse/SPARK-60127
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Environment: Spark: 5.0.0-SNAPSHOT
Commit: 38422aefaf599ffb4cfa579d98898d95b1f02081
Java: OpenJDK 17.0.17
OS: macOS 15.7.4
Reporter: Yibo Dong
h2. What happened
{{ALTER TABLE ADD IF NOT EXISTS PARTITION}} changes the location of an existing
partition when a different {{LOCATION}} is specified. As a result, subsequent
queries return data from the new location instead of the original partition
data.
h2. How to reproduce
Run the following SQL using {{{}spark-sql{}}}:
{code:sql}
DROP TABLE IF EXISTS t_add_partition_if_not_exists;
DROP TABLE IF EXISTS t_add_partition_if_not_exists_alternate;
CREATE TABLE t_add_partition_if_not_exists (
id INT,
value STRING
)
USING PARQUET
PARTITIONED BY (p STRING)
LOCATION '/tmp/spark_t_add_partition_if_not_exists_old';
INSERT INTO t_add_partition_if_not_exists VALUES (1, 'old', 'a');
CREATE TABLE t_add_partition_if_not_exists_alternate (
id INT,
value STRING
)
USING PARQUET
PARTITIONED BY (p STRING)
LOCATION '/tmp/spark_t_add_partition_if_not_exists_alternate';
INSERT INTO t_add_partition_if_not_exists_alternate VALUES (2, 'alternate',
'a');
DROP TABLE t_add_partition_if_not_exists_alternate;
SELECT id, value, p
FROM t_add_partition_if_not_exists
ORDER BY id;
ALTER TABLE t_add_partition_if_not_exists
ADD IF NOT EXISTS PARTITION (p = 'a')
LOCATION '/tmp/spark_t_add_partition_if_not_exists_alternate/p=a';
SHOW PARTITIONS t_add_partition_if_not_exists;
SELECT id, value, p
FROM t_add_partition_if_not_exists
ORDER BY id;
{code}
Save the SQL as {{{}repro.sql{}}}. Start with clean directories and run:
{code:bash}
rm -rf /tmp/spark_t_add_partition_if_not_exists_old
/tmp/spark_t_add_partition_if_not_exists_alternate
spark-sql --master 'local[2]'
--conf spark.ui.enabled=false
-f repro.sql
{code}
h2. Expected result
Since partition {{p=a}} already exists, {{ADD IF NOT EXISTS PARTITION}} should
leave its location unchanged.
Both queries should return:
{code}
id | value | p
---+-------+---
1 | old | a
{code}
h2. Actual result
Before the {{ALTER TABLE}} statement, the query returns:
{code}
id | value | p
---+-------+---
1 | old | a
{code}
After {{{}ADD IF NOT EXISTS PARTITION{}}}, the query returns:
{code}
id | value | p
---+-----------+---
2 | alternate | a
{code}
{{SHOW PARTITIONS}} still lists only {{{}p=a{}}}, but the existing partition
now reads data from the alternate location.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
