[
https://issues.apache.org/jira/browse/HIVE-28822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-28822:
--------------------------------
Description:
Concurrent `INSERT INTO` statements that create the same new dynamic partition
on an external non-ACID Hive table backed by S3 can fail with `MoveTask return
code 40000`. The root cause is a check-then-rename race in `Hive.mvFile`'s
`_copy_N` naming loop: two writers pick the same destination path between an
`exists()` probe and the subsequent `rename()` call, and the loser gets a
`FileAlreadyExistsException` from `S3AFileSystem.initiateRename`.
This is separate from HIVE-29744, which addresses the *silent* data loss twin
of this race (concurrent writers overwriting each other with no error). This
ticket is specifically about the *fail-loud* variant that survives even after
HIVE-29744, but also it fixes another source of silent data loss scenario,
which is because of the unstable incrementing copy_N loop in case of some
object stores. See details in PR.
Local repro against table on S3 gave:
{code}
2026-07-27T07:11:51,807 ERROR [load-dynamic-partitionsToAdd-0] metadata.Hive:
Exception when loading partition with parameters
partPath=s3a://dw-team-bucket/tmp/p_test/.hive-staging_hive_2026-07-27_07-11-26_4
03_8045012243677926207-34/-ext-10000/b=2, table=p_test, partSpec={b=2},
loadFileType=KEEP_EXISTING, listBucketingLevel=0, isAcid=false,
resetStatistics=false
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source
s3a://dw-team-bucket/tmp/p_test/.hive-staging_hive_2026-07-27_07-11-26_403_804501224367
7926207-34/-ext-10000/b=2/000001_0 to destination
s3a://dw-team-bucket/tmp/p_test/b=2
at
org.apache.hadoop.hive.ql.metadata.Hive.handlePoolException(Hive.java:5458)
at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:5121)
at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:5618)
at
org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:2912)
at
org.apache.hadoop.hive.ql.metadata.Hive.lambda$loadDynamicPartitions$7(Hive.java:3394)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move
source
s3a://dw-team-bucket/tmp/p_test/.hive-staging_hive_2026-07-27_07-11-26_403_8045012243677926207-34/-ext-10000/b=2/000001_0
to d
estination s3a://dw-team-bucket/tmp/p_test/b=2
at
org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:5486)
at
org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:5441)
at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:5107)
at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:5093)
... 4 more
Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: Failed to rename
s3a://dw-team-bucket/tmp/p_test/.hive-staging_hive_2026-07-27_07-11-26_403_8045012243677926207-34/-ext-10000/b=2/000001_0
to s3a://dw-team-bucket/tmp/p_test/b=2/000001_0_copy_5; destination file exists
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initiateRename(S3AFileSystem.java:2508)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:2573)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$rename$6(S3AFileSystem.java:2434)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2917)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:2432)
at org.apache.hadoop.hive.ql.metadata.Hive.mvFile(Hive.java:5235)
at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:5100)
... 5 more
{code}
was:
Concurrent `INSERT INTO` statements that create the same new dynamic partition
on an external non-ACID Hive table backed by S3 can fail with `MoveTask return
code 40000`. The root cause is a check-then-rename race in `Hive.mvFile`'s
`_copy_N` naming loop: two writers pick the same destination path between an
`exists()` probe and the subsequent `rename()` call, and the loser gets a
`FileAlreadyExistsException` from `S3AFileSystem.initiateRename`.
This is separate from HIVE-29744, which addresses the *silent* data-loss twin
of this race (concurrent writers overwriting each other with no error). This
ticket is specifically about the *fail-loud* variant that survives even after
HIVE-29744.
Local repro against table on S3 gave:
{code}
2026-07-27T07:11:51,807 ERROR [load-dynamic-partitionsToAdd-0] metadata.Hive:
Exception when loading partition with parameters
partPath=s3a://dw-team-bucket/tmp/p_test/.hive-staging_hive_2026-07-27_07-11-26_4
03_8045012243677926207-34/-ext-10000/b=2, table=p_test, partSpec={b=2},
loadFileType=KEEP_EXISTING, listBucketingLevel=0, isAcid=false,
resetStatistics=false
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source
s3a://dw-team-bucket/tmp/p_test/.hive-staging_hive_2026-07-27_07-11-26_403_804501224367
7926207-34/-ext-10000/b=2/000001_0 to destination
s3a://dw-team-bucket/tmp/p_test/b=2
at
org.apache.hadoop.hive.ql.metadata.Hive.handlePoolException(Hive.java:5458)
at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:5121)
at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:5618)
at
org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:2912)
at
org.apache.hadoop.hive.ql.metadata.Hive.lambda$loadDynamicPartitions$7(Hive.java:3394)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move
source
s3a://dw-team-bucket/tmp/p_test/.hive-staging_hive_2026-07-27_07-11-26_403_8045012243677926207-34/-ext-10000/b=2/000001_0
to d
estination s3a://dw-team-bucket/tmp/p_test/b=2
at
org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:5486)
at
org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:5441)
at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:5107)
at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:5093)
... 4 more
Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: Failed to rename
s3a://dw-team-bucket/tmp/p_test/.hive-staging_hive_2026-07-27_07-11-26_403_8045012243677926207-34/-ext-10000/b=2/000001_0
to s3a://dw-team-bucket/tmp/p_test/b=2/000001_0_copy_5; destination file exists
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initiateRename(S3AFileSystem.java:2508)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:2573)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$rename$6(S3AFileSystem.java:2434)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2917)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:2432)
at org.apache.hadoop.hive.ql.metadata.Hive.mvFile(Hive.java:5235)
at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:5100)
... 5 more
{code}
> Concurrent INSERTs into a new dynamic partition can silently lose rows or
> fail with FileAlreadyExistsException on S3 (non-ACID)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-28822
> URL: https://issues.apache.org/jira/browse/HIVE-28822
> Project: Hive
> Issue Type: Bug
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Labels: pull-request-available
>
> Concurrent `INSERT INTO` statements that create the same new dynamic
> partition on an external non-ACID Hive table backed by S3 can fail with
> `MoveTask return code 40000`. The root cause is a check-then-rename race in
> `Hive.mvFile`'s `_copy_N` naming loop: two writers pick the same destination
> path between an `exists()` probe and the subsequent `rename()` call, and the
> loser gets a `FileAlreadyExistsException` from `S3AFileSystem.initiateRename`.
> This is separate from HIVE-29744, which addresses the *silent* data loss twin
> of this race (concurrent writers overwriting each other with no error). This
> ticket is specifically about the *fail-loud* variant that survives even after
> HIVE-29744, but also it fixes another source of silent data loss scenario,
> which is because of the unstable incrementing copy_N loop in case of some
> object stores. See details in PR.
> Local repro against table on S3 gave:
> {code}
> 2026-07-27T07:11:51,807 ERROR [load-dynamic-partitionsToAdd-0] metadata.Hive:
> Exception when loading partition with parameters
> partPath=s3a://dw-team-bucket/tmp/p_test/.hive-staging_hive_2026-07-27_07-11-26_4
> 03_8045012243677926207-34/-ext-10000/b=2, table=p_test, partSpec={b=2},
> loadFileType=KEEP_EXISTING, listBucketingLevel=0, isAcid=false,
> resetStatistics=false
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source
> s3a://dw-team-bucket/tmp/p_test/.hive-staging_hive_2026-07-27_07-11-26_403_804501224367
> 7926207-34/-ext-10000/b=2/000001_0 to destination
> s3a://dw-team-bucket/tmp/p_test/b=2
> at
> org.apache.hadoop.hive.ql.metadata.Hive.handlePoolException(Hive.java:5458)
> at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:5121)
> at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:5618)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:2912)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.lambda$loadDynamicPartitions$7(Hive.java:3394)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move
> source
> s3a://dw-team-bucket/tmp/p_test/.hive-staging_hive_2026-07-27_07-11-26_403_8045012243677926207-34/-ext-10000/b=2/000001_0
> to d
> estination s3a://dw-team-bucket/tmp/p_test/b=2
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:5486)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:5441)
> at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:5107)
> at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:5093)
> ... 4 more
> Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: Failed to rename
> s3a://dw-team-bucket/tmp/p_test/.hive-staging_hive_2026-07-27_07-11-26_403_8045012243677926207-34/-ext-10000/b=2/000001_0
> to s3a://dw-team-bucket/tmp/p_test/b=2/000001_0_copy_5; destination file
> exists
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.initiateRename(S3AFileSystem.java:2508)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:2573)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$rename$6(S3AFileSystem.java:2434)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2917)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:2432)
> at org.apache.hadoop.hive.ql.metadata.Hive.mvFile(Hive.java:5235)
> at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:5100)
> ... 5 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
