[
https://issues.apache.org/jira/browse/HIVE-28822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-28822:
--------------------------------
Description:
Concurrent `INSERT INTO` statements that create the same new dynamic
partition on an external non-ACID Hive table backed by S3 can fail with
`MoveTask return code 40000`. The root cause is a check-then-rename race in
`Hive.mvFile`'s `_copy_N` naming loop: two writers pick the same
destination path between an `exists()` probe and the subsequent
`rename()` call, and the loser gets a `FileAlreadyExistsException` from
`S3AFileSystem.initiateRename`.
This is separate from HIVE-29744, which addresses the *silent* data-loss
twin of this race (concurrent writers overwriting each other with no
error). This ticket is specifically about the *fail-loud* variant that
survives even after HIVE-29744.
Local repro against table on S3 gave:
{code}
2026-07-27T07:11:51,807 ERROR [load-dynamic-partitionsToAdd-0] metadata.Hive:
Exception when loading partition with parameters
partPath=s3a://dw-team-bucket/tmp/p_test/.hive-staging_hive_2026-07-27_07-11-26_4
03_8045012243677926207-34/-ext-10000/b=2, table=p_test, partSpec={b=2},
loadFileType=KEEP_EXISTING, listBucketingLevel=0, isAcid=false,
resetStatistics=false
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source
s3a://dw-team-bucket/tmp/p_test/.hive-staging_hive_2026-07-27_07-11-26_403_804501224367
7926207-34/-ext-10000/b=2/000001_0 to destination
s3a://dw-team-bucket/tmp/p_test/b=2
at
org.apache.hadoop.hive.ql.metadata.Hive.handlePoolException(Hive.java:5458)
at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:5121)
at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:5618)
at
org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:2912)
at
org.apache.hadoop.hive.ql.metadata.Hive.lambda$loadDynamicPartitions$7(Hive.java:3394)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move
source
s3a://dw-team-bucket/tmp/p_test/.hive-staging_hive_2026-07-27_07-11-26_403_8045012243677926207-34/-ext-10000/b=2/000001_0
to d
estination s3a://dw-team-bucket/tmp/p_test/b=2
at
org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:5486)
at
org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:5441)
at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:5107)
at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:5093)
... 4 more
Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: Failed to rename
s3a://dw-team-bucket/tmp/p_test/.hive-staging_hive_2026-07-27_07-11-26_403_8045012243677926207-34/-ext-10000/b=2/000001_0
to s3a://dw-team-bucket/tmp/p_test/b=2/000001_0_copy_5; destination file exists
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initiateRename(S3AFileSystem.java:2508)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:2573)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$rename$6(S3AFileSystem.java:2434)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2917)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:2432)
at org.apache.hadoop.hive.ql.metadata.Hive.mvFile(Hive.java:5235)
at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:5100)
... 5 more
{code}
was:
Concurrent `INSERT INTO` statements that create the same new dynamic
partition on an external non-ACID Hive table backed by S3 can fail with
`MoveTask return code 40000`. The root cause is a check-then-rename race in
`Hive.mvFile`'s `_copy_N` naming loop: two writers pick the same
destination path between an `exists()` probe and the subsequent
`rename()` call, and the loser gets a `FileAlreadyExistsException` from
`S3AFileSystem.initiateRename`.
This is separate from HIVE-29744, which addresses the *silent* data-loss
twin of this race (concurrent writers overwriting each other with no
error). This ticket is specifically about the *fail-loud* variant that
survives even after HIVE-29744.
{code}
<11>1 2025-03-05T11:36:16.852Z hiveserver2-0 hiveserver2 1
77d253bd-12ed-422f-98a1-242b4d393532 [mdc@38374 class="metadata.Hive"
level="ERROR" thread="load-dynamic-partitionsToAdd-0"] Exception when loading
partition with parameters
partPath=s3a://dev-nfqe-base/cc-cdw-nfqe-vt405x/warehouse/tablespace/external/hive/highpartition_small_impala.db/stocks_highpartition/.hive-staging_hive_2025-03-05_11-36-12_903_341992364183254247-79381/-ext-10000/year=2022/month=Mar/day=6,
table=stocks_highpartition, partSpec={year=2022, month=Mar, day=6},
loadFileType=KEEP_EXISTING, listBucketingLevel=0, isAcid=false,
resetStatistics=true\rorg.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source
s3a://dev-nfqe-base/cc-cdw-nfqe-vt405x/warehouse/tablespace/external/hive/highpartition_small_impala.db/stocks_highpartition/.hive-staging_hive_2025-03-05_11-36-12_903_341992364183254247-79381/-ext-10000/year=2022/month=Mar/day=6/000001_0
to destination
s3a://dev-nfqe-base/cc-cdw-nfqe-vt405x/warehouse/tablespace/external/hive/highpartition_small_impala.db/stocks_highpartition/year=2022/month=Mar/day=6\r
at
org.apache.hadoop.hive.ql.metadata.Hive.handlePoolException(Hive.java:5267)\r
at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:4913)\r at
org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:5427)\r at
org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:2815)\r
at
org.apache.hadoop.hive.ql.metadata.Hive.lambda$loadDynamicPartitions$7(Hive.java:3409)\r
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\r at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\r
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\r
at java.base/java.lang.Thread.run(Thread.java:829)\rCaused by:
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source
s3a://dev-nfqe-base/cc-cdw-nfqe-vt405x/warehouse/tablespace/external/hive/highpartition_small_impala.db/stocks_highpartition/.hive-staging_hive_2025-03-05_11-36-12_903_341992364183254247-79381/-ext-10000/year=2022/month=Mar/day=6/000001_0
to destination
s3a://dev-nfqe-base/cc-cdw-nfqe-vt405x/warehouse/tablespace/external/hive/highpartition_small_impala.db/stocks_highpartition/year=2022/month=Mar/day=6\r
at
org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:5295)\r at
org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:5250)\r at
org.apache.hadoop.hive.ql.metadata.Hive.access$200(Hive.java:262)\r at
org.apache.hadoop.hive.ql.metadata.Hive$2.call(Hive.java:4899)\r at
org.apache.hadoop.hive.ql.metadata.Hive$2.call(Hive.java:4885)\r ... 4
more\rCaused by: org.apache.hadoop.fs.FileAlreadyExistsException: Failed to
rename
s3a://dev-nfqe-base/cc-cdw-nfqe-vt405x/warehouse/tablespace/external/hive/highpartition_small_impala.db/stocks_highpartition/.hive-staging_hive_2025-03-05_11-36-12_903_341992364183254247-79381/-ext-10000/year=2022/month=Mar/day=6/000001_0
to
s3a://dev-nfqe-base/cc-cdw-nfqe-vt405x/warehouse/tablespace/external/hive/highpartition_small_impala.db/stocks_highpartition/year=2022/month=Mar/day=6/000001_0_copy_2;
destination file exists\r at
org.apache.hadoop.fs.s3a.S3AFileSystem.initiateRename(S3AFileSystem.java:2379)\r
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:2444)\r
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$rename$6(S3AFileSystem.java:2305)\r
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:543)\r
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:524)\r
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:445)\r
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2776)\r
at
org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:2303)\r at
org.apache.hadoop.hive.ql.metadata.Hive.mvFile(Hive.java:5028)\r at
org.apache.hadoop.hive.ql.metadata.Hive.access$100(Hive.java:262)\r at
org.apache.hadoop.hive.ql.metadata.Hive$2.call(Hive.java:4892)\r ... 5
more\r
{code}
> Concurrent INSERTs into a new non-ACID dynamic partition fail with
> FileAlreadyExistsException on S3
> ----------------------------------------------------------------------------------------------------
>
> Key: HIVE-28822
> URL: https://issues.apache.org/jira/browse/HIVE-28822
> Project: Hive
> Issue Type: Bug
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Labels: pull-request-available
>
> Concurrent `INSERT INTO` statements that create the same new dynamic
> partition on an external non-ACID Hive table backed by S3 can fail with
> `MoveTask return code 40000`. The root cause is a check-then-rename race in
> `Hive.mvFile`'s `_copy_N` naming loop: two writers pick the same
> destination path between an `exists()` probe and the subsequent
> `rename()` call, and the loser gets a `FileAlreadyExistsException` from
> `S3AFileSystem.initiateRename`.
> This is separate from HIVE-29744, which addresses the *silent* data-loss
> twin of this race (concurrent writers overwriting each other with no
> error). This ticket is specifically about the *fail-loud* variant that
> survives even after HIVE-29744.
> Local repro against table on S3 gave:
> {code}
> 2026-07-27T07:11:51,807 ERROR [load-dynamic-partitionsToAdd-0] metadata.Hive:
> Exception when loading partition with parameters
> partPath=s3a://dw-team-bucket/tmp/p_test/.hive-staging_hive_2026-07-27_07-11-26_4
> 03_8045012243677926207-34/-ext-10000/b=2, table=p_test, partSpec={b=2},
> loadFileType=KEEP_EXISTING, listBucketingLevel=0, isAcid=false,
> resetStatistics=false
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source
> s3a://dw-team-bucket/tmp/p_test/.hive-staging_hive_2026-07-27_07-11-26_403_804501224367
> 7926207-34/-ext-10000/b=2/000001_0 to destination
> s3a://dw-team-bucket/tmp/p_test/b=2
> at
> org.apache.hadoop.hive.ql.metadata.Hive.handlePoolException(Hive.java:5458)
> at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:5121)
> at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:5618)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:2912)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.lambda$loadDynamicPartitions$7(Hive.java:3394)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move
> source
> s3a://dw-team-bucket/tmp/p_test/.hive-staging_hive_2026-07-27_07-11-26_403_8045012243677926207-34/-ext-10000/b=2/000001_0
> to d
> estination s3a://dw-team-bucket/tmp/p_test/b=2
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:5486)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:5441)
> at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:5107)
> at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:5093)
> ... 4 more
> Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: Failed to rename
> s3a://dw-team-bucket/tmp/p_test/.hive-staging_hive_2026-07-27_07-11-26_403_8045012243677926207-34/-ext-10000/b=2/000001_0
> to s3a://dw-team-bucket/tmp/p_test/b=2/000001_0_copy_5; destination file
> exists
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.initiateRename(S3AFileSystem.java:2508)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:2573)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$rename$6(S3AFileSystem.java:2434)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2917)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:2432)
> at org.apache.hadoop.hive.ql.metadata.Hive.mvFile(Hive.java:5235)
> at org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:5100)
> ... 5 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
