[
https://issues.apache.org/jira/browse/HIVE-28822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-28822:
--------------------------------
Description:
Concurrent `INSERT INTO` statements that create the same new dynamic
partition on an external non-ACID Hive table backed by S3 can fail with
`MoveTask return code 40000`. The root cause is a check-then-rename race in
`Hive.mvFile`'s `_copy_N` naming loop: two writers pick the same
destination path between an `exists()` probe and the subsequent
`rename()` call, and the loser gets a `FileAlreadyExistsException` from
`S3AFileSystem.initiateRename`.
This is separate from HIVE-29744, which addresses the *silent* data-loss
twin of this race (concurrent writers overwriting each other with no
error). This ticket is specifically about the *fail-loud* variant that
survives even after HIVE-29744.
{code}
<11>1 2025-03-05T11:36:16.852Z hiveserver2-0 hiveserver2 1
77d253bd-12ed-422f-98a1-242b4d393532 [mdc@38374 class="metadata.Hive"
level="ERROR" thread="load-dynamic-partitionsToAdd-0"] Exception when loading
partition with parameters
partPath=s3a://dev-nfqe-base/cc-cdw-nfqe-vt405x/warehouse/tablespace/external/hive/highpartition_small_impala.db/stocks_highpartition/.hive-staging_hive_2025-03-05_11-36-12_903_341992364183254247-79381/-ext-10000/year=2022/month=Mar/day=6,
table=stocks_highpartition, partSpec={year=2022, month=Mar, day=6},
loadFileType=KEEP_EXISTING, listBucketingLevel=0, isAcid=false,
resetStatistics=true\rorg.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source
s3a://dev-nfqe-base/cc-cdw-nfqe-vt405x/warehouse/tablespace/external/hive/highpartition_small_impala.db/stocks_highpartition/.hive-staging_hive_2025-03-05_11-36-12_903_341992364183254247-79381/-ext-10000/year=2022/month=Mar/day=6/000001_0
to destination
s3a://dev-nfqe-base/cc-cdw-nfqe-vt405x/warehouse/tablespace/external/hive/highpartition_small_impala.db/stocks_highpartition/year=2022/month=Mar/day=6\r
at
org.apache.hadoop.hive.ql.metadata.Hive.handlePoolException(Hive.java:5267)\r
at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:4913)\r at
org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:5427)\r at
org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:2815)\r
at
org.apache.hadoop.hive.ql.metadata.Hive.lambda$loadDynamicPartitions$7(Hive.java:3409)\r
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\r at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\r
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\r
at java.base/java.lang.Thread.run(Thread.java:829)\rCaused by:
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source
s3a://dev-nfqe-base/cc-cdw-nfqe-vt405x/warehouse/tablespace/external/hive/highpartition_small_impala.db/stocks_highpartition/.hive-staging_hive_2025-03-05_11-36-12_903_341992364183254247-79381/-ext-10000/year=2022/month=Mar/day=6/000001_0
to destination
s3a://dev-nfqe-base/cc-cdw-nfqe-vt405x/warehouse/tablespace/external/hive/highpartition_small_impala.db/stocks_highpartition/year=2022/month=Mar/day=6\r
at
org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:5295)\r at
org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:5250)\r at
org.apache.hadoop.hive.ql.metadata.Hive.access$200(Hive.java:262)\r at
org.apache.hadoop.hive.ql.metadata.Hive$2.call(Hive.java:4899)\r at
org.apache.hadoop.hive.ql.metadata.Hive$2.call(Hive.java:4885)\r ... 4
more\rCaused by: org.apache.hadoop.fs.FileAlreadyExistsException: Failed to
rename
s3a://dev-nfqe-base/cc-cdw-nfqe-vt405x/warehouse/tablespace/external/hive/highpartition_small_impala.db/stocks_highpartition/.hive-staging_hive_2025-03-05_11-36-12_903_341992364183254247-79381/-ext-10000/year=2022/month=Mar/day=6/000001_0
to
s3a://dev-nfqe-base/cc-cdw-nfqe-vt405x/warehouse/tablespace/external/hive/highpartition_small_impala.db/stocks_highpartition/year=2022/month=Mar/day=6/000001_0_copy_2;
destination file exists\r at
org.apache.hadoop.fs.s3a.S3AFileSystem.initiateRename(S3AFileSystem.java:2379)\r
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:2444)\r
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$rename$6(S3AFileSystem.java:2305)\r
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:543)\r
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:524)\r
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:445)\r
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2776)\r
at
org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:2303)\r at
org.apache.hadoop.hive.ql.metadata.Hive.mvFile(Hive.java:5028)\r at
org.apache.hadoop.hive.ql.metadata.Hive.access$100(Hive.java:262)\r at
org.apache.hadoop.hive.ql.metadata.Hive$2.call(Hive.java:4892)\r ... 5
more\r
{code}
was:
Concurrent `INSERT INTO` statements that create the same new dynamic
partition on an external non-ACID Hive table backed by S3 can fail with
`MoveTask return code 40000`. The root cause is a check-then-rename race in
`Hive.mvFile`'s `_copy_N` naming loop: two writers pick the same
destination path between an `exists()` probe and the subsequent
`rename()` call, and the loser gets a `FileAlreadyExistsException` from
`S3AFileSystem.initiateRename`.
This is separate from HIVE-29744, which addresses the *silent* data-loss
twin of this race (concurrent writers overwriting each other with no
error). This ticket is specifically about the *fail-loud* variant that
survives even after HIVE-29744.
> Concurrent INSERTs into a new non-ACID dynamic partition fail with
> FileAlreadyExistsException on S3
> ----------------------------------------------------------------------------------------------------
>
> Key: HIVE-28822
> URL: https://issues.apache.org/jira/browse/HIVE-28822
> Project: Hive
> Issue Type: Bug
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Labels: pull-request-available
>
> Concurrent `INSERT INTO` statements that create the same new dynamic
> partition on an external non-ACID Hive table backed by S3 can fail with
> `MoveTask return code 40000`. The root cause is a check-then-rename race in
> `Hive.mvFile`'s `_copy_N` naming loop: two writers pick the same
> destination path between an `exists()` probe and the subsequent
> `rename()` call, and the loser gets a `FileAlreadyExistsException` from
> `S3AFileSystem.initiateRename`.
> This is separate from HIVE-29744, which addresses the *silent* data-loss
> twin of this race (concurrent writers overwriting each other with no
> error). This ticket is specifically about the *fail-loud* variant that
> survives even after HIVE-29744.
> {code}
> <11>1 2025-03-05T11:36:16.852Z hiveserver2-0 hiveserver2 1
> 77d253bd-12ed-422f-98a1-242b4d393532 [mdc@38374 class="metadata.Hive"
> level="ERROR" thread="load-dynamic-partitionsToAdd-0"] Exception when loading
> partition with parameters
> partPath=s3a://dev-nfqe-base/cc-cdw-nfqe-vt405x/warehouse/tablespace/external/hive/highpartition_small_impala.db/stocks_highpartition/.hive-staging_hive_2025-03-05_11-36-12_903_341992364183254247-79381/-ext-10000/year=2022/month=Mar/day=6,
> table=stocks_highpartition, partSpec={year=2022, month=Mar, day=6},
> loadFileType=KEEP_EXISTING, listBucketingLevel=0, isAcid=false,
> resetStatistics=true\rorg.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source
> s3a://dev-nfqe-base/cc-cdw-nfqe-vt405x/warehouse/tablespace/external/hive/highpartition_small_impala.db/stocks_highpartition/.hive-staging_hive_2025-03-05_11-36-12_903_341992364183254247-79381/-ext-10000/year=2022/month=Mar/day=6/000001_0
> to destination
> s3a://dev-nfqe-base/cc-cdw-nfqe-vt405x/warehouse/tablespace/external/hive/highpartition_small_impala.db/stocks_highpartition/year=2022/month=Mar/day=6\r
> at
> org.apache.hadoop.hive.ql.metadata.Hive.handlePoolException(Hive.java:5267)\r
> at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:4913)\r
> at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:5427)\r at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:2815)\r
> at
> org.apache.hadoop.hive.ql.metadata.Hive.lambda$loadDynamicPartitions$7(Hive.java:3409)\r
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\r
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\r
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\r
> at java.base/java.lang.Thread.run(Thread.java:829)\rCaused by:
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source
> s3a://dev-nfqe-base/cc-cdw-nfqe-vt405x/warehouse/tablespace/external/hive/highpartition_small_impala.db/stocks_highpartition/.hive-staging_hive_2025-03-05_11-36-12_903_341992364183254247-79381/-ext-10000/year=2022/month=Mar/day=6/000001_0
> to destination
> s3a://dev-nfqe-base/cc-cdw-nfqe-vt405x/warehouse/tablespace/external/hive/highpartition_small_impala.db/stocks_highpartition/year=2022/month=Mar/day=6\r
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:5295)\r
> at org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:5250)\r
> at org.apache.hadoop.hive.ql.metadata.Hive.access$200(Hive.java:262)\r at
> org.apache.hadoop.hive.ql.metadata.Hive$2.call(Hive.java:4899)\r at
> org.apache.hadoop.hive.ql.metadata.Hive$2.call(Hive.java:4885)\r ... 4
> more\rCaused by: org.apache.hadoop.fs.FileAlreadyExistsException: Failed to
> rename
> s3a://dev-nfqe-base/cc-cdw-nfqe-vt405x/warehouse/tablespace/external/hive/highpartition_small_impala.db/stocks_highpartition/.hive-staging_hive_2025-03-05_11-36-12_903_341992364183254247-79381/-ext-10000/year=2022/month=Mar/day=6/000001_0
> to
> s3a://dev-nfqe-base/cc-cdw-nfqe-vt405x/warehouse/tablespace/external/hive/highpartition_small_impala.db/stocks_highpartition/year=2022/month=Mar/day=6/000001_0_copy_2;
> destination file exists\r at
> org.apache.hadoop.fs.s3a.S3AFileSystem.initiateRename(S3AFileSystem.java:2379)\r
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:2444)\r
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$rename$6(S3AFileSystem.java:2305)\r
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:543)\r
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:524)\r
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:445)\r
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2776)\r
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:2303)\r
> at org.apache.hadoop.hive.ql.metadata.Hive.mvFile(Hive.java:5028)\r at
> org.apache.hadoop.hive.ql.metadata.Hive.access$100(Hive.java:262)\r at
> org.apache.hadoop.hive.ql.metadata.Hive$2.call(Hive.java:4892)\r ... 5
> more\r
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
