Reinier Kip created HIVE-18931:
----------------------------------
Summary: Race condition when ensuring a partition exists often
causes AlreadyExistsException for the partition
Key: HIVE-18931
URL: https://issues.apache.org/jira/browse/HIVE-18931
Project: Hive
Issue Type: Bug
Components: Clients, HCatalog
Affects Versions: 1.2.1
Reporter: Reinier Kip
Hiya! I'm using Apache Beam's HCatalogIO to store data in Hive. As part of
HCatOutputFormatWriter#commit(), partitions are registered in
FileOutputCommitterContainer#registerPartitions(). Here, it checks for the
existence of the partitions, and, if so needed, creates them. For parallel
processes, this fails a lot of the time, because, as far as I understand,
another process creates the partition in the meantime. This causes an
AlreadyExistsException to be created for the partition.
Relevant logic:
[https://github.com/apache/hive/blob/release-1.2.1/hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java#L898-L920].
Logic is also present on master.
Am I missing something, and if not, is it an acceptable solution to swallow the
AlreadyExistsException when adding the partitions?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
