Andrew Wong created HIVE-21759:
----------------------------------
Summary: Confusing error message as a result of concurrent drop
and create operations for managed tables
Key: HIVE-21759
URL: https://issues.apache.org/jira/browse/HIVE-21759
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 2.3.1
Environment: Kudu has been working to synchronize its internal catalog
with the Hive Metastore. This issue was caught in the context of a Kudu unit
test when we started using MANAGED tables for the tables created by Kudu. I
note in this [WIP patch|https://gerrit.cloudera.org/c/13315/2] the test in
which I am seeing the flakiness.
Reporter: Andrew Wong
It's possible for concurrent DROP TABLE and CREATE TABLE operations to yield
the following sequence of events:
# The HMS receives a request to create managed table T
# The HMS will create a (HD)FS directory for T if one doesn't already exist
# The HMS will create a metadata entry for table T, making it visible to
clients
# The client sends a DROP TABLE request for table T
# The table entry is dropped in the HMS
# The HMS will eventually drop the directory from (HD)FS
If in between #5 and #6, the client sends another create table request for T,
the request will begin to be processed; the table will not exist in the HMS
(even though its directory exists), so the create will begin to proceed as if
no underlying data exists. Eventually, #6 will complete, and if this coincides
with the part of the second create table operation that enumerates the files in
the directory (e.g. when [generating the CreateTable notification
event|https://github.com/apache/hive/blob/branch-2/hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java#L156]),
the directory will not be there, potentially resulting in the following stack
traces:
{{21:44:29.974 [HMS - ERROR - pool-8-thread-7] (RetryingHMSHandler.java:200)
MetaException(message:java.lang.RuntimeException:
java.io.FileNotFoundException: File
[file:/tmp/dist-test-taskI7tPbq/test-tmp/master-stress-test.0.HmsSentryConfigurations_MasterStressTest.Test_2.1558388453623586-26465/minicluster-data/warehouse/table_c469fde3312d473fb7ac37f526e4dce0|file:///tmp/dist-test-taskI7tPbq/test-tmp/master-stress-test.0.HmsSentryConfigurations_MasterStressTest.Test_2.1558388453623586-26465/minicluster-data/warehouse/table_c469fde3312d473fb7ac37f526e4dce0]
does not exist)}}
{{ at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:6189)}}
{{ at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1553)}}
{{ at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)}}
{{ at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
{{ at java.lang.reflect.Method.invoke(Method.java:498)}}
{{ at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)}}
{{ at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)}}
{{ at com.sun.proxy.$Proxy24.create_table_with_environment_context(Unknown
Source)}}
{{ at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:11191)}}
{{ at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:11175)}}
{{ at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)}}
{{ at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)}}
{{ at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:626)}}
{{ at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:621)}}
{{ at java.security.AccessController.doPrivileged(Native Method)}}
{{ at javax.security.auth.Subject.doAs(Subject.java:422)}}
{{ at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)}}
{{ at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:621)}}
{{ at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)}}
{{ at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}
{{ at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
{{ at java.lang.Thread.run(Thread.java:748)}}
{{ Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File
[file:/tmp/dist-test-taskI7tPbq/test-tmp/master-stress-test.0.HmsSentryConfigurations_MasterStressTest.Test_2.1558388453623586-26465/minicluster-data/warehouse/table_c469fde3312d473fb7ac37f526e4dce0|file:///tmp/dist-test-taskI7tPbq/test-tmp/master-stress-test.0.HmsSentryConfigurations_MasterStressTest.Test_2.1558388453623586-26465/minicluster-data/warehouse/table_c469fde3312d473fb7ac37f526e4dce0]
does not exist}}
{{ at
org.apache.hive.hcatalog.listener.DbNotificationListener$FileIterator.<init>(DbNotificationListener.java:222)}}
{{ at
org.apache.hive.hcatalog.listener.DbNotificationListener.onCreateTable(DbNotificationListener.java:156)}}
{{ at
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier$13.notify(MetaStoreListenerNotifier.java:76)}}
{{ at
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:175)}}
{{ at
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:205)}}
{{ at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1500)}}
{{ at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1539)}}
{{ ... 20 more}}
{{ Caused by: java.io.FileNotFoundException: File
[file:/tmp/dist-test-taskI7tPbq/test-tmp/master-stress-test.0.HmsSentryConfigurations_MasterStressTest.Test_2.1558388453623586-26465/minicluster-data/warehouse/table_c469fde3312d473fb7ac37f526e4dce0|file:///tmp/dist-test-taskI7tPbq/test-tmp/master-stress-test.0.HmsSentryConfigurations_MasterStressTest.Test_2.1558388453623586-26465/minicluster-data/warehouse/table_c469fde3312d473fb7ac37f526e4dce0]
does not exist}}
{{ at
org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:455)}}
{{ at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1537)}}
{{ at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1580)}}
{{ at
org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:675)}}
{{ at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1537)}}
{{ at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1580)}}
{{ at
org.apache.hive.hcatalog.listener.DbNotificationListener$FileIterator.<init>(DbNotificationListener.java:219)}}
{{ ... 26 more}}
The client will see that the create table failed because of a MetaException, in
which a file was not found. This is a somewhat confusing message, though the
end behavior is cut and dry: there will be no HMS entry (should be rolled back
as a part of the transaction) and no directory following this sequence of
events.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
