[
https://issues.apache.org/jira/browse/HIVE-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077005#comment-16077005
]
Dan Burkert edited comment on HIVE-17008 at 7/6/17 6:46 PM:
------------------------------------------------------------
My comment yesterday evening was a bit brief because I was low on time, but
here's the exact sequence of events that leads to the NPE which originally
prompted this issue. As we've discussed it's just one of many inter-related
issues in the class, but I wanted to make it clear what's happening in this
specific case:
1. Application calls {{ThriftHiveMetastore.drop_database}} with a non-existent
database name via the HMS thrift API¹.
2. In {{HiveMetaStore.drop_database_core}}, the {{db}} local variable is
[initialized to
{{null}}|https://github.com/apache/hive/blob/555f001146c4fc471e29e18899a0e02a4043cca5/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L1023].
3. In {{HiveMetaStore.drop_database_core}}, the [lookup of the non-existent
database
fails|https://github.com/apache/hive/blob/555f001146c4fc471e29e18899a0e02a4043cca5/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L1029],
leaving {{db}} set to {{null}} and unwinding to the {{finally}} block.
4. In {{HiveMetaSotre.drop_database_core}} {{finally}} block, a new
[{{DropDatabaseEvent}} is
created|https://github.com/apache/hive/blob/555f001146c4fc471e29e18899a0e02a4043cca5/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L1139-L1143]
with the {{null}} database, and listeners are notified with this event.
5. Somewhere (I haven't traced this bit), the notification log event listener
is dereferencing the null database.
¹ Although I haven't reproduced it, it should be possible to reproduce this bug
through the Java {{HiveMetastoreClient}} API as well, but it would require
concurrent DDL operations. The {{HiveMetastoreClient}} [checks that the
database
exists|https://github.com/apache/hive/blob/master/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java#L851-L858]
before attempting to drop it; this is a benign TOCTOU which makes it difficult
to reproduce using that API (again, it should still possible with the right
interleavings of concurrent DDL ops). A light skimming through that class
reveals that the client is very aggressive about validating state exists before
issuing DDL operations. Just my opinion, but that's a big code smell; the
client should rely on the server to validate arguments.
was (Author: danburkert):
My comment yesterday evening was a bit brief because I was low on time, but
here's the exact sequence of events that leads to the NPE which originally
prompted this issue. As we've discussed it's just one of many inter-related
issues in the class, but I wanted to make it clear what's happening in this
specific case:
1. Application calls {{ThriftHiveMetastore.drop_database}} with a non-existent
database name via the HMS thrift API¹.
2. In {{HiveMetaStore.drop_database_core}}, the {{db}} local variable is
[initialized to
{{null}}|https://github.com/apache/hive/blob/555f001146c4fc471e29e18899a0e02a4043cca5/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L1023].
3. In {{HiveMetaStore.drop_database_core}}, the [lookup of the non-existent
database
fails|https://github.com/apache/hive/blob/555f001146c4fc471e29e18899a0e02a4043cca5/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L1029],
leaving {{db}} set to {{null}} and unwinding to the {{finally}} block.
4. In {{HiveMetaSotre.drop_database_core}} {{finally}} block, a new
[{{DropDatabaseEvent}} is
created|https://github.com/apache/hive/blob/555f001146c4fc471e29e18899a0e02a4043cca5/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L1139-L1143]
with the {{null}} database, and listeners are notified with this event.
5. Somewhere (I haven't traced this bit), the notification log event listener
is dereferencing the null database.
¹ Although I haven't reproduced it, it should be possible to reproduce this bug
through the Java {{HiveMetastoreClient}} API as well, but it would require
concurrent DDL operations. The {{HiveMetastoreClient}} [checks that the
database
exists|https://github.com/apache/hive/blob/master/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java#L851-L858]
before attempting to drop it; this is a benign TOCTOU which makes it difficult
to reproduce using that API (again, it should still possible with the right
interleavings of concurrent DDL ops). A light skimming through that class
reveals that the client is _very_ aggressive about validating state exists
before issuing DDL operations. Just my opinion, but that's a big code smell;
the client should rely on the server to validate arguments.
> HiveMetastore.drop_database can return NPE if database does not exist
> ---------------------------------------------------------------------
>
> Key: HIVE-17008
> URL: https://issues.apache.org/jira/browse/HIVE-17008
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Dan Burkert
> Assignee: Dan Burkert
> Attachments: HIVE-17008.0.patch
>
>
> When dropping a non-existent database, the HMS will still fire registered
> {{DROP_DATABASE}} event listeners. This results in an NPE when the listeners
> attempt to deref the {{null}} database parameter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
