[
https://issues.apache.org/jira/browse/HIVE-9543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14335926#comment-14335926
]
Junyong Li commented on HIVE-9543:
----------------------------------
Hi, please look at
https://github.com/apache/hive/blob/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L6617
getMetaStoreSchemaVersion() will return null if it have MysqlException. this
will cause Hive to insert another version record. I think maybe I met this
situation
> MetaException(message:Metastore contains multiple versions)
> -----------------------------------------------------------
>
> Key: HIVE-9543
> URL: https://issues.apache.org/jira/browse/HIVE-9543
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.13.1
> Reporter: Junyong Li
>
> When i run bin/hive command, i got the following exception:
> {noformat}
> Logging initialized using configuration in
> jar:file:/home/hadoop/apache-hive-0.13.1-bin/lib/hive-common-0.13.1.jar!/hive-log4j.properties
> Exception in thread "main" java.lang.RuntimeException:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
> ... 7 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
> ... 12 more
> Caused by: MetaException(message:Metastore contains multiple versions)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:6368)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:6330)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6289)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6277)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
> at com.sun.proxy.$Proxy9.verifySchema(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:476)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:523)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:397)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:356)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:171)
> ... 17 more
> {noformat}
> And i have found two record in metastore table VERSION. after reading source
> code, i found following code maybe cause the problem:
> In the org.apache.hadoop.hive.metastore.ObjectStore.java:6289:
> {noformat}
> String schemaVer = getMetaStoreSchemaVersion();
> if (schemaVer == null) {
> // metastore has no schema version information
> if (strictValidation) {
> throw new MetaException("Version information not found in
> metastore. ");
> } else {
> LOG.warn("Version information not found in metastore. "
> + HiveConf.ConfVars.METASTORE_SCHEMA_VERIFICATION.toString() +
> " is not enabled so recording the schema version " +
> MetaStoreSchemaInfo.getHiveSchemaVersion());
>
> setMetaStoreSchemaVersion(MetaStoreSchemaInfo.getHiveSchemaVersion(),
> "Set by MetaStore");
> }
> }
> {noformat}
> If there is exception in the {{getMetaStoreSchemaVersion()}}, Hive will
> insert a duplication record
> And i also found my another exception maybe the same cause as above.the
> exception is :
> {noformat}
> Exception in thread "main" java.lang.RuntimeException:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
> ... 7 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
> ... 12 more
> Caused by: javax.jdo.JDODataStoreException: Insert of object
> "org.apache.hadoop.hive.metastore.model.MRoleMap@2fd02622" using statement
> "INSERT INTO `ROLE_MAP` (`ROLE_GRANT_ID`,`GRANTOR`,`A
> DD_TIME`,`ROLE_ID`,`GRANTOR_TYPE`,`GRANT_OPTION`,`PRINCIPAL_NAME`,`PRINCIPAL_TYPE`)
> VALUES (?,?,?,?,?,?,?,?)" failed : Duplicate entry 'hadoop-1-admin-ROLE' for
> key 'USERROLEMAPINDEX'
> NestedThrowables:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Duplicate entry 'hadoop-1-admin-ROLE' for key 'USERROLEMAPINDEX'
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.grantRole(ObjectStore.java:3058)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
> at com.sun.proxy.$Proxy9.grantRole(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.addAdminUsers(HiveMetaStore.java:611)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:399)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:356)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:55)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:60)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:171)
> ... 17 more
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Duplicate entry 'hadoop-1-admin-ROLE' for key 'USERROLEMAPINDEX'
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
> at com.mysql.jdbc.Util.getInstance(Util.java:381)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1015)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3515)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3447)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1951)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2101)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2554)
> at
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1761)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2046)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1964)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1949)
> at
> com.jolbox.bonecp.PreparedStatementHandle.executeUpdate(PreparedStatementHandle.java:205)
> at
> org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeUpdate(ParamLoggingPreparedStatement.java:399)
> at
> org.datanucleus.store.rdbms.SQLController.executeStatementUpdate(SQLController.java:439)
> at
> org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:410)
> at
> org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(RDBMSPersistenceHandler.java:167)
> at
> org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:143)
> at
> org.datanucleus.state.JDOStateManager.internalMakePersistent(JDOStateManager.java:3784)
> at
> org.datanucleus.state.JDOStateManager.makePersistent(JDOStateManager.java:3760)
> at
> org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2219)
> at
> org.datanucleus.ExecutionContextImpl.persistObjectWork(ExecutionContextImpl.java:2065)
> at
> org.datanucleus.ExecutionContextImpl.persistObject(ExecutionContextImpl.java:1913)
> at
> org.datanucleus.ExecutionContextThreadedImpl.persistObject(ExecutionContextThreadedImpl.java:217)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:727)
> ... 32 more
> {noformat}
> The following code maybe cause the exception, in In the
> org.apache.hadoop.hive.metastore.ObjectStore.java:3018:
> {noformat}
> MRole nameCheck = this.getMRole(roleName);
> if (nameCheck != null) {
> throw new InvalidObjectException("Role " + roleName + " already
> exists.");
> }
> int now = (int)(System.currentTimeMillis()/1000);
> MRole mRole = new MRole(roleName, now,
> ownerName);
> pm.makePersistent(mRole);
> commited = commitTransaction();
> success = true;
> {noformat}
> If there is exception in the {{getMRole(roleName)}} function, Hive will try
> to insert a duplicate record into table ROLE_MAP, this will cause exception
> above.
> So can we just throw the exception instead of ignoring the exception in the
> {{getMRole(ObjectStore.java:3292)}} and in the
> {{getMetaStoreSchemaVersion(ObjectStore.java:6326)}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
