Yes, it works. Thank you very much, Garry From: Suresh Kumar Sethuramaswamy <rock...@gmail.com> Reply-To: "user@hive.apache.org" <user@hive.apache.org> Date: Wednesday, November 7, 2018 at 3:10 PM To: "user@hive.apache.org" <user@hive.apache.org> Subject: Re: Create external table with s3 location error
Thanks for the logs. Couple of things here, 1. Based on the logs,the HiveServer2 seem to be down. This is an issue if you use Hue/Beeline 2. If you are using Hive CLI, you can still test it, where you have to add the (fs.s3a.access.key,fs.s3.secret.key)s3 keys in hdfs advanced configuration snippet for core-site.xml and hdfs-site.xml via CM.Also for location it should be s3a:// not s3:// On Wed, Nov 7, 2018, 2:58 PM Garry Chen <g...@cornell.edu<mailto:g...@cornell.edu> wrote: Hi Suresh, I am using Hive 1.1.0-cdh5.14.4 and hive server log as below. 2018-11-07 19:43:16,581 WARN [main]: server.HiveServer2 (HiveServer2.java:startHiveServer2(581)) - Error starting HiveServer2 on attempt 1, will retry in 60000ms java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:840) at org.apache.hadoop.hive.ql.session.SessionState.getAuthorizationMode(SessionState.java:1686) at org.apache.hadoop.hive.ql.session.SessionState.isAuthorizationModeV2(SessionState.java:1697) at org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1745) at org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:125) at org.apache.hive.service.cli.CLIService.init(CLIService.java:111) at org.apache.hive.service.CompositeService.init(CompositeService.java:59) at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:125) at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:542) at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:89) at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:793) at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:666) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:226) at org.apache.hadoop.util.RunJar.main(RunJar.java:141) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:391) at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:817) ... 17 more Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProviderBase.setConf(HiveAuthorizationProviderBase.java:114) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:388) ... 18 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:220) at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:338) at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:299) at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:274) at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:256) at org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider.init(DefaultHiveAuthorizationProvider.java:29) at org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProviderBase.setConf(HiveAuthorizationProviderBase.java:112) ... 21 more Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1638) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:67) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:82) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3411) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3430) at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3655) at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:231) at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:215) ... 27 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1636) ... 34 more Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused) at org.apache.thrift.transport.TSocket.open(TSocket.java:226) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:244) at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1636) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:67) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:82) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3411) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3430) at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3655) at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:231) at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:215) at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:338) at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:299) at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:274) at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:256) at org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider.init(DefaultHiveAuthorizationProvider.java:29) at org.apache.hadoop.hive.ql.security.authorization.HiveAuthorizationProviderBase.setConf(HiveAuthorizationProviderBase.java:112) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthorizeProviderManager(HiveUtils.java:388) at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:817) at org.apache.hadoop.hive.ql.session.SessionState.getAuthorizationMode(SessionState.java:1686) at org.apache.hadoop.hive.ql.session.SessionState.isAuthorizationModeV2(SessionState.java:1697) at org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1745) at org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:125) at org.apache.hive.service.cli.CLIService.init(CLIService.java:111) at org.apache.hive.service.CompositeService.init(CompositeService.java:59) at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:125) at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:542) at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:89) at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:793) at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:666) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:226) at org.apache.hadoop.util.RunJar.main(RunJar.java:141) Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.thrift.transport.TSocket.open(TSocket.java:221) ... 42 more ) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:512) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:244) at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74) From: Suresh Kumar Sethuramaswamy <rock...@gmail.com<mailto:rock...@gmail.com>> Reply-To: "user@hive.apache.org<mailto:user@hive.apache.org>" <user@hive.apache.org<mailto:user@hive.apache.org>> Date: Wednesday, November 7, 2018 at 2:50 PM To: "user@hive.apache.org<mailto:user@hive.apache.org>" <user@hive.apache.org<mailto:user@hive.apache.org>> Subject: Re: Create external table with s3 location error Are you using EMR or Apache hadoop open source? Can you share your hive megastore logs? On Wed, Nov 7, 2018, 2:19 PM Garry Chen <g...@cornell.edu<mailto:g...@cornell.edu> wrote: hi All, I am try to create a external table using s3 as location but failed. I add my access key and security key in hive-site.xml and reboot the server. Any suggestion? hive> create external table kv (key int, values string) location 's3://cu-iclick/test'; FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.NullPointerException) Garry
