Hello, I am trying to use Sqoop to import data into Hive, and am running the following command:
sqoop import --connect 'jdbc:mysql://dbhost/dbname' --username xxuser --table xxtable --password xxpassword --hive-import --direct and am receiving the following exception from Hive: 11/10/24 20:36:31 INFO hive.HiveImport: Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.security.UserGroupInformation.login(Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/hadoop/security/UserGroupInformation; 11/10/24 20:36:31 INFO hive.HiveImport: at org.apache.hadoop.hive.shims.Hadoop20Shims.getUGIForConf(Hadoop20Shims.java:448) 11/10/24 20:36:31 INFO hive.HiveImport: at org.apache.hadoop.hive.ql.security.HadoopDefaultAuthenticator.setConf(HadoopDefaultAuthenticator.java:51) 11/10/24 20:36:31 INFO hive.HiveImport: at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) 11/10/24 20:36:31 INFO hive.HiveImport: at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) 11/10/24 20:36:31 INFO hive.HiveImport: at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthenticator(HiveUtils.java:222) 11/10/24 20:36:31 INFO hive.HiveImport: at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:241) 11/10/24 20:36:31 INFO hive.HiveImport: at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:466) 11/10/24 20:36:31 INFO hive.HiveImport: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 11/10/24 20:36:31 INFO hive.HiveImport: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 11/10/24 20:36:31 INFO hive.HiveImport: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 11/10/24 20:36:31 INFO hive.HiveImport: at java.lang.reflect.Method.invoke(Method.java:616) 11/10/24 20:36:31 INFO hive.HiveImport: at org.apache.hadoop.util.RunJar.main(RunJar.java:186) I've seen others with the same issue, but no mention of a resolution. I'm fairly certain this is an issue with Hive, but haven't been able to reproduce the issue directly with Hive... only using Sqoop. I'm using the Cloudera-distributed release of all 3 applications (downloaded from https://ccp.cloudera.com/display/SUPPORT/CDH3+Downloadable+Tarballs). The specific versions are: * hadoop-0.20.2-cdh3u2 * hive-0.7.1-cdh3u2 * sqoop-1.3.0-cdh3u2 I have seen that others are using hive 0.7.1 with hadoop 0.20.2, but can't figure out why it's working at all for them. In the file "shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java", getUGIForConf is defined as: public UserGroupInformation getUGIForConf(Configuration conf) throws LoginException { UserGroupInformation ugi = UnixUserGroupInformation.readFromConf(conf, UnixUserGroupInformation.UGI_PROPERTY_NAME); if(ugi == null) { ugi = UserGroupInformation.login(conf); } return ugi; } and the offending line is "ugi = UserGroupInformation.login(conf)". Looking at the source for hadoop's UserGroupInformation.java, there is no "login()" method defined -- the only ones I could see working are "getLoginUser" and "getCurrentUser". Attempting to change the "login" to one of these fails immediately, because 0.20.1 (which Hive is built with) doesn't contain these methods, but 0.20.2 does. Looking at the shems, there's no support for sub-versions, only major version (.19, .20, etc), so I'm not sure how trivial it would be to break it apart into minor releases. Given that I know these two versions should be pretty compatible with each other (since Cloudera is distributing them together), they are stable releases, and I haven't seen others mentioning a problem, does anyone know 1) if this is actually a problem and 2) Is there a workaround or simple fix that can be applied. Thanks, -Josh Braegger
