As an update for others if you have this problem, I have setup a vm with the same packages, and things are running fine. What I've traced it down to is how the shims are loaded. The shim 2.0S (the correct shim for 0.20.2) is used if the class "org.apache.hadoop.security.UnixUserInformation" doesn't exist (or if it's a secure Hadoop). For some reason, my install is using the 2.0 shim instead of 2.0S like it should. This makes me think that the class "org.apache.hadoop.security.UnixUserInformation" actually exists in my install for some reason, even though I'm using 0.20.2 (and I've searched through the hadoop code, and can't find anything relating to it).
My assumption is that there's a 0.20.1 jar being included somewhere, so I need to track it down and remove it. Thanks, -Josh Braegger On Mon, Oct 24, 2011 at 8:49 PM, Joshua Braegger <rckcl...@gmail.com> wrote: > Hello, > > I am trying to use Sqoop to import data into Hive, and am running the > following command: > > sqoop import --connect 'jdbc:mysql://dbhost/dbname' --username xxuser > --table xxtable --password xxpassword --hive-import --direct > > and am receiving the following exception from Hive: > > 11/10/24 20:36:31 INFO hive.HiveImport: Exception in thread "main" > java.lang.NoSuchMethodError: > org.apache.hadoop.security.UserGroupInformation.login(Lorg/apache/hadoop/conf/Configuration;)Lorg/apache/hadoop/security/UserGroupInformation; > 11/10/24 20:36:31 INFO hive.HiveImport: at > org.apache.hadoop.hive.shims.Hadoop20Shims.getUGIForConf(Hadoop20Shims.java:448) > 11/10/24 20:36:31 INFO hive.HiveImport: at > org.apache.hadoop.hive.ql.security.HadoopDefaultAuthenticator.setConf(HadoopDefaultAuthenticator.java:51) > 11/10/24 20:36:31 INFO hive.HiveImport: at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) > 11/10/24 20:36:31 INFO hive.HiveImport: at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) > 11/10/24 20:36:31 INFO hive.HiveImport: at > org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthenticator(HiveUtils.java:222) > 11/10/24 20:36:31 INFO hive.HiveImport: at > org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:241) > 11/10/24 20:36:31 INFO hive.HiveImport: at > org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:466) > 11/10/24 20:36:31 INFO hive.HiveImport: at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > 11/10/24 20:36:31 INFO hive.HiveImport: at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > 11/10/24 20:36:31 INFO hive.HiveImport: at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > 11/10/24 20:36:31 INFO hive.HiveImport: at > java.lang.reflect.Method.invoke(Method.java:616) > 11/10/24 20:36:31 INFO hive.HiveImport: at > org.apache.hadoop.util.RunJar.main(RunJar.java:186) > > I've seen others with the same issue, but no mention of a resolution. I'm > fairly certain this is an issue with Hive, but haven't been able to > reproduce the issue directly with Hive... only using Sqoop. I'm using the > Cloudera-distributed release of all 3 applications (downloaded from > https://ccp.cloudera.com/display/SUPPORT/CDH3+Downloadable+Tarballs). The > specific versions are: > > * hadoop-0.20.2-cdh3u2 > * hive-0.7.1-cdh3u2 > * sqoop-1.3.0-cdh3u2 > > I have seen that others are using hive 0.7.1 with hadoop 0.20.2, but can't > figure out why it's working at all for them. In the file > "shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java", > getUGIForConf is defined as: > > public UserGroupInformation getUGIForConf(Configuration conf) throws > LoginException { > UserGroupInformation ugi = > UnixUserGroupInformation.readFromConf(conf, > UnixUserGroupInformation.UGI_PROPERTY_NAME); > if(ugi == null) { > ugi = UserGroupInformation.login(conf); > } > return ugi; > } > > and the offending line is "ugi = UserGroupInformation.login(conf)". > Looking at the source for hadoop's UserGroupInformation.java, there is no > "login()" method defined -- the only ones I could see working are > "getLoginUser" and "getCurrentUser". Attempting to change the "login" to > one of these fails immediately, because 0.20.1 (which Hive is built with) > doesn't contain these methods, but 0.20.2 does. Looking at the shems, > there's no support for sub-versions, only major version (.19, .20, etc), so > I'm not sure how trivial it would be to break it apart into minor releases. > > Given that I know these two versions should be pretty compatible with each > other (since Cloudera is distributing them together), they are stable > releases, and I haven't seen others mentioning a problem, does anyone know > 1) if this is actually a problem and 2) Is there a workaround or simple fix > that can be applied. > > Thanks, > -Josh Braegger >
