Hi Sam, thank you for sharing all the details. As a next step I would suggest to explore the classpath that Sqoop will end up having (for example using jinfo) and verify whether there are HDFS jars available.
Jarcec
On Mon, Jul 22, 2013 at 12:08:08AM -0700, Abraham Elmahrek wrote:
> Sam,
>
> Have you tried removing the trailing '/' in the value of fs.default.name?
> Also, I believe that is deprecated and should be replaced by fs.defaultFS.
>
> -Abe
>
>
> On Sun, Jul 21, 2013 at 10:42 PM, sam liu <liuqiyun2...@gmail.com> wrote:
>
> > Hi Jarek,
> >
> > Sqoop import tool still failed on my hadoop 2.x cluster, and above is my
> > core-site.xml. can you help take a look at it?
> >
> > Thanks!
> >
> >
> > 2013/7/19 sam liu <liuqiyun2...@gmail.com>
> >
> >> Hi Jarek,
> >>
> >> Yes, the hdfs-site.xml file is in ${HADOOP_HOME}/etc/hadoop. I also added
> >> hadoop related jars into CLASSPATH:
> >> '/home/hadoop-2.0.3-alpha/share/hadoop/common/hadoop-common-2.0.3-alpha.jar:/home/hadoop-2.0.3-alpha/share/hadoop/mapreduce/*.jar:/home/hadoop-2.0.3-alpha/share/hadoop/yarn/*.jar:/home/hadoop-2.0.3-alpha/share/hadoop/hdfs/hadoop-hdfs-2.0.3-alpha.jar:...',
> >> but Sqoop import tool still failed due to same exception.
> >>
> >> Below is my core-site.xml:
> >> <configuration>
> >> <property>
> >> <name>fs.default.name</name>
> >> <value>hdfs://namenode_hostname:9010/</value>
> >> </property>
> >>
> >> <property>
> >> <name>hadoop.tmp.dir</name>
> >> <value>/home/temp/hadoop/core_temp</value>
> >> </property>
> >>
> >> </configuration>
> >>
> >>
> >> Thanks!
> >>
> >>
> >> 2013/7/17 Jarek Jarcec Cecho <jar...@apache.org>
> >>
> >>> Hi Sam,
> >>> thank you for sharing the details. I'm assuming that the hdfs-site.xml
> >>> file is in ${HADOOP_HOME}/etc/hadoop, is that correct? As the
> >>> hdfs-site.xml
> >>> contains more server side configuration of HDFS, I would be more
> >>> interested
> >>> to know the content of core-site.xml file. I would suggest to explore the
> >>> classpath that Sqoop will end up having (for example using jinfo) and
> >>> verify whether there are HDFS jars available.
> >>>
> >>> Jarcec
> >>>
> >>> On Wed, Jul 17, 2013 at 10:54:05AM +0800, sam liu wrote:
> >>> > Hi Jarek,
> >>> >
> >>> > Below are my configurations:
> >>> >
> >>> > 1) Env Parameters:
> >>> > export HADOOP_HOME=/opt/hadoop-2.0.3-alpha
> >>> > export PATH=$HADOOP_HOME/bin:$PATH
> >>> > export PATH=$HADOOP_HOME/sbin:$PATH
> >>> > export HADOOP_MAPARED_HOME=${HADOOP_HOME}
> >>> > export HADOOP_COMMON_HOME=${HADOOP_HOME}
> >>> > export HADOOP_HDFS_HOME=${HADOOP_HOME}
> >>> > export YARN_HOME=${HADOOP_HOME}
> >>> > export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
> >>> > export HDFS_CONF_DIR=${HADOOP_HOME}/etc/hadoop
> >>> > export YARN_CONF_DIR=${HADOOP_HOME}/etc/hadoop
> >>> >
> >>> > 2) hdfs-site.xml:
> >>> > <?xml version="1.0"?>
> >>> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> >>> >
> >>> > <configuration>
> >>> >
> >>> > <property>
> >>> > <name>dfs.replication</name>
> >>> > <value>1</value>
> >>> > </property>
> >>> >
> >>> > <property>
> >>> > <name>dfs.name.dir</name>
> >>> > <value>/home/temp/hadoop/dfs_name_dir</value>
> >>> > </property>
> >>> >
> >>> > <property>
> >>> > <name>dfs.data.dir</name>
> >>> > <value>/home/temp/hadoop/dfs_data_dir</value>
> >>> > </property>
> >>> >
> >>> > <property>
> >>> > <name>dfs.webhdfs.enabled</name>
> >>> > <value>true</value>
> >>> > </property>
> >>> > </configuration>
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > 2013/7/17 Jarek Jarcec Cecho <jar...@apache.org>
> >>> >
> >>> > > Hi sir,
> >>> > > the exception is suggesting that FileSystem implementation for your
> >>> > > default FS can't be found. I would check HDFS configuration to
> >>> ensure that
> >>> > > it's configured properly and that Sqoop is properly picking the
> >>> > > configuration and all the HDFS libraries.
> >>> > >
> >>> > > Jarcec
> >>> > >
> >>> > > On Tue, Jul 16, 2013 at 11:29:07AM +0800, sam liu wrote:
> >>> > > > I also tried another sqoop build
> >>> sqoop-1.4.2.bin__hadoop-2.0.0-alpha on
> >>> > > > hadoop 2.0.3-alpha, but failed as well. The exception is same as
> >>> above:
> >>> > > > 'java.lang.UnsupportedOperationException: Not implemented by the
> >>> > > > DistributedFileSystem FileSystem implementation'.
> >>> > > >
> >>> > > > This issue blocks me quite a while...
> >>> > > >
> >>> > > >
> >>> > > > 2013/6/21 Abraham Elmahrek <a...@cloudera.com>
> >>> > > >
> >>> > > > > Hey Sam,
> >>> > > > >
> >>> > > > > My understanding is that Sqoop 1.4.3 should work with Hadoop
> >>> 2.0.x
> >>> > > (which
> >>> > > > > would include Hadoop 2.0.4 alpha). Any ways, there seems to be
> >>> some
> >>> > > version
> >>> > > > > conflicting going on here. Do you have any other builds of sqoop
> >>> > > installed?
> >>> > > > >
> >>> > > > > -Abe
> >>> > > > >
> >>> > > > >
> >>> > > > > On Thu, Jun 20, 2013 at 6:39 PM, sam liu <liuqiyun2...@gmail.com
> >>> >
> >>> > > wrote:
> >>> > > > >
> >>> > > > >> Anyone could provide a answer? We are making decision whether to
> >>> > > leverage
> >>> > > > >> Sqoop 1.4.3 on yarn or not.
> >>> > > > >>
> >>> > > > >> Thanks!
> >>> > > > >>
> >>> > > > >>
> >>> > > > >> 2013/6/20 sam liu <liuqiyun2...@gmail.com>
> >>> > > > >>
> >>> > > > >>> Hi,
> >>> > > > >>>
> >>> > > > >>> Sqoop website says Sqoop 1.4.3 support Hadoop 2.0, but I
> >>> failed to
> >>> > > run
> >>> > > > >>> import tool against hadoop-2.0.4-alpha, using
> >>> > > > >>> sqoop-1.4.3.bin__hadoop-2.0.0-alpha. Can anyone help provide
> >>> > > > >>> triage/suggestion? Thanks in advance!
> >>> > > > >>>
> >>> > > > >>> - Command:
> >>> > > > >>> sqoop import --connect jdbc:db2://host:50000/SAMPLE --table
> >>> > > > >>> DB2ADMIN.DB2TEST_TBL001 --username user --password pwd -m 1
> >>> > > --target-dir
> >>> > > > >>> /tmp/DB2TEST_TBL001
> >>> > > > >>>
> >>> > > > >>> - Exception:
> >>> > > > >>> 13/06/19 23:28:28 INFO manager.SqlManager: Executing SQL
> >>> statement:
> >>> > > > >>> SELECT t.* FROM DB2ADMIN.DB2TEST_TBL001 AS t WHERE 1=0
> >>> > > > >>> 13/06/19 23:28:28 ERROR sqoop.Sqoop: Got exception running
> >>> Sqoop:
> >>> > > > >>> java.lang.UnsupportedOperationException: Not implemented by the
> >>> > > > >>> DistributedFileSystem FileSystem implementation
> >>> > > > >>> java.lang.UnsupportedOperationException: Not implemented by the
> >>> > > > >>> DistributedFileSystem FileSystem implementation
> >>> > > > >>> at
> >>> > > org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:207)
> >>> > > > >>> at
> >>> > > > >>>
> >>> org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2245)
> >>> > > > >>> at
> >>> > > > >>>
> >>> > >
> >>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2255)
> >>> > > > >>> at
> >>> > > > >>>
> >>> > >
> >>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2272)
> >>> > > > >>> at
> >>> > > org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:86)
> >>> > > > >>> at
> >>> > > > >>>
> >>> > >
> >>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2311)
> >>> > > > >>> at
> >>> > > > >>> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2293)
> >>> > > > >>> at
> >>> org.apache.hadoop.fs.FileSystem.get(FileSystem.java:317)
> >>> > > > >>> at
> >>> > > org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:288)
> >>> > > > >>> at
> >>> > > org.apache.sqoop.mapreduce.JobBase.cacheJars(JobBase.java:134)
> >>> > > > >>> at
> >>> > > > >>>
> >>> > >
> >>> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:197)
> >>> > > > >>> at
> >>> > > > >>>
> >>> org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:413)
> >>> > > > >>> at
> >>> > > > >>>
> >>> org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:380)
> >>> > > > >>> at
> >>> org.apache.sqoop.tool.ImportTool.run(ImportTool.java:453)
> >>> > > > >>> at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> >>> > > > >>> at
> >>> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> >>> > > > >>> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> >>> > > > >>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> >>> > > > >>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> >>> > > > >>> at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> >>> > > > >>> at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)
> >>> > > > >>>
> >>> > > > >>
> >>> > > > >>
> >>> > > > >>
> >>> > > > >> --
> >>> > > > >>
> >>> > > > >> Sam Liu
> >>> > > > >>
> >>> > > > >
> >>> > > > >
> >>> > > >
> >>> > > >
> >>> > > > --
> >>> > > >
> >>> > > > Sam Liu
> >>> > >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> >
> >>> > Sam Liu
> >>>
> >>
> >>
> >>
> >> --
> >>
> >> Sam Liu
> >>
> >
> >
> >
> > --
> >
> > Sam Liu
> >
signature.asc
Description: Digital signature
