I have two cluster A and B.
A works as main cluster, with hive and mr running on it and no hbase;
B works as hbase cluster, with hbase running on it and no hive.
the following is my local config files:
hadoop core-site.xml (points to Cluster A)
<property>
<name>fs.defaultFS</name>
<value>hdfs://A</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>xxxx</value>
</property>
hadoop hdfs-site.xml (points to hdfs in Cluster A)
<property>
<name>dfs.nameservices</name>
<value>c3prc-hadoop</value>
</property>
hbase client core-site.xml (points to Cluster B)
<property>
<name>fs.defaultFS</name>
<value>hdfs://c3prc-xiaomi98</value>
</property>
hbase hdfs-site.xml (points to hdfs in Cluster B)
<property>
<name>dfs.nameservices</name>
<value>c3prc-xiaomi98</value>
</property>
hive client points to Cluster A
Here's my problem:
when kylin runs my job, it throws the following exception In the step Create
Intermediate Flat Hive Table:
2016-10-26 10:27:05,885 ERROR [pool-7-thread-4]
hive.CreateFlatHiveTableStep:125 : job:1a2ae09d-5cb9-4c99-83a4-8e0d53b29966-02
execute finished with exception
java.lang.IllegalArgumentException: java.net.UnknownHostException: A
at
org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:377)
at
org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:249)
at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:151)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:599)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:544)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:147)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2405)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2439)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2421)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
at
org.apache.kylin.source.hive.CreateFlatHiveTableStep.readRowCountFromFile(CreateFlatHiveTableStep.java:52)
at
org.apache.kylin.source.hive.CreateFlatHiveTableStep.doWork(CreateFlatHiveTableStep.java:112)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:116)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:61)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:116)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:137)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
This exception happens when kylin tries to read the rowCountFile created in
step Create Count File.
In my case, this file is created in Cluster A. It seems that kylin has loaded
local hbase config files. So it cannot parse Cluster A.
is there any suggestion?
thanks.
________________________________
With Best regards
Yu Kuixing
________________________________
发件人: 于奎星
发送时间: 2016年10月26日 10:36
收件人: user
主题: 答复: which hdfs should the property "kylin.hdfs.working.dir" point to when
deploying kylin with standalone hbase cluster
Yes, I configured kylin.hbase.cluster.fs to my hbase cluster.
And I want to know, what does "kylin.hdfs.working.dir" exactly mean?
Because it is confusing when kylin works with two different hdfs.
________________________________
With Best regards
Yu Kuixing
________________________________
发件人: Billy(Yiming) Liu <[email protected]>
发送时间: 2016年10月26日 10:08
收件人: user
主题: Re: which hdfs should the property "kylin.hdfs.working.dir" point to when
deploying kylin with standalone hbase cluster
use kylin.hbase.cluster.fs
2016-10-26 9:46 GMT+08:00 于奎星
<[email protected]<mailto:[email protected]>>:
hi
now I'm trying this:
http://kylin.apache.org/blog/2016/06/10/standalone-hbase-cluster/
and I'm not sure which hdfs the property "kylin.hdfs.working.dir" should point
to, hdfs in main cluster or hdfs in hbase cluter?
Any help would be great.
thanks.
________________________________
With Best regards
Yu Kuixing
--
With Warm regards
Yiming Liu (刘一鸣)
