Hi Sir,
Please retry with the following steps, if it still not works, please let us
know. I have verified it at HDP 3.1.
1. export KAFKA_HOME before you start kylin process (better with equal or
higher than 1.0)
export KAFKA_HOME=XXX
sh bin/kylin.sh start
2. copy mapreduce.tar.gz from hdfs to local
hadoop fs -copyToLocal
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
3. remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz and repackage
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
4. put it back to original place
hadoop fs -copyFromLocal mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce
5. drop and restart the build job and see if it works
Note:
You should change to your specific version, for example: 3.0.1.0-187 and
0.8.2.1.
----------------
Best wishes,
Xiaoxiang Yu
在 2019/10/24 14:24，“田家铮”<[email protected]> 写入:
The error reported in the log is the same as the error reported in the
issue, and the scheme provided by the issue has been tried, but it still cannot
be solved, and the reason is unknown.
This is the logs , please take a look ,thanks
————————————————————————————————————————————————————————————————
2019-10-09 23:58:44,395 ERROR [pool-11-thread-1]
threadpool.DefaultScheduler:116 : ExecuteException
job:dab5ba1f-e1ed-c8fa-409c-29f71715e8e0
org.apache.kylin.job.exception.ExecuteException:
org.apache.kylin.job.exception.ExecuteException:
org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job
job_1570617080701_0005Job Diagnostics:Task failed
task_1570617080701_0005_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0
killedReduces: 0
Failure task Diagnostics:
Error:
org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:182)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kylin.job.exception.ExecuteException:
org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job
job_1570617080701_0005Job Diagnostics:Task failed
task_1570617080701_0005_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0
killedReduces: 0
Failure task Diagnostics:
Error:
org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:182)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
... 4 more
Caused by: org.apache.kylin.engine.mr.exception.MapReduceException: no
counters for job job_1570617080701_0005Job Diagnostics:Task failed
task_1570617080701_0005_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0
killedReduces: 0
Failure task Diagnostics:
Error:
org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167)
... 6 more
2019-10-09 23:58:44,396 INFO [FetcherRunner 1609136206-49]
threadpool.DefaultFetcherRunner:85 : Job Fetcher: 0 should running, 0 actual
running, 0 stopped, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 others
2019-10-09 23:58:44,400 DEBUG [http-nio-7070-exec-7]
cachesync.Broadcaster:246 : Broadcasting UPDATE, execute_output,
dab5ba1f-e1ed-c8fa-409c-29f71715e8e0
2019-10-09 23:58:44,401 DEBUG [http-nio-7070-exec-7]
cachesync.Broadcaster:280 : Done broadcasting UPDATE, execute_output,
dab5ba1f-e1ed-c8fa-409c-29f71715e8e0
2019-10-09 23:59:12,827 INFO [FetcherRunner 1609136206-49]
threadpool.DefaultFetcherRunner:85 : Job Fetcher: 0 should running, 0 actual
running, 0 stopped, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 others
————————————————————————————————————————————————————————————————
------------------ Original ------------------
From: "Xiaoxiang Yu"<[email protected]>;
Date: Thu, Oct 24, 2019 01:32 PM
To: "[email protected]"<[email protected]>;
"[email protected]"<[email protected]>;
Subject: Re: kylin2.6.4流式构建不能与hdp3.1.4.0使用(issue3970)
Hi Sir,
I cannot see your screenshot, could you please paste the stack trace (in
kylin.log) or provided more description to us?
And have you ever try use the workaround which was provided in KYLIN-3970
to fix your problem?
----------------
Best wishes,
Xiaoxiang Yu
发件人: 田家铮 <[email protected]>
答复: "[email protected]" <[email protected]>
日期: 2019年10月24日 星期四 11:44
收件人: dev <[email protected]>
主题: kylin2.6.4流式构建不能与hdp3.1.4.0使用(issue3970)
环境配置：
kylin-2.6.4
HDP-3.1.4。0
issue:
3970
https://issues.apache.org/jira/browse/KYLIN-3970?jql=text%20~%20%22partition.assignment.strategy%22<https://issues.apache.org/jira/browse/KYLIN-3970?jql=text%20%7E%20%22partition.assignment.strategy%22>
问题：
流式构建读取kafka数据时，无法构建，但是hive构建没问题
[cid:[email protected]]
[cid:[email protected]]
