I have set up a Kylin instance on the master node of my Hadoop cluster. I was trying on a separate client node, but had some permission issues, so to simplify the test case, I've just installed it on master. Now I am getting the below error.
To correct this, I've tried the solution to distribute the jars in https://issues.apache.org/jira/browse/KYLIN-1082 using ' kylin.job.mr.lib.dir'. I'm not sure how to append to 'kylin.hive.dependency' as I cannot find information on that (perhaps I'm not looking in the right place). But the lib dir setting did not help and it still is unable to find that class. On #2 Step Name: Extract Fact Table Distinct Columns Kylin executes with the following parameters: -conf /opt/kylin/bin/../conf/kylin_job_conf.xml -cubename Testing -output /kylin/kylin_metadata/kylin-40827168-d18f-4b17-a613-3febe773ce2c/Testing/fact_distinct_columns -segmentname 19700101000000_20160731000000 -statisticsenabled true -statisticsoutput /kylin/kylin_metadata/kylin-40827168-d18f-4b17-a613-3febe773ce2c/Testing/statistics -statisticssamplingpercent 100 -jobname Kylin_Fact_Distinct_Columns_Testing_Step Error Msg: 2016-07-27 21:54:03,387 ERROR [pool-6-thread-2] execution.AbstractExecutable:116 : error running Executable java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:81) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:111) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:91) at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91) at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:121) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.mapreduce.HCatInputFormat at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571) ... 12 more 2016-07-27 21:54:03,399 INFO [pool-6-thread-2] manager.ExecutableManager:274 : job id:40827168-d18f-4b17-a613-3febe773ce2c-01 from RUNNING to ERROR 2016-07-27 21:54:03,399 ERROR [pool-6-thread-2] execution.AbstractExecutable:116 : error running Executable org.apache.kylin.job.exception.ExecuteException: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:81) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:111) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:91) at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91) at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:121) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114) ... 6 more Caused by: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.mapreduce.HCatInputFormat at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571) ... 12 more 2016-07-27 21:54:03,410 INFO [pool-6-thread-2] manager.ExecutableManager:274 : job id:40827168-d18f-4b17-a613-3febe773ce2c from RUNNING to ERROR 2016-07-27 21:54:03,410 ERROR [pool-6-thread-2] threadpool.DefaultScheduler:128 : ExecuteException job:40827168-d18f-4b17-a613-3febe773ce2c org.apache.kylin.job.exception.ExecuteException: org.apache.kylin.job.exception.ExecuteException: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.kylin.job.exception.ExecuteException: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114) ... 4 more Caused by: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:81) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:111) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:91) at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91) at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:121) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114) ... 6 more Caused by: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.mapreduce.HCatInputFormat at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571) ... 12 more
