Hello Ted,
It is OK now, it wxas just oa problem of rights on user hive on /kylin/metadata
on HDFS.
I succeeded in created my first cube.
Unfortunately, I cannot query it. I selected the project, and the cube then
moved to ‘Insight tab’ and typed : select count(*) from kylin_sales. I got the
following message :
null while executing SQL: "select count(*) from kylin_sales LIMIT 50000"
the log says : ‘unknown source’
2018-02-13 09:21:16,854 ERROR [http-bio-7070-exec-6]
controller.BasicController:57 :
org.apache.kylin.rest.exception.InternalErrorException: null
while executing SQL: "select count(*) from kylin_sales LIMIT 50000"
at
org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:479)
at
org.apache.kylin.rest.controller.QueryController.query(QueryController.java:78)
at sun.reflect.GeneratedMethodAccessor205.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
How could I solve this ? By rebuilding the model and then the cube ?
If yes, which commands/process shoud I use ?
Thank you for your help.
Best regards,
Jean-Luc.
De : BELLIER Jean-luc
Envoyé : lundi 12 février 2018 15:20
À : '[email protected]'
<[email protected]<mailto:[email protected]>>
Objet : RE: Cannot rebuild cubes
Hello again Ted,
I did the following :
· launched the command ‘beeline’
· launched the command (under beeline) : !connect
jdbc:hive2://<host>:<port>/<db>;principal=<Server_Principal_of_HiveServer2>
And I got the following message :
Driver: Hive JDBC (version 1.1.0-cdh5.14.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Could this be the source of the problem, and, if yes, how to correct it ?
Thank you for your help.
Best regards,
Jean-Luc.
De : BELLIER Jean-luc
Envoyé : lundi 12 février 2018 14:39
À : [email protected]<mailto:[email protected]>
Objet : RE: Cannot rebuild cubes
Hello again,
My test is not conclusive, because the time slice I chose was before 2012.
There is no data before this year. So it is normal.
I did then another test, to get the data between 2010-01-01 and 2014-02-02, but
I still get the same result : no data available on step 1. I attach my
kylin.log file.
In my kylin.log file, I can find the following error message while trying to
create the temp Hive tables :
2018-02-12 14:30:01,574 INFO [Scheduler 481383589 Job
8a97abb9-69c0-4484-b2e0-ed801dcb845a-161] hive.CreateFlatHiveTableStep:38 :
Connecting to jdbc:hive2://cloudera-manager.hades.rte-france.com:10000/
2018-02-12 14:30:01,754 INFO [Scheduler 481383589 Job
8a97abb9-69c0-4484-b2e0-ed801dcb845a-161] hive.CreateFlatHiveTableStep:38 :
Unknown HS2 problem when communicating with Thrift server.
2018-02-12 14:30:01,754 INFO [Scheduler 481383589 Job
8a97abb9-69c0-4484-b2e0-ed801dcb845a-161] hive.CreateFlatHiveTableStep:38 :
Error: Could not open client transport with JDBC Uri:
jdbc:hive2://cloudera-manager.hades.rte-france.com:10000/: Peer indicated
failure: Unsupported mechanism type PLAIN (state=08S01,code=0)
2018-02-12 14:30:01,754 INFO [Scheduler 481383589 Job
8a97abb9-69c0-4484-b2e0-ed801dcb845a-161] hive.CreateFlatHiveTableStep:38 :
Beeline version 1.1.0-cdh5.14.0 by Apache Hive
2018-02-12 14:30:11,729 DEBUG
[localhost-startStop-1-SendThread(node4.hades.rte-france.com:2181)]
zookeeper.ClientCnxn:717 : Got ping response for sessionid: 0x261895d4a1c0140
after 0ms
Could you please give me any clue ?
NB : my kylin.properties file remains the same as before (see parameters below).
Thank you for your help.
Have a good day.
Best regards,
Jean-Luc.
De : BELLIER Jean-luc [mailto:[email protected]]
Envoyé : lundi 12 février 2018 12:29
À : [email protected]<mailto:[email protected]>
Objet : RE: Cannot rebuild cubes
Hello Ted
Thank you for your reply. What is the command to check what you are suggesting ?
In the meanwhile, I changed the way accessing to hive to ‘beeline’, and faced
to write rights problems to HDFS.
This is now solved.
Now I try to build the cube ‘kylin_sales_cube’. All the tables are present in
Hive, and they contain data. The metadata are also in HBase in ‘kylin_metadata’
table.
What I do not understand is that the job seems to run with no data. In the step
#1, the log tells me ‘No Data Available’
[cid:[email protected]][cid:[email protected]]
Could it be a problem of rights ?
Here is my kylin.properties
## Hive client, valid value [cli, beeline]
kylin.source.hive.client=beeline
#
## Parameters for beeline client, only necessary if hive client is beeline
kylin.source.hive.beeline-params=-n root --hiveconf
hive.security.authorization.sqlstd.confwhitelist.append='mapreduce.job.*|dfs.*'
-u
jdbc:hive2://cloudera-manager.hades.rte-france.com:10000/;principal=hive/[email protected]
#
#kylin.source.hive.keep-flat-table=false
#
## Hive database name for putting the intermediate flat tables
kylin.source.hive.database-for-flat-table=default
The full log files have been attached in this mail (using ‘Diagnosis’ option)
on the job in Web interface.
PS : I noticed the presence of file ‘health-check.sh’ in /tmp/kylin on HDFS.
Is is normal ?
Thank you in advance for your help.
Best regards,
Jean-Luc.
De : Ted Yu [mailto:[email protected]]
Envoyé : vendredi 9 février 2018 22:28
À : [email protected]<mailto:[email protected]>
Objet : Re: Cannot rebuild cubes
In the log, I saw:
2018-02-09 17:39:57,634 ERROR [http-bio-7070-exec-8]
controller.CubeController:337 : Segments overlap:
kylin_sales_cube[20100101000000_20140102000000] and
kylin_sales_cube[20120101000000_20140102000000]
java.lang.IllegalStateException: Segments overlap:
kylin_sales_cube[20100101000000_20140102000000] and
kylin_sales_cube[20120101000000_20140102000000]
at org.apache.kylin.metadata.model.Segments.validate(Segments.java:404)
at org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:393)
at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:349)
at org.apache.kylin.cube.CubeManager.appendSegment(CubeManager.java:485)
at org.apache.kylin.cube.CubeManager.appendSegment(CubeManager.java:455)
at
org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:239)
It seems there was some wrong segment in metadata.
Have you tried dumping out metadata and find the invalid segment ?
Cheers
On Fri, Feb 9, 2018 at 9:18 AM, BELLIER Jean-luc
<[email protected]<mailto:[email protected]>> wrote:
Hello,
I am unable to rebuild cubes. The logs give me the following error :
Exception in thread "Thread-13" java.lang.RuntimeException:
java.lang.IllegalStateException: Cannot start job scheduler due to lack of job
lock
at org.apache.kylin.rest.service.JobService$1.run(JobService.java:131)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Cannot start job scheduler due to
lack of job lock
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler.init(DefaultScheduler.java:211)
at org.apache.kylin.rest.service.JobService$1.run(JobService.java:126)
... 1 more
As far as I can see on this topic, I understand that there should be a process
that is still running or was not remove correctly.
I tried several ‘kylin.sh stop’ / ‘kylin.sh start’ with no effect.
I attach my kylin.out and kylin.log files.
Thank you in advance for your help.
Best regards,
Jean-Luc.
"Ce message est destiné exclusivement aux personnes ou entités auxquelles il
est adressé et peut contenir des informations privilégiées ou confidentielles.
Si vous avez reçu ce document par erreur, merci de nous l'indiquer par retour,
de ne pas le transmettre et de procéder à sa destruction.
This message is solely intended for the use of the individual or entity to
which it is addressed and may contain information that is privileged or
confidential. If you have received this communication by error, please notify
us immediately by electronic mail, do not disclose it and delete the original
message."
"Ce message est destiné exclusivement aux personnes ou entités auxquelles il
est adressé et peut contenir des informations privilégiées ou confidentielles.
Si vous avez reçu ce document par erreur, merci de nous l'indiquer par retour,
de ne pas le transmettre et de procéder à sa destruction.
This message is solely intended for the use of the individual or entity to
which it is addressed and may contain information that is privileged or
confidential. If you have received this communication by error, please notify
us immediately by electronic mail, do not disclose it and delete the original
message."
"Ce message est destiné exclusivement aux personnes ou entités auxquelles il
est adressé et peut contenir des informations privilégiées ou confidentielles.
Si vous avez reçu ce document par erreur, merci de nous l'indiquer par retour,
de ne pas le transmettre et de procéder à sa destruction.
This message is solely intended for the use of the individual or entity to
which it is addressed and may contain information that is privileged or
confidential. If you have received this communication by error, please notify
us immediately by electronic mail, do not disclose it and delete the original
message."
