Hi, I wrote this hive ql ast visualizer tool last summer using JRuby,hope it would help. here: https://github.com/liprais/hive_ast_tree_vis
2016-05-19 14:31 GMT+08:00 Heng Chen <heng.chen.1...@gmail.com>: > We should use pd.parse(sql) instead of pd.parse(sql, context); > > And when i use pd.parse(sql), i could see exception thrown out like below, > it is due to HIVE-10731 <https://issues.apache.org/jira/browse/HIVE-10731>, > the issue was fixed in 1.3.0, but i use hive-exec-1.2.0.jar in my > application. > > java.lang.NullPointerException > at org.apache.hadoop.hive.conf.HiveConf.getBoolVar(HiveConf.java:2583) > at > org.apache.hadoop.hive.ql.parse.HiveParser.useSQL11ReservedKeywordsForIdentifier(HiveParser.java:1000) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.useSQL11ReservedKeywordsForIdentifier(HiveParser_IdentifiersParser.java:726) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.functionName(HiveParser_IdentifiersParser.java:4143) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.synpred11_IdentifiersParser_fragment(HiveParser_IdentifiersParser.java:12082) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.synpred11_IdentifiersParser(HiveParser_IdentifiersParser.java:12180) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser$DFA32.specialStateTransition(HiveParser_IdentifiersParser.java) > at org.antlr.runtime.DFA.predict(DFA.java:80) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.atomExpression(HiveParser_IdentifiersParser.java:6177) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6383) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105) > at > org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectExpression(HiveParser_SelectClauseParser.java:3755) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectExpression(HiveParser.java:45854) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.function(HiveParser_IdentifiersParser.java:3850) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.atomExpression(HiveParser_IdentifiersParser.java:6280) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6383) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105) > at > org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectExpression(HiveParser_SelectClauseParser.java:3755) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectExpression(HiveParser.java:45854) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.function(HiveParser_IdentifiersParser.java:3878) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.atomExpression(HiveParser_IdentifiersParser.java:6280) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6383) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455) > at > org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105) > at > org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2907) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1401) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1128) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:45827) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41495) > at > org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283) > at > org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.subQuerySource(HiveParser_FromClauseParser.java:5307) > at > org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromSource(HiveParser_FromClauseParser.java:3741) > at > org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.joinSource(HiveParser_FromClauseParser.java:1910) > at > org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromClause(HiveParser_FromClauseParser.java:1518) > at > org.apache.hadoop.hive.ql.parse.HiveParser.fromClause(HiveParser.java:45873) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41516) > at > org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283) > at > org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590) > at > org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109) > at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202) > at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166) > at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:161) > at > org.apache.hadoop.hive.ql.tools.LineageInfo.getLineageInfo(LineageInfo.java:113) > at > com.fenbi.pipe.utils.LineageInfoUtils.getInputTables(LineageInfoUtils.java:19) > at com.fenbi.pipe.job.tool.ParseSQLTool.run(ParseSQLTool.java:45) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at > com.fenbi.pipe.job.hadoop.AbstractPipeJob.runJob(AbstractPipeJob.java:109) > at com.fenbi.pipe.job.tool.ParseSQLTool.main(ParseSQLTool.java:54) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144) > > > > > After i upgrade my hive-exec.jar to 2.0.0, the above exception disappear, > but it introduce another exception like below by some sql. Maybe it is > also a bug like HIVE-10731 > > > java.lang.NullPointerException > at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:2605) > at > org.apache.hadoop.hive.ql.parse.HiveLexer.allowQuotedId(HiveLexer.java:360) > at > org.apache.hadoop.hive.ql.parse.HiveLexer$DFA21.specialStateTransition(HiveLexer.java:11522) > at org.antlr.runtime.DFA.predict(DFA.java:80) > at > org.apache.hadoop.hive.ql.parse.HiveLexer.mIdentifier(HiveLexer.java:8357) > at org.apache.hadoop.hive.ql.parse.HiveLexer.mTokens(HiveLexer.java:11395) > at org.antlr.runtime.Lexer.nextToken(Lexer.java:89) > at > org.antlr.runtime.BufferedTokenStream.fetch(BufferedTokenStream.java:133) > at org.antlr.runtime.BufferedTokenStream.sync(BufferedTokenStream.java:127) > at org.antlr.runtime.CommonTokenStream.consume(CommonTokenStream.java:67) > at org.antlr.runtime.DFA.predict(DFA.java:120) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2847) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1429) > at > org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1156) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:48008) > at > org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:42231) > at > org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:42138) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:41154) > at > org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:41024) > at > org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1653) > at > org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1137) > at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:204) > at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166) > at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:161) > at > org.apache.hadoop.hive.ql.tools.LineageInfo.getLineageInfo(LineageInfo.java:113) > at > com.fenbi.pipe.utils.LineageInfoUtils.getInputTables(LineageInfoUtils.java:19) > at com.fenbi.pipe.job.tool.ParseSQLTool.run(ParseSQLTool.java:45) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at > com.fenbi.pipe.job.hadoop.AbstractPipeJob.runJob(AbstractPipeJob.java:109) > at com.fenbi.pipe.job.tool.ParseSQLTool.main(ParseSQLTool.java:54) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144) > > > > My sql is > > select alla.id as questionid,alla.courseid as courseid,alla.grade as > grade,alla.paperid as paperid ,alla.num as num, > alla.type as type,alla.censorstatus as censorstatus,alla.sourcetype as > sourcetype,alla.spidersourcekey as spidersourcekey,t2.toid as > toid,t2.`deleted` as `deleted`, > CASE when sourcetype='1' THEN '小猿搜题' > when sourcetype='2' THEN '猿题库' > when sourcetype='3' THEN '网络资源' > when sourcetype='4' THEN '答题系统' > when sourcetype='5' THEN '合作资源' > when sourcetype='6' THEN 'Koba' > when sourcetype='7' THEN '纸质书' > when sourcetype='8' THEN '框图' > when sourcetype='9' THEN 'PUZZLE' else 'unknown' > end as sourcefrom, > CASE when length(alla.answer)+length(alla.solution)<100 THEN '没有解析' > else '有解析' > end as judgesolution, > CASE when t2.toid is null and t2.`deleted`=0 THEN '不是主原题' > else '是主原题' > end as judgeform > from > ( > select aq.* ,top.num from > (select count(*) as num, result from dw_solar_query where result > 0 and > dt>='{{last7Day}}' and dt<='{{yesterday}}' > group by result order by num desc limit 100 ) top > left join apolo_question aq on top.result=aq.id > ) alla > left join > (select fromid,`deleted`,toid from sqoop_solar_apolo_question_redirect ) > t2 > on alla.id=t2.fromid > > > > > 2016-05-19 12:43 GMT+08:00 Markovitz, Dudu <dmarkov...@paypal.com>: > >> Hi >> >> >> >> Can you please share what was the problem? >> >> >> >> Thanks >> >> >> >> Dudu >> >> >> >> *From:* Heng Chen [mailto:heng.chen.1...@gmail.com] >> *Sent:* Thursday, May 19, 2016 7:07 AM >> *To:* user@hive.apache.org >> *Subject:* Re: Could i use Hive SQL parser in our application? >> >> >> >> Got it now! Thanks again for your help! guys! >> >> >> >> 2016-05-19 11:09 GMT+08:00 Heng Chen <heng.chen.1...@gmail.com>: >> >> Hi, guys. >> >> >> >> I write one example as @furcy said like this. >> >> >> >> *public static void *main(String[] args) *throws *SemanticException, >> ParseException, IOException { >> String sql = *"select * from table1 where a > 100"*; >> Context context = *new *Context(*new *HiveConf()); >> ParseDriver pd = *new *ParseDriver(); >> ASTNode tree = pd.parse(sql, context); >> System.*out*.println(tree); >> } >> >> >> >> When i run it, exception thrown out, did i miss something? >> >> >> >> >> >> Exception in thread "main" java.lang.NullPointerException: Conf non-local >> session path expected to be non-null >> >> at >> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) >> >> at >> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:669) >> >> at org.apache.hadoop.hive.ql.Context.<init>(Context.java:133) >> >> at org.apache.hadoop.hive.ql.Context.<init>(Context.java:120) >> >> at com.fenbi.pipe.utils.LineageInfoUtils.main(LineageInfoUtils.java:24) >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:497) >> >> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144) >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> 2016-05-19 10:20 GMT+08:00 Heng Chen <heng.chen.1...@gmail.com>: >> >> Thanks guys! Let me try it firstly. >> >> >> >> 2016-05-19 1:44 GMT+08:00 Pengcheng Xiong <pxi...@apache.org>: >> >> Hi Heng, >> >> >> >> Sure you can. Hive SQL parser is based on ANTLR and you can do that >> by taking that part out of Hive and integrate in to your application. >> Please let me know if you need any further help. Thanks. >> >> >> >> Best >> >> Pengcheng Xiong >> >> >> >> On Wed, May 18, 2016 at 3:43 AM, Heng Chen <heng.chen.1...@gmail.com> >> wrote: >> >> Hi, guys. >> >> >> >> Recently, we need to integrate Hive SQL parser in our >> application. Is there any way to do it? >> >> >> >> Thanks! >> >> >> >> >> >> >> >> >> > >
