We should use pd.parse(sql) instead of pd.parse(sql, context);
And when i use pd.parse(sql), i could see exception thrown out like below,
it is due to HIVE-10731 <https://issues.apache.org/jira/browse/HIVE-10731>,
the issue was fixed in 1.3.0, but i use hive-exec-1.2.0.jar in my
application.
java.lang.NullPointerException
at org.apache.hadoop.hive.conf.HiveConf.getBoolVar(HiveConf.java:2583)
at
org.apache.hadoop.hive.ql.parse.HiveParser.useSQL11ReservedKeywordsForIdentifier(HiveParser.java:1000)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.useSQL11ReservedKeywordsForIdentifier(HiveParser_IdentifiersParser.java:726)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.functionName(HiveParser_IdentifiersParser.java:4143)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.synpred11_IdentifiersParser_fragment(HiveParser_IdentifiersParser.java:12082)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.synpred11_IdentifiersParser(HiveParser_IdentifiersParser.java:12180)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser$DFA32.specialStateTransition(HiveParser_IdentifiersParser.java)
at org.antlr.runtime.DFA.predict(DFA.java:80)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.atomExpression(HiveParser_IdentifiersParser.java:6177)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6383)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105)
at
org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840)
at
org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectExpression(HiveParser_SelectClauseParser.java:3755)
at
org.apache.hadoop.hive.ql.parse.HiveParser.selectExpression(HiveParser.java:45854)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.function(HiveParser_IdentifiersParser.java:3850)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.atomExpression(HiveParser_IdentifiersParser.java:6280)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6383)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105)
at
org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840)
at
org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectExpression(HiveParser_SelectClauseParser.java:3755)
at
org.apache.hadoop.hive.ql.parse.HiveParser.selectExpression(HiveParser.java:45854)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.function(HiveParser_IdentifiersParser.java:3878)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.atomExpression(HiveParser_IdentifiersParser.java:6280)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6383)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105)
at
org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840)
at
org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2907)
at
org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1401)
at
org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1128)
at
org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:45827)
at
org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41495)
at
org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402)
at
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413)
at
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283)
at
org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.subQuerySource(HiveParser_FromClauseParser.java:5307)
at
org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromSource(HiveParser_FromClauseParser.java:3741)
at
org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.joinSource(HiveParser_FromClauseParser.java:1910)
at
org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromClause(HiveParser_FromClauseParser.java:1518)
at
org.apache.hadoop.hive.ql.parse.HiveParser.fromClause(HiveParser.java:45873)
at
org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41516)
at
org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402)
at
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413)
at
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283)
at
org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590)
at
org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:161)
at
org.apache.hadoop.hive.ql.tools.LineageInfo.getLineageInfo(LineageInfo.java:113)
at
com.fenbi.pipe.utils.LineageInfoUtils.getInputTables(LineageInfoUtils.java:19)
at com.fenbi.pipe.job.tool.ParseSQLTool.run(ParseSQLTool.java:45)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
com.fenbi.pipe.job.hadoop.AbstractPipeJob.runJob(AbstractPipeJob.java:109)
at com.fenbi.pipe.job.tool.ParseSQLTool.main(ParseSQLTool.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
After i upgrade my hive-exec.jar to 2.0.0, the above exception disappear,
but it introduce another exception like below by some sql. Maybe it is
also a bug like HIVE-10731
java.lang.NullPointerException
at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:2605)
at
org.apache.hadoop.hive.ql.parse.HiveLexer.allowQuotedId(HiveLexer.java:360)
at
org.apache.hadoop.hive.ql.parse.HiveLexer$DFA21.specialStateTransition(HiveLexer.java:11522)
at org.antlr.runtime.DFA.predict(DFA.java:80)
at
org.apache.hadoop.hive.ql.parse.HiveLexer.mIdentifier(HiveLexer.java:8357)
at org.apache.hadoop.hive.ql.parse.HiveLexer.mTokens(HiveLexer.java:11395)
at org.antlr.runtime.Lexer.nextToken(Lexer.java:89)
at org.antlr.runtime.BufferedTokenStream.fetch(BufferedTokenStream.java:133)
at org.antlr.runtime.BufferedTokenStream.sync(BufferedTokenStream.java:127)
at org.antlr.runtime.CommonTokenStream.consume(CommonTokenStream.java:67)
at org.antlr.runtime.DFA.predict(DFA.java:120)
at
org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2847)
at
org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1429)
at
org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1156)
at
org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:48008)
at
org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:42231)
at
org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:42138)
at
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:41154)
at
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:41024)
at
org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1653)
at
org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1137)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:204)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:161)
at
org.apache.hadoop.hive.ql.tools.LineageInfo.getLineageInfo(LineageInfo.java:113)
at
com.fenbi.pipe.utils.LineageInfoUtils.getInputTables(LineageInfoUtils.java:19)
at com.fenbi.pipe.job.tool.ParseSQLTool.run(ParseSQLTool.java:45)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
com.fenbi.pipe.job.hadoop.AbstractPipeJob.runJob(AbstractPipeJob.java:109)
at com.fenbi.pipe.job.tool.ParseSQLTool.main(ParseSQLTool.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
My sql is
select alla.id as questionid,alla.courseid as courseid,alla.grade as
grade,alla.paperid as paperid ,alla.num as num,
alla.type as type,alla.censorstatus as censorstatus,alla.sourcetype as
sourcetype,alla.spidersourcekey as spidersourcekey,t2.toid as
toid,t2.`deleted` as `deleted`,
CASE when sourcetype='1' THEN '小猿搜题'
when sourcetype='2' THEN '猿题库'
when sourcetype='3' THEN '网络资源'
when sourcetype='4' THEN '答题系统'
when sourcetype='5' THEN '合作资源'
when sourcetype='6' THEN 'Koba'
when sourcetype='7' THEN '纸质书'
when sourcetype='8' THEN '框图'
when sourcetype='9' THEN 'PUZZLE' else 'unknown'
end as sourcefrom,
CASE when length(alla.answer)+length(alla.solution)<100 THEN '没有解析'
else '有解析'
end as judgesolution,
CASE when t2.toid is null and t2.`deleted`=0 THEN '不是主原题'
else '是主原题'
end as judgeform
from
(
select aq.* ,top.num from
(select count(*) as num, result from dw_solar_query where result > 0 and
dt>='{{last7Day}}' and dt<='{{yesterday}}'
group by result order by num desc limit 100 ) top
left join apolo_question aq on top.result=aq.id
) alla
left join
(select fromid,`deleted`,toid from sqoop_solar_apolo_question_redirect ) t2
on alla.id=t2.fromid
2016-05-19 12:43 GMT+08:00 Markovitz, Dudu <dmarkov...@paypal.com>:
> Hi
>
>
>
> Can you please share what was the problem?
>
>
>
> Thanks
>
>
>
> Dudu
>
>
>
> *From:* Heng Chen [mailto:heng.chen.1...@gmail.com]
> *Sent:* Thursday, May 19, 2016 7:07 AM
> *To:* user@hive.apache.org
> *Subject:* Re: Could i use Hive SQL parser in our application?
>
>
>
> Got it now! Thanks again for your help! guys!
>
>
>
> 2016-05-19 11:09 GMT+08:00 Heng Chen <heng.chen.1...@gmail.com>:
>
> Hi, guys.
>
>
>
> I write one example as @furcy said like this.
>
>
>
> *public static void *main(String[] args) *throws *SemanticException,
> ParseException, IOException {
> String sql = *"select * from table1 where a > 100"*;
> Context context = *new *Context(*new *HiveConf());
> ParseDriver pd = *new *ParseDriver();
> ASTNode tree = pd.parse(sql, context);
> System.*out*.println(tree);
> }
>
>
>
> When i run it, exception thrown out, did i miss something?
>
>
>
>
>
> Exception in thread "main" java.lang.NullPointerException: Conf non-local
> session path expected to be non-null
>
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
>
> at
> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:669)
>
> at org.apache.hadoop.hive.ql.Context.<init>(Context.java:133)
>
> at org.apache.hadoop.hive.ql.Context.<init>(Context.java:120)
>
> at com.fenbi.pipe.utils.LineageInfoUtils.main(LineageInfoUtils.java:24)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:497)
>
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> 2016-05-19 10:20 GMT+08:00 Heng Chen <heng.chen.1...@gmail.com>:
>
> Thanks guys! Let me try it firstly.
>
>
>
> 2016-05-19 1:44 GMT+08:00 Pengcheng Xiong <pxi...@apache.org>:
>
> Hi Heng,
>
>
>
> Sure you can. Hive SQL parser is based on ANTLR and you can do that by
> taking that part out of Hive and integrate in to your application. Please
> let me know if you need any further help. Thanks.
>
>
>
> Best
>
> Pengcheng Xiong
>
>
>
> On Wed, May 18, 2016 at 3:43 AM, Heng Chen <heng.chen.1...@gmail.com>
> wrote:
>
> Hi, guys.
>
>
>
> Recently, we need to integrate Hive SQL parser in our application.
> Is there any way to do it?
>
>
>
> Thanks!
>
>
>
>
>
>
>
>
>
