Hello again,
We're now testing our system against a corpus of Hive SQL statements in an
effort to quickly highlight edge cases, limitations etc. We're finding that
org.apache.hadoop.hive.ql.parse.ParseUtils is stumbling on variables such
as ${hiveconf:varname}. Are variable substitutions handled prior to parsing
or within the parser itself? If in a pre-procesing stage, is there any code
or utility classes within Hive that we can use as a reference, or to
provide this functionality?
Cheers,
Elliot.
On 19 February 2018 at 11:10, Elliot West <tea...@gmail.com> wrote:
> Thank you all for your rapid responses; some really useful information and
> pointers in there.
>
> We'll keep the list updated with our progress.
>
> On 18 February 2018 at 19:00, Dharmesh Kakadia <dhkaka...@gmail.com>
> wrote:
>
>> +1 for using ParseDriver for this. I also have used it to intercept and
>> augment query AST.
>>
>> Also, I would echo others sentiment that its quite ugly. It would be
>> great if we can refactor/standardize this. That will make integrating
>> other system a lot easier.
>>
>> Thanks,
>> Dharmesh
>>
>> On Sat, Feb 17, 2018 at 12:07 AM, Furcy Pin <pin.fu...@gmail.com> wrote:
>>
>>> Hi Elliot,
>>>
>>> Actually, I have done quite similar work regarding Hive custom Parsing,
>>> you should have a look at my project: https://github.com/flaminem/flamy
>>>
>>> The Hive parsing related stuff is here: https://github.com/flami
>>> nem/flamy/tree/master/src/main/scala/com/flaminem/flamy/parsing/hive
>>> A good starting point to see how to parse queries is here:
>>> https://github.com/flaminem/flamy/blob/master/src/main/scala
>>> /com/flaminem/flamy/parsing/hive/PopulateParserInfo.scala#L492
>>>
>>>
>>> Basically, all you need is to use a org.apache.hadoop.hive.ql.pars
>>> e.ParseDriver.
>>>
>>> val pd: ParseDriver = new ParseDriver
>>> val tree: ASTNode = pd.parse(query, hiveContext)
>>>
>>> You then get the ASTNode, that you can freely parse and change.
>>> Also, I must say that it is quite ugly to manipulate, and the Presto
>>> Parser seems to be much better designed (but it is not the same syntax,
>>> unfortunately),
>>> I recommend to look at it to get better design ideas.
>>>
>>>
>>> If you want to enrich your Hive syntax like I did (I wanted to be able
>>> to parse ${VARS} in queries),
>>> you will not be able to use the HiveParser without some workaround.
>>> What I did was replacing these ${VARS} by strings "${VARS}" that the
>>> HiveParser would agree to parse,
>>> and that I could recognize afterwards...
>>>
>>> Also, if you are familiar with Scala, I recommend using it, it helps a
>>> lot...
>>>
>>> For instance, I have this class that transforms an AST back into a
>>> string query:
>>> https://github.com/flaminem/flamy/blob/master/src/main/scala
>>> /com/flaminem/flamy/parsing/hive/ast/SQLFormatter.scala
>>> I could never have done something that good looking in Java...
>>>
>>> Finally this method helps a lot to understand how the hell the AST works:
>>> https://github.com/flaminem/flamy/blob/master/src/main/scala
>>> /com/flaminem/flamy/parsing/hive/HiveParserUtils.scala#L593
>>>
>>> Make sure to write *tons* of unit tests too, you'll need them.
>>>
>>> Hope this helps,
>>>
>>> Furcy
>>>
>>>
>>>
>>> On 16 February 2018 at 21:20, Gopal Vijayaraghavan <gop...@apache.org>
>>> wrote:
>>>
>>>>
>>>> > However, ideally we wish to manipulate the original query as
>>>> delivered by the user (or as close to it as possible), and we’re finding
>>>> that the tree has been modified significantly by the time it hits the hook
>>>>
>>>> That's CBO. It takes the Query - > AST -> Calcite Tree -> AST -> hook -
>>>> the bushy join conversion is already done by the time the hook gets called.
>>>>
>>>> We need a Parser hook to hook it ahead of CBO, not a Semantic Analyzer
>>>> hook.
>>>>
>>>> > Additionally we wish to track back ASTNodes to the character
>>>> sequences in the source HQL that were their origin (where sensible), and
>>>> ultimately hope to be able regenerate the query text from the AST.
>>>>
>>>> I started work on a Hive-unparser a while back based on this class, but
>>>> it a world of verbose coding.
>>>>
>>>> https://github.com/apache/hive/blob/master/ql/src/java/org/a
>>>> pache/hadoop/hive/ql/optimizer/calcite/translator/ASTConvert
>>>> er.java#L850
>>>>
>>>> If you're doing active work on this, I'd like to help, because I need
>>>> the AST -> query to debug CBO.
>>>>
>>>> > The use case, if you are interested, is a mutation testing framework
>>>> for HQL. The testing of mutants is operational, but now we need to report
>>>> on survivors, hence the need to track back from specific query elements to
>>>> character sequences in the original query string.
>>>>
>>>> This sounds a lot like the fuzzing random-query-gen used in Cloudera to
>>>> have Impala vs Hive bug-for-bug compat.
>>>>
>>>> https://cwiki.apache.org/confluence/download/attachments/273
>>>> 62054/Random%20Query%20Gen-%20Hive%20Meetup.pptx
>>>>
>>>> Cheers,
>>>> Gopal
>>>>
>>>>
>>>>
>>>
>>
>
