Ok so after rereading the tutorial and testing again and again , I think I
have finally managed to understand how SmaCC really works and I was
succesful into converting simple python litsts to pharo arrays and ordered
collections.
The tricky part now is to apply this knowledge to complex python types like
multi dimensional lists, tuples and dictionaries. I understand that the
visitor allows me to visit a specific object instances each time they are
found in the AST . But because I want to walk the AST in order to build
multi dimensional ordered collections I need something more, or maybe my
understanding of the visitor pattern is flawed.
The problem I am having here is that each time I parse a python type that
python type is not necessarily represented by different kind of node. For
example whether its a list or a tuple or a dictionary the same class is
used PyAtomNode. In order to differentiate between those diffirent python
types PyAtomNode has instance variables for right and left bracket,
parantheses, and curly. So my initial thinking is to check those instance
variables to see if they are nil and from that I can conclude which python
type I am parsing.
So I can preform simple ifs that check that the instance variable is Nil or
not but the question is if my strategy is a good one or a bad one.
I could define my own syntax to simplify the AST tree including different
nodes for different python types , because from the looks of it , it seems
it is a bit too verbose for my needs but On the other hand I am not so sure
because in the future my needs may become more verbose too.
So I am very close and ready to create my full python types converter for
pharo but I wanted some good advice before wasting time on something that
is not efficient.
By the way Thierry I have to agree with you Smacc is a very capable parser,
also I like the use of regex syntax, makes it uglier compared Pettit Parser
but I prefer the compact regex syntax to having to define and browse tons
of classes and send tons of messages. Also the Python support is very good
and I am impressed how easily SmaCC can parse whole python applications
since some of the test are very complex. Well done great work!
On Sat, Jan 31, 2015 at 12:04 AM, kilon alios <kilon.al...@gmail.com> wrote:
> thank for your congratulations, because at times I fear I ask too obvious
> questions. I have to say I find this parsing very complex but very
> fascinating too :) Time to experiment with the visitor.
>
> On Fri, Jan 30, 2015 at 11:49 PM, Thierry Goubier <
> thierry.goub...@gmail.com> wrote:
>
>>
>>
>> 2015-01-30 14:04 GMT+01:00 kilon alios <kilon.al...@gmail.com>:
>>
>>> Ok thanks for the info, I am still however curious about these "tests"
>>> are just tests (which may or may not happen) that determine the AST, for
>>> example which node to use ?
>>>
>>
>> 'tests' is just there because, in the grammar, there is this at a certain
>> point:
>>
>> testlist:
>> test 'test' "," testlist
>> | test 'test' comma_opt
>> ;
>>
>> I have named this use of test 'test', so SmaCC has deduced that testlist
>> will be a list of test(s) nodes (or maybe other stuff such as atoms,
>> depending on the productions for test).. so, SmaCC, in each rule where
>> testlist is found, it will add a 'tests' instance variable.
>>
>> So, basically, the grammar rules explain how each node can be decomposed
>> in sub-nodes, and the additional annotations (the 'test' naming and the
>> {{}} or {{}}) drive how the classes for the nodes you want to keep are
>> generated. In that testlist case, no node will be generated, but everywhere
>> testlist appear on the right of the rule, then it will add a 'tests'
>> instance variable.
>>
>>
>>>
>>> Or are they tests related to unit testing class PythonParserTests ?
>>>
>>
>> Not at all :)
>>
>>
>>>
>>> Also you said I need to use the visitor created by PythonParser I assume
>>> you mean PyRootNodeVisitor ? Just as it is explained in the AST chapter of
>>> the documentation.
>>>
>>> In my case this simple python list will need me to subclass it and
>>> override method visitListmaker , the aListmaker passed as argument to the
>>> method should I assume it is PyListmakerNode ?
>>>
>>
>> In my experience, what you need to do is you have a look at the ast
>> generated and see if you can recognize the elements. From what I see in
>> your simple example, the key to your list is that PyAtomNode instance with
>> something in list. Once you have that, you know that you need to visit
>> PyAtomNode (and check that it has the [ ] tokens).
>>
>> Looking into what is listmaker in atom in the grammar (congratulations by
>> the way, you have seen it :) ), you'll see that it creates a listmaker node
>> only in the first case: test followed by a list_for, otherwise it falls
>> back to testlist...
>>
>> Thierry
>>
>>
>>
>>>
>>> On Fri, Jan 30, 2015 at 10:50 AM, Thierry Goubier <
>>> thierry.goub...@gmail.com> wrote:
>>>
>>>> Hi kilon,
>>>>
>>>> The tests instance variable is linked to the python grammar: top level
>>>> items in an expression are probably tests, and, through the grammar, tests
>>>> can be just atoms.
>>>>
>>>> So the tests instance variable doesn't means it is testing anything :)
>>>>
>>>> Thierry
>>>> Le 30 janv. 2015 09:23, "kilon alios" <kilon.al...@gmail.com> a écrit :
>>>>
>>>> Ok so I tried to parse a very simple list like
>>>>>
>>>>> [ 67,12342,5 ]
>>>>>
>>>>> using Parse and explore , I can find these number by following this
>>>>> AST (for example 67)
>>>>>
>>>>> PyFileInputNode>>statements: -> 1: PySimpleStmNode>>stmts: -> 1:
>>>>> PyExprStmtNode>>tests: ->1: PyPowerNode>>atom: -> PyAtomNode>>list: -> 1:
>>>>> PyPowerNode>>atom: -> PyPowerNode>>numberToken -> numberToken>>value -> 67
>>>>>
>>>>> quite a structure, but the one thing I dont get is "tests" , why
>>>>> "tests" ?
>>>>>
>>>>> Does it tests something and if yes what ?
>>>>>
>>>>
>>>
>>
>
