My personal two cents from someone who knows very little about the workings of
grammars and regexes is this is quite a bit of a WAT. `regex` is meant to
backtrack and it does, for example here:
<ZoffixW> m: say grammar { regex TOP { <foo> 'foo' }; regex foo { [ ‘a’ ||
‘abc’ ] } }.parse: 'abcfoo'
<camelia> rakudo-moar dfb58d: OUTPUT«｢abcfoo｣ foo => ｢abc｣»
<ZoffixW> m: say grammar { regex TOP { <foo> 'foo' }; token foo { [ ‘a’ ||
‘abc’ ] } }.parse: 'abcfoo'
<camelia> rakudo-moar dfb58d: OUTPUT«Nil»
However, if the `regex` is a top-level rule, then we add an extra requirement
that to enable backtracking the user also needs an anchor to the end of string.
It's a special exception to the rule and would need to be documented in Traps,
but why do we have it?
On Sat, 12 Nov 2016 18:36:15 -0800, alex.jakime...@gmail.com wrote:
> To demonstrate how it works, let's say we have this grammar:
>
> grammar G { regex TOP { ‘a’ || ‘abc’ } };
>
> So you might think that it will match either “a” or “abc”, but no,
> “abc”
> will never work.
>
> *Code:*
> grammar G { regex TOP { ‘a’ || ‘abc’ } };
> say G.parse(‘abc’)
>
> *Result:*
> Nil
>
> Why? Well, see this:
> https://github.com/rakudo/rakudo/blob/cb8f783eeb8ab25a5090fdc4e5cc318c36ee1afa/src/core/Grammar.pm#L23
>
> Basically, TOP will first match ‘a’, and given that it is a good
> enough
> result it will bail out without trying anything else. Then, .parse
> method
> will see that it did not parse the whole string, so it fails.
>
> I am not sure how this behavior could be useful or even how could
> anyone
> expect this, to me it feels like the top rule should always have an
> implicit $ on the end so that it knows that it should keep trying to
> find a
> better solution.
>
> But yes, we can close our eyes on this issue and add yet another trap
> to
> the documentation.
>
> Another closely related issue is method .subparse. As it seems,
> subparse is
> supposed to return partial results (.partial-parse ?), but for some
> reason
> instead of returning Nil it returns a failed Match, unlike .parse.
