Forcing Bison's yacc.c to report more expected tokens on syntax error

[Kevin Villela]

Hi all, I saw that the yacc.c skeleton only supports reporting a maximum or
5 expected tokens. The page
 on
Look-Ahead Correction states this is "because of internationalization
considerations". I was wondering if I would be able to contribute a
configuration option to increase this number arbitrarily, or even just up
to a higher number (e.g. 30) ?

Thanks in advance for the help! -Kevin
___
help-bison@gnu.org https://lists.gnu.org/mailman/listinfo/help-bison

Re: Forcing Bison's yacc.c to report more expected tokens on syntax error

[Akim Demaille]

Hi Kevin,

> Le 6 févr. 2019 à 03:01, Kevin Villela  a écrit :
> 
> Hi all, I saw that the yacc.c skeleton only supports reporting a maximum or
> 5 expected tokens. The page
>  on
> Look-Ahead Correction states this is "because of internationalization
> considerations". I was wondering if I would be able to contribute a
> configuration option to increase this number arbitrarily, or even just up
> to a higher number (e.g. 30) ?

The thing is that it's unclear that it actually helps to report all the 
possibilities when there are too many.
___
help-bison@gnu.org https://lists.gnu.org/mailman/listinfo/help-bison

Re: Forcing Bison's yacc.c to report more expected tokens on syntax error

[Adrian Vogelsgesang]

Hi Kevin, hi Akim,

I can imagine cases where having a longer list of expected tokens might be 
useful.
However, I wouldn’t like to have them in a string stitched together by bison, 
but instead I would like to have access to them in a std::vector or 
something similar.
That way, I could do some post-processing of the expected token list (e.g., 
expand the “variable_name” token to all variables which are currently in scope).
The result could then be integrated with, e.g., a dropdown list for 
autocompletion pretty nicely :)

Having programmatic access to the list of expected tokens would probably also 
solve Kevin’s request for reporting more than 5 tokens: he could just do it 
himself in his own code after getting the token list from Bison.

Cheers,
Adrian

From: help-bison  on 
behalf of Akim Demaille 
Date: Wednesday, 6 February 2019 at 10:43
To: Kevin Villela 
Cc: Bison Help 
Subject: Re: Forcing Bison's yacc.c to report more expected tokens on syntax 
error

Hi Kevin,

> Le 6 févr. 2019 à 03:01, Kevin Villela  a écrit :
>
> Hi all, I saw that the yacc.c skeleton only supports reporting a maximum or
> 5 expected tokens. The page
> >
>  on
> Look-Ahead Correction states this is "because of internationalization
> considerations". I was wondering if I would be able to contribute a
> configuration option to increase this number arbitrarily, or even just up
> to a higher number (e.g. 30) ?

The thing is that it's unclear that it actually helps to report all the 
possibilities when there are too many.
___
help-bison@gnu.org 
https://lists.gnu.org/mailman/listinfo/help-bison
___
help-bison@gnu.org https://lists.gnu.org/mailman/listinfo/help-bison

Re: Forcing Bison's yacc.c to report more expected tokens on syntax error

[Derek Clegg]

I agree with Adrian — it would be nicer, in my opinion, to allow users to 
handle syntax errors directly, rather than always going through the bison code 
which formats syntax errors in a fixed way.

This seems like it might be not too hard to do: replace the code starting with

char const* yyformat = YY_NULLPTR;
switch (yycount)
  {
#define YYCASE_(N, S) \
case N:   \
  yyformat = S;   \
break
…
return yyres;

in yysyntax_error_ with a call like

my_yysyntax_error(yyarg, yycount);

where “my_yysyntax_error” is a user-defined function which does the work of 
handling syntax error reporting.

In C++, we could be fancier: have two functions, one for a “pure” syntax error:

void my_yysyntax_error();

and one for syntax errors involving unexpected tokens:

 void my_yysyntax_error(std::string unexpected_token, 
std::vector expected_tokens);

The call would be something like this:

if (yycount == 0) {
  my_yysyntax_error();
} else {
  my_yysyntax_error(yyarg[0], std::vector{yyarg + 1, yyarg + 
yycount});
}

Derek

> On Feb 6, 2019, at 11:23 AM, Adrian Vogelsgesang  
> wrote:
> 
> Hi Kevin, hi Akim,
> 
> I can imagine cases where having a longer list of expected tokens might be 
> useful.
> However, I wouldn’t like to have them in a string stitched together by bison, 
> but instead I would like to have access to them in a std::vector or 
> something similar.
> That way, I could do some post-processing of the expected token list (e.g., 
> expand the “variable_name” token to all variables which are currently in 
> scope).
> The result could then be integrated with, e.g., a dropdown list for 
> autocompletion pretty nicely :)
> 
> Having programmatic access to the list of expected tokens would probably also 
> solve Kevin’s request for reporting more than 5 tokens: he could just do it 
> himself in his own code after getting the token list from Bison.
> 
> Cheers,
> Adrian
> 
> From: help-bison  on 
> behalf of Akim Demaille 
> Date: Wednesday, 6 February 2019 at 10:43
> To: Kevin Villela 
> Cc: Bison Help 
> Subject: Re: Forcing Bison's yacc.c to report more expected tokens on syntax 
> error
> 
> Hi Kevin,
> 
>> Le 6 févr. 2019 à 03:01, Kevin Villela  a écrit :
>> 
>> Hi all, I saw that the yacc.c skeleton only supports reporting a maximum or
>> 5 expected tokens. The page
>> >
>>  on
>> Look-Ahead Correction states this is "because of internationalization
>> considerations". I was wondering if I would be able to contribute a
>> configuration option to increase this number arbitrarily, or even just up
>> to a higher number (e.g. 30) ?
> 
> The thing is that it's unclear that it actually helps to report all the 
> possibilities when there are too many.
> ___
> help-bison@gnu.org 
> https://lists.gnu.org/mailman/listinfo/help-bison
> ___
> help-bison@gnu.org https://lists.gnu.org/mailman/listinfo/help-bison

___
help-bison@gnu.org https://lists.gnu.org/mailman/listinfo/help-bison

Re: Forcing Bison's yacc.c to report more expected tokens on syntax error

[Akim Demaille]

Hi all,

> Le 6 févr. 2019 à 20:53, Derek Clegg  a écrit :
> 
> I agree with Adrian — it would be nicer, in my opinion, to allow users to 
> handle syntax errors directly, rather than always going through the bison 
> code which formats syntax errors in a fixed way.

I also agree with this idea.  But it's not that simple to change the existing 
API, and the last time I asked for comments, I had nothing 
(https://lists.gnu.org/archive/html/bison-patches/2018-11/msg00030.html, 
https://lists.gnu.org/archive/html/bison-patches/2019-01/msg00037.html).  Also, 
the way to denote the culprit (the invalid token) is quite delicate (except for 
the obvious case of keywords).

I will be back on this issue _after_ I have found a satisfactory answer about 
internationalization, and escaping.
___
help-bison@gnu.org https://lists.gnu.org/mailman/listinfo/help-bison

Re: Forcing Bison's yacc.c to report more expected tokens on syntax error

[Kevin Villela]

Thanks everyone for your responses! You hit the nail on the head, ideally I
would ideally want access to the list of tokens.

Akim, I understand your concerns about making API changes. However, I think
the change described by Derek is actually additive - that is, we would
still by default give the existing behavior, but would also provide the
option for users to override this default behavior by providing their own
syntax error function. Would something like that be possible? AFAICT the
other issues (internationalization, escaping) are actually orthogonal to
providing a new syntax error function option (but please correct me if I'm
wrong).

On Wed, Feb 6, 2019 at 12:07 PM Akim Demaille  wrote:

> Hi all,
>
> > Le 6 févr. 2019 à 20:53, Derek Clegg  a écrit :
> >
> > I agree with Adrian — it would be nicer, in my opinion, to allow users
> to handle syntax errors directly, rather than always going through the
> bison code which formats syntax errors in a fixed way.
>
> I also agree with this idea.  But it's not that simple to change the
> existing API, and the last time I asked for comments, I had nothing (
> https://lists.gnu.org/archive/html/bison-patches/2018-11/msg00030.html,
> https://lists.gnu.org/archive/html/bison-patches/2019-01/msg00037.html).
> Also, the way to denote the culprit (the invalid token) is quite delicate
> (except for the obvious case of keywords).
>
> I will be back on this issue _after_ I have found a satisfactory answer
> about internationalization, and escaping.
___
help-bison@gnu.org https://lists.gnu.org/mailman/listinfo/help-bison