Collecting statistics after parsing

2021-05-04 Thread Maury Markowitz
Before I reinvent the wheel: I'm wondering if anyone has some boilerplate code for printing out the frequency of tokens found in the source? Right now I'm counting out specific tokens, like constants for zero and one, strings, etc. This is with explicit code in the bison side that populate some

Re: Collecting statistics after parsing

2021-05-04 Thread Hans Åberg
> On 4 May 2021, at 20:09, Maury Markowitz wrote: > > Before I reinvent the wheel: I'm wondering if anyone has some boilerplate > code for printing out the frequency of tokens found in the source? The Bison parser calls yylex when getting new tokens, so you might it make it.

Re: Collecting statistics after parsing

2021-05-04 Thread Adrian Vogelsgesang via Users list for the GNU Bison parser generator
Hi Maury, Another potential approach: Hook the lexer function. Instead of embedding the counting directly into the bison grammar, you could do the counting directly in the `yylex` function, as a wrapper around your actual lexer. That way, you can separate the “structural concerns” of your langu