Package: libopentoken-dev
Version: 3.0b-8
Severity: normal
I'm the author of the Ada lexer in OpenToken. The following code fragment is
not correctly analyzed: Character'('x')
The result is
IDENTIFIER_T Character
CHARACTER_T '('
IDENTIFIER_T x
TICK_T '
RIGHT_PARENTHESIS_T )
This would be the correct result:
IDENTIFIER_T Character
TICK_T '
LEFT_PARENTHESIS_T (
CHARACTER_T 'x'
RIGHT_PARENTHESIS_T )
I have a solution for this, but am not overly happy with it since I had to
change the lexer interface, which introduces an incompatibility.
I have no idea how to change the character literal recognizer itself in order
to deliver the correct result. It would need to know the previous token (a
Character_Literal cannot follow an Identifier, so it must be a Tick that
follows the identifier Character).
So I changed the lexer to handle this case by disabling the recognizer after an
identifier and enabling it else.
Please see the changes in the attachments.
(The Character_Set change in fact has nothing to do with the above - I only
removed an unused declaration.)
I'm not a Debian user.
-------------------------------------------------------------------------------
--
-- Copyright (C) 1999, 2008 Christoph Karl Walter Grein
--
-- This file is part of the OpenToken package.
--
-- The OpenToken package is free software; you can redistribute it and/or
-- modify it under the terms of the GNU General Public License as published
-- by the Free Software Foundation; either version 2, or (at your option)
-- any later version. The OpenToken package is distributed in the hope that
-- it will be useful, but WITHOUT ANY WARRANTY; without even the implied
-- warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU General Public License for more details. You should have received
-- a copy of the GNU General Public License distributed with the OpenToken
-- package; see file GPL.txt. If not, write to the Free Software Foundation,
-- 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
--
-- As a special exception, if other files instantiate generics from this
-- unit, or you link this unit with other files to produce an executable,
-- this unit does not by itself cause the resulting executable to be
-- covered by the GNU General Public License. This exception does not
-- however invalidate any other reasons why the executable file might be
-- covered by the GNU Public License.
--
-- Maintainer: Christoph K. W. Grein ([EMAIL PROTECTED])
--
-- Update History:
-- $Log: test_ada_lexer.adb,v $
-- Revision 1.3 2000/01/27 21:16:09 Ted
-- Fix to use new text feeder objects
--
-- Revision 1.2 1999/12/27 19:56:05 Ted
-- fix file contents to work w/ new hierarchy
--
-- Revision 1.1 1999/08/17 03:36:53 Ted
-- Initial Version
--
-- 0.1 - 23 June 1999 Input via command line parameter; bug fix for EOF
-- 0.0 - 22 June 1999 First preliminary release
-------------------------------------------------------------------------------
with Ada.Text_IO;
with Ada.Command_Line;
with Ada_Lexer;
use Ada_Lexer;
procedure Test_Ada_Lexer is
-- Global text file for reading parse data
File: Ada.Text_IO.File_Type;
begin
Ada.Text_IO.Open (File => File,
Mode => Ada.Text_IO.In_File,
Name => Ada.Command_Line.Argument (1));
Set_Input_Feeder (File);
Bad_Token_on_Syntax_Error;
loop
Find_Next;
Ada.Text_IO.Put_Line (Ada_Token'Image (Token_ID) & ' ' & Lexeme);
exit when Token_ID = End_of_File_T;
end loop;
end Test_Ada_Lexer;
-------------------------------------------------------------------------------
--
-- Copyright (C) 1999, 2008 Christoph Karl Walter Grein
--
-- This file is part of the OpenToken package.
--
-- The OpenToken package is free software; you can redistribute it and/or
-- modify it under the terms of the GNU General Public License as published
-- by the Free Software Foundation; either version 2, or (at your option)
-- any later version. The OpenToken package is distributed in the hope that
-- it will be useful, but WITHOUT ANY WARRANTY; without even the implied
-- warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU General Public License for more details. You should have received
-- a copy of the GNU General Public License distributed with the OpenToken
-- package; see file GPL.txt. If not, write to the Free Software Foundation,
-- 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
--
-- As a special exception, if other files instantiate generics from this
-- unit, or you link this unit with other files to produce an executable,
-- this unit does not by itself cause the resulting executable to be
-- covered by the GNU General Public License. This exception does not
-- however invalidate any other reasons why the executable file might be
-- covered by the GNU Public License.
--
-- Maintainer: Christoph K. W. Grein ([EMAIL PROTECTED])
--
-- Update History:
-- $Log: ada_lexer.ads,v $
-- Revision 1.5 2000/08/07 00:17:43 Ted
-- Change to work w/ new package hierarchy
--
-- Revision 1.4 2000/01/27 21:21:05 Ted
-- Fix to work with 2.0
--
-- Revision 1.3 1999/12/27 19:56:05 Ted
-- fix file contents to work w/ new hierarchy
--
-- Revision 1.2 1999/10/08 23:19:01 Ted
-- Disable sign recognition in integer and real literals
--
-- Revision 1.1 1999/08/17 03:40:24 Ted
-- Initial Version
--
-- 1.0 - 8 August 1999 Final complete version
-- 0.3 - 26 June 1999 Added character literals
-- 0.2 - 25 June 1999 Added based numbers
-- (still missing: character literals and strings)
-- 0.1 - 23 June 1999 Bug fix (numeric literals)
-- 0.0 - 22 June 1999 First preliminary release
-------------------------------------------------------------------------------
with Ada.Strings.Maps.Constants;
with OpenToken.Token.Enumerated;
with OpenToken.Token.Enumerated.Analyzer;
with OpenToken.Recognizer.Keyword, OpenToken.Recognizer.Separator;
with OpenToken.Recognizer.Identifier;
with OpenToken.Recognizer.Graphic_Character, OpenToken.Recognizer.String;
with OpenToken.Recognizer.Integer, OpenToken.Recognizer.Based_Integer_Ada_Style,
OpenToken.Recognizer.Real, OpenToken.Recognizer.Based_Real_Ada_Style;
with OpenToken.Recognizer.Character_Set;
with OpenToken.Recognizer.Line_Comment;
with OpenToken.Recognizer.Nothing;
with OpenToken.Recognizer.End_Of_File;
with OpenToken.Text_Feeder.Text_IO;
pragma Elaborate_All (OpenToken.Token.Enumerated,
OpenToken.Token.Enumerated.Analyzer,
OpenToken.Recognizer.Keyword,
OpenToken.Recognizer.Separator,
OpenToken.Recognizer.Identifier,
OpenToken.Recognizer.Graphic_Character,
OpenToken.Recognizer.String,
OpenToken.Recognizer.Integer,
OpenToken.Recognizer.Based_Integer_Ada_Style,
OpenToken.Recognizer.Real,
OpenToken.Recognizer.Based_Real_Ada_Style,
OpenToken.Recognizer.Character_Set,
OpenToken.Recognizer.Line_Comment,
OpenToken.Recognizer.Nothing,
OpenToken.Recognizer.End_Of_File,
OpenToken.Text_Feeder.Text_IO);
package body Ada_Lexer is
package Master_Ada_Token is new OpenToken.Token.Enumerated (Ada_Token);
package Tokenizer is new Master_Ada_Token.Analyzer;
Syntax : constant Tokenizer.Syntax :=
(Abort_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("abort")),
Abs_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get ("abs")),
Abstract_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("abstract")),
Accept_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("accept")),
Access_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("access")),
Aliased_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("aliased")),
All_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get ("all")),
And_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get ("and")),
Array_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("array")),
At_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get ("at")),
Begin_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("begin")),
Body_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("body")),
Case_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("case")),
Constant_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("constant")),
Declare_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("declare")),
Delay_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("delay")),
Delta_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("delta")),
Digits_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("digits")),
Do_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get ("do")),
Else_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("else")),
Elsif_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("elsif")),
End_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get ("end")),
Entry_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("entry")),
Exception_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("exception")),
Exit_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("exit")),
For_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get ("for")),
Function_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("function")),
Generic_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("generic")),
Goto_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("goto")),
If_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get ("if")),
In_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get ("in")),
Interface_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("interface")),
Is_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get ("is")),
Limited_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("limited")),
Loop_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("loop")),
Mod_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get ("mod")),
New_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get ("new")),
Not_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get ("not")),
Null_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("null")),
Of_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get ("of")),
Or_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get ("or")),
Others_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("others")),
Out_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get ("out")),
Overriding_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("overriding")),
Package_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("package")),
Pragma_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("pragma")),
Private_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("private")),
Procedure_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("procedure")),
Protected_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("protected")),
Raise_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("raise")),
Range_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("range")),
Record_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("record")),
Rem_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get ("rem")),
Renames_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("renames")),
Requeue_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("requeue")),
Return_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("return")),
Reverse_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("reverse")),
Select_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("select")),
Separate_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("separate")),
Subtype_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("subtype")),
Synchronized_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("synchronized")),
Tagged_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("tagged")),
Task_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("task")),
Terminate_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("terminate")),
Then_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("then")),
Type_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("type")),
Until_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("until")),
Use_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get ("use")),
When_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("when")),
While_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("while")),
With_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get
("with")),
Xor_T => Tokenizer.Get (OpenToken.Recognizer.Keyword.Get ("xor")),
Colon_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
(":")),
Comma_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
(",")),
Dot_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
(".")),
Semicolon_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
(";")),
Tick_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
("'")),
Left_Parenthesis_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
("(")),
Right_Parenthesis_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
(")")),
Concatenate_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
("&")),
Alternative_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
("|")),
Equal_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
("=")),
Not_Equal_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
("/=")),
Greater_Than_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
(">")),
Less_Than_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
("<")),
Greater_Equal_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
(">=")),
Less_Equal_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
("<=")),
Plus_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
("+")),
Minus_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
("-")),
Times_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
("*")),
Divide_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
("/")),
Arrow_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
("=>")),
Assignment_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
(":=")),
Double_Dot_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
("..")),
Exponentiate_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
("**")),
Left_Label_Bracket_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
("<<")),
Right_Label_Bracket_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
(">>")),
Box_T => Tokenizer.Get (OpenToken.Recognizer.Separator.Get
("<>")),
Integer_T => Tokenizer.Get (OpenToken.Recognizer.Integer.Get
(Allow_Signs => False)),
Based_Integer_T => Tokenizer.Get
(OpenToken.Recognizer.Based_Integer_Ada_Style.Get),
Real_T => Tokenizer.Get (OpenToken.Recognizer.Real.Get
(Allow_Signs => False)),
Based_Real_T => Tokenizer.Get
(OpenToken.Recognizer.Based_Real_Ada_Style.Get),
Character_T => Tokenizer.Get
(OpenToken.Recognizer.Graphic_Character.Get),
String_T => Tokenizer.Get (OpenToken.Recognizer.String.Get),
Identifier_T => Tokenizer.Get (OpenToken.Recognizer.Identifier.Get),
Comment_T => Tokenizer.Get (OpenToken.Recognizer.Line_Comment.Get
("--")),
Whitespace_T => Tokenizer.Get (OpenToken.Recognizer.Character_Set.Get
(OpenToken.Recognizer.Character_Set.Standard_Whitespace)),
Bad_Token_T => Tokenizer.Get (OpenToken.Recognizer.Nothing.Get),
End_of_File_T => Tokenizer.Get (OpenToken.Recognizer.End_Of_File.Get));
Analyzer: Tokenizer.Instance := Tokenizer.Initialize (Syntax);
procedure Set_Input_Feeder (File: in Ada.Text_IO.File_Type) is
begin
Ada.Text_IO.Set_Input (File);
Tokenizer.Input_Feeder := OpenToken.Text_Feeder.Text_IO.Create;
end Set_Input_Feeder;
procedure Exception_on_Syntax_Error is
begin
Tokenizer.Unset_Default (Analyzer);
end Exception_on_Syntax_Error;
procedure Bad_Token_on_Syntax_Error is
begin
Tokenizer.Set_Default (Analyzer, Bad_Token_T);
end Bad_Token_on_Syntax_Error;
procedure Set_Comments_Reportable (To: in Boolean) is
begin
Syntax (Comment_T).Recognizer.Report := To;
end Set_Comments_Reportable;
Exclusion: constant array (Boolean) of Ada.Strings.Maps.Character_Set := --
see Find_Next
(False => Ada.Strings.Maps.Null_Set, -- character literal
enabled
True => Ada.Strings.Maps.Constants.Graphic_Set); --
disabled
procedure Find_Next is
-- Take care that the expression Character'('x') is correctly processed:
-- A character literal cannot follow an identifier.
begin
Tokenizer.Find_Next (Analyzer, Look_Ahead => False);
OpenToken.Recognizer.Graphic_Character.Redefine
(OpenToken.Recognizer.Graphic_Character.Instance (Syntax
(Character_T).Recognizer.all),
Exclusion (Token_ID = Identifier_T));
end Find_Next;
function Line return Natural is
begin
return Tokenizer.Line (Analyzer);
end Line;
function Column return Natural is
begin
return Tokenizer.Column (Analyzer);
end Column;
function Token_ID return Ada_Token is
begin
return Tokenizer.ID (Analyzer);
end Token_ID;
function Lexeme return String is
begin
return Tokenizer.Lexeme (Analyzer);
end Lexeme;
end Ada_Lexer;
-------------------------------------------------------------------------------
--
-- Copyright (C) 1999, 2008 Christoph Karl Walter Grein
--
-- This file is part of the OpenToken package.
--
-- The OpenToken package is free software; you can redistribute it and/or
-- modify it under the terms of the GNU General Public License as published
-- by the Free Software Foundation; either version 2, or (at your option)
-- any later version. The OpenToken package is distributed in the hope that
-- it will be useful, but WITHOUT ANY WARRANTY; without even the implied
-- warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU General Public License for more details. You should have received
-- a copy of the GNU General Public License distributed with the OpenToken
-- package; see file GPL.txt. If not, write to the Free Software Foundation,
-- 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
--
-- As a special exception, if other files instantiate generics from this
-- unit, or you link this unit with other files to produce an executable,
-- this unit does not by itself cause the resulting executable to be
-- covered by the GNU General Public License. This exception does not
-- however invalidate any other reasons why the executable file might be
-- covered by the GNU Public License.
--
-- Maintainer: Christoph K. W. Grein ([EMAIL PROTECTED])
--
-- Update History:
-- $Log: ada_lexer.ads,v $
-- Revision 1.5 2000/08/07 00:17:43 Ted
-- Change to work w/ new package hierarchy
--
-- Revision 1.4 2000/01/27 21:21:05 Ted
-- Fix to work with 2.0
--
-- Revision 1.3 1999/12/27 19:56:05 Ted
-- fix file contents to work w/ new hierarchy
--
-- Revision 1.2 1999/10/08 23:19:01 Ted
-- Disable sign recognition in integer and real literals
--
-- Revision 1.1 1999/08/17 03:40:24 Ted
-- Initial Version
--
-- 1.0 - 8 August 1999 Final complete version
-- 0.3 - 26 June 1999 Added character literals
-- 0.2 - 25 June 1999 Added based numbers
-- (still missing: character literals and strings)
-- 0.1 - 23 June 1999 Bug fix (numeric literals)
-- 0.0 - 22 June 1999 First preliminary release
-------------------------------------------------------------------------------
with Ada.Text_IO;
package Ada_Lexer is
---------------------------------------------------------------------
-- This ia a lexical analyser for the Ada language.
--
-- There is another lexer for the Ada and Java languages at:
-- <http://home.T-Online.de/home/Christ-Usch.Grein/Ada/Lexer.html>
---------------------------------------------------------------------
type Ada_Token is
(-- Reserved words ARM 2.9 (2)
Abort_T, Abs_T, Abstract_T, Accept_T, Access_T, Aliased_T, All_T, And_T,
Array_T, At_T,
Begin_T, Body_T,
Case_T, Constant_T,
Declare_T, Delay_T, Delta_T, Digits_T, Do_T,
Else_T, Elsif_T, End_T, Entry_T, Exception_T, Exit_T,
For_T, Function_T,
Generic_T, Goto_T,
If_T, In_T, Interface_T, Is_T,
Limited_T, Loop_T,
Mod_T,
New_T, Not_T, Null_T,
Of_T, Or_T, Others_T, Out_T, Overriding_T,
Package_T, Pragma_T, Private_T, Procedure_T, Protected_T,
Raise_T, Range_T, Record_T, Rem_T, Renames_T, Requeue_T, Return_T,
Reverse_T,
Select_T, Separate_T, Subtype_T, Synchronized_T,
Tagged_T, Task_T, Terminate_T, Then_T, Type_T,
Until_T, Use_T,
When_T, While_T, With_T,
Xor_T,
-- Delimiters ARM 2.2 (9)
-- & ' ( ) * + , - . / : ; < = > |
-- Compound delimiters ARM 2.2 (11)
-- => .. ** := /= >= <= << >> <>
Colon_T, Comma_T, Dot_T, Semicolon_T, Tick_T, -- : , . ; '
Left_Parenthesis_T, Right_Parenthesis_T, -- ( )
Concatenate_T, -- &
Alternative_T, -- |
Equal_T, Not_Equal_T, Greater_Than_T, Less_Than_T, -- = /= > <
Greater_Equal_T, Less_Equal_T, -- >= <=
Plus_T, Minus_T, Times_T, Divide_T, -- + - * /
Arrow_T, Assignment_T, Double_Dot_T, Exponentiate_T, -- => := .. **
Left_Label_Bracket_T, Right_Label_Bracket_T, Box_T, -- << >> <>
-- Literals ARM 2.4 .. 2.6
Integer_T, -- 1, 1E+10
Based_Integer_T, -- 13#C#, 13#C#E+10
Real_T, -- -3.141, 1.0E+10
Based_Real_T, -- 13#C.B#, 13#C.B#E+5
Character_T, String_T,
-- Other tokens
Identifier_T,
Comment_T,
Whitespace_T,
-- Syntax error
Bad_Token_T,
--
End_of_File_T);
-- Define the file where to find the code to be processed.
-- The file must be open for reading.
procedure Set_Input_Feeder (File: in Ada.Text_IO.File_Type);
-- In case of syntax errors:
-- Define whether the Syntax_Error exception shall be raised (default)
-- or the Bad_Token_T token shall be be returned.
procedure Exception_on_Syntax_Error;
procedure Bad_Token_on_Syntax_Error;
-- Change reportability of comments (off by default).
procedure Set_Comments_Reportable (To: in Boolean);
-- Find the next reportable token.
procedure Find_Next;
-- Query the current token:
function Line return Natural;
function Column return Natural;
function Token_ID return Ada_Token;
function Lexeme return String;
end Ada_Lexer;
-------------------------------------------------------------------------------
--
-- Copyright (C) 1999 FlightSafety International and Ted Dennison
--
-- This file is part of the OpenToken package.
--
-- The OpenToken package is free software; you can redistribute it and/or
-- modify it under the terms of the GNU General Public License as published
-- by the Free Software Foundation; either version 2, or (at your option)
-- any later version. The OpenToken package is distributed in the hope that
-- it will be useful, but WITHOUT ANY WARRANTY; without even the implied
-- warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU General Public License for more details. You should have received
-- a copy of the GNU General Public License distributed with the OpenToken
-- package; see file GPL.txt. If not, write to the Free Software Foundation,
-- 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
--
-- As a special exception, if other files instantiate generics from this
-- unit, or you link this unit with other files to produce an executable,
-- this unit does not by itself cause the resulting executable to be
-- covered by the GNU General Public License. This exception does not
-- however invalidate any other reasons why the executable file might be
-- covered by the GNU Public License.
--
-- Maintainer: Ted Dennison ([EMAIL PROTECTED])
--
-- This software was originally developed by the following company, and was
-- released as open-source software as a service to the community:
--
-- FlightSafety International Simulation Systems Division
-- Broken Arrow, OK USA 918-259-4000
--
-- Update History:
-- $Log: opentoken-recognizer-character_set.adb,v $
-- Revision 1.2 1999/12/27 19:56:00 Ted
-- fix file contents to work w/ new hierarchy
--
-- Revision 1.1 1999/12/27 17:11:33 Ted
-- renamed everything to new hierarchy
--
-- Revision 1.2 1999/08/17 03:06:59 Ted
-- Add log line
--
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
-- This package implements a token recognizer for a token of any number of
-- characters from a given set.
-------------------------------------------------------------------------------
package body Opentoken.Recognizer.Character_Set is
----------------------------------------------------------------------------
-- This procedure will be called when analysis on a new candidate string
-- is started. The Token needs to clear its state (if any).
----------------------------------------------------------------------------
procedure Clear (The_Token: in out Instance) is
begin
The_Token.State := Text;
end Clear;
----------------------------------------------------------------------------
-- This procedure will be called to perform further analysis on a token
-- based on the given next character.
----------------------------------------------------------------------------
procedure Analyze (The_Token: in out Instance;
Next_Char: in Character;
Verdict : out Analysis_Verdict) is
begin
case The_Token.State is
when Text =>
if Ada.Strings.Maps.Is_In (Element => Next_Char, Set => The_Token.Set)
then
Verdict := Matches;
else
Verdict := Failed;
The_Token.State := Done;
end if;
when Done =>
-- We shouldn't get called from here.
Verdict := Failed;
end case;
end Analyze;
----------------------------------------------------------------------------
-- This procedure will be called to create a character set token
----------------------------------------------------------------------------
function Get (Set : in Ada.Strings.Maps.Character_Set;
Reportable: in Boolean := False) return Instance is
begin
return (Report => Reportable,
State => Text,
Set => Set);
end Get;
end Opentoken.Recognizer.Character_Set;
-------------------------------------------------------------------------------
--
-- Copyright (C) 1999, 2008 Christoph Karl Walter Grein
--
-- This file is part of the OpenToken package.
--
-- The OpenToken package is free software; you can redistribute it and/or
-- modify it under the terms of the GNU General Public License as published
-- by the Free Software Foundation; either version 2, or (at your option)
-- any later version. The OpenToken package is distributed in the hope that
-- it will be useful, but WITHOUT ANY WARRANTY; without even the implied
-- warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU General Public License for more details. You should have received
-- a copy of the GNU General Public License distributed with the OpenToken
-- package; see file GPL.txt. If not, write to the Free Software Foundation,
-- 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
--
-- As a special exception, if other files instantiate generics from this
-- unit, or you link this unit with other files to produce an executable,
-- this unit does not by itself cause the resulting executable to be
-- covered by the GNU General Public License. This exception does not
-- however invalidate any other reasons why the executable file might be
-- covered by the GNU Public License.
--
-- Maintainer: Christoph K. W. Grein ([EMAIL PROTECTED])
--
-- Update History:
-- $Log: opentoken-recognizer-graphic_character.adb,v $
-- Revision 1.2 1999/12/27 19:56:01 Ted
-- fix file contents to work w/ new hierarchy
--
-- Revision 1.1 1999/12/27 17:11:34 Ted
-- renamed everything to new hierarchy
--
-- Revision 1.2 1999/10/08 23:12:00 Ted
-- Add ability to exclude characters from the set
--
-- 1.1 - 4 July 1999 Exclusion set
-- 1.0 - 26 June 1999 First release
-------------------------------------------------------------------------------
with Ada.Characters.Handling;
-------------------------------------------------------------------------------
-- This package implements a token recognizer for a graphic character.
-------------------------------------------------------------------------------
package body Opentoken.Recognizer.Graphic_Character is
----------------------------------------------------------------------------
-- This procedure will be called when analysis on a new candidate string
-- is started. The Token needs to clear its state (if any).
----------------------------------------------------------------------------
procedure Clear (The_Token: in out Instance) is
begin
The_Token.State := Opening_Tick;
end Clear;
----------------------------------------------------------------------------
-- This procedure will be called to perform further analysis on a token
-- based on the given next character.
----------------------------------------------------------------------------
procedure Analyze (The_Token: in out Instance;
Next_Char: in Character;
Verdict : out Analysis_Verdict) is
begin
case The_Token.State is
when Opening_Tick =>
if Next_Char = ''' then -- '' (this comment works around an emacs
colorizing bug)
Verdict := So_Far_So_Good;
The_Token.State := The_Character;
else
Verdict := Failed;
The_Token.State := Done;
end if;
when The_Character =>
if Ada.Characters.Handling.Is_Graphic (Next_Char) and
not Ada.Strings.Maps.Is_In (Element => Next_Char, Set =>
The_Token.Excluded) then
Verdict := So_Far_So_Good;
The_Token.State := Closing_Tick;
else
Verdict := Failed;
The_Token.State := Done;
end if;
when Closing_Tick =>
if Next_Char = ''' then
Verdict := Matches;
The_Token.State := Done;
else
Verdict := Failed;
The_Token.State := Done;
end if;
when Done =>
Verdict := Failed;
end case;
end Analyze;
----------------------------------------------------------------------------
-- This procedure will be called to create a character set token.
----------------------------------------------------------------------------
function Get (Exclude: Ada.Strings.Maps.Character_Set :=
Ada.Strings.Maps.Null_Set)
return Instance is
begin
return (Report => True,
Excluded => Exclude,
State => Opening_Tick);
end Get;
procedure Redefine (Inst : in out Instance;
Exclude: in Ada.Strings.Maps.Character_Set) is
begin
Inst.Excluded := Exclude;
end Redefine;
end Opentoken.Recognizer.Graphic_Character;
-------------------------------------------------------------------------------
--
-- Copyright (C) 1999, 2008 Christoph Karl Walter Grein
--
-- This file is part of the OpenToken package.
--
-- The OpenToken package is free software; you can redistribute it and/or
-- modify it under the terms of the GNU General Public License as published
-- by the Free Software Foundation; either version 2, or (at your option)
-- any later version. The OpenToken package is distributed in the hope that
-- it will be useful, but WITHOUT ANY WARRANTY; without even the implied
-- warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU General Public License for more details. You should have received
-- a copy of the GNU General Public License distributed with the OpenToken
-- package; see file GPL.txt. If not, write to the Free Software Foundation,
-- 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
--
-- As a special exception, if other files instantiate generics from this
-- unit, or you link this unit with other files to produce an executable,
-- this unit does not by itself cause the resulting executable to be
-- covered by the GNU General Public License. This exception does not
-- however invalidate any other reasons why the executable file might be
-- covered by the GNU Public License.
--
-- Maintainer: Christoph K. W. Grein ([EMAIL PROTECTED])
--
-- Update History:
-- $Log: opentoken-recognizer-graphic_character.ads,v $
-- Revision 1.2 1999/12/27 19:56:01 Ted
-- fix file contents to work w/ new hierarchy
--
-- Revision 1.1 1999/12/27 17:11:35 Ted
-- renamed everything to new hierarchy
--
-- Revision 1.2 1999/10/08 23:12:00 Ted
-- Add ability to exclude characters from the set
--
-- 1.1 - 4 July 1999 Exclusion set
-- 1.0 - 26 June 1999 First release
-------------------------------------------------------------------------------
with Ada.Strings.Maps.Constants;
-------------------------------------------------------------------------------
-- This package implements a token recognizer for a character literal 'x',
-- where all graphic characters are allowed for x except those given in an
-- exclusion set.
-------------------------------------------------------------------------------
package Opentoken.Recognizer.Graphic_Character is
type Instance is new Opentoken.Recognizer.Instance with private;
----------------------------------------------------------------------------
-- Call this procedure to create a Graphic_Character token.
----------------------------------------------------------------------------
function Get (Exclude: Ada.Strings.Maps.Character_Set :=
Ada.Strings.Maps.Null_Set)
return Instance;
----------------------------------------------------------------------------
-- Call this procedure to redefine the exclusion set.
----------------------------------------------------------------------------
procedure Redefine (Inst : in out Instance;
Exclude: in Ada.Strings.Maps.Character_Set);
private
type State_ID is (Opening_Tick, The_Character, Closing_Tick, Done);
type Instance is new Opentoken.Recognizer.Instance with record
Excluded: Ada.Strings.Maps.Character_Set;
-- The finite state machine state
State: State_ID := Opening_Tick;
end record;
----------------------------------------------------------------------------
-- This procedure will be called when analysis on a new candidate string
-- is started. The Token needs to clear its state (if any).
----------------------------------------------------------------------------
procedure Clear (The_Token: in out Instance);
----------------------------------------------------------------------------
-- This procedure will be called to perform further analysis on a token
-- based on the given next character.
----------------------------------------------------------------------------
procedure Analyze (The_Token: in out Instance;
Next_Char: in Character;
Verdict : out Analysis_Verdict);
end Opentoken.Recognizer.Graphic_Character;
