[antlr-interest] Lexer question (Update on Input from Dominik)

Jiho Han jhan at InfinityInfo.com
Thu Jul 27 07:45:50 PDT 2006


Perhaps, I don't understand what tokens section does.

Is

tokens
{
    AND = "and";
}

different from,

AND: "and";

?  And what is an imaginary token?  I read the meta language document
but I can't seem to grasp them.

And what does,

tokens
{
    INTEGER :;
}

really mean?  There is the token defined but it doesn't represent
anything?

-----Original Message-----
From: antlr-interest-bounces at antlr.org
[mailto:antlr-interest-bounces at antlr.org] On Behalf Of Micheal J
Sent: Thursday, July 27, 2006 10:54 AM
To: antlr-interest at antlr.org
Subject: RE: [antlr-interest] Lexer question (Update on Input from
Dominik)

> I was looking at ms sql grammar by Tomasz Jastrzebski (on
> antlr.org) and this is what I found:
> 
> protected
> Integer :;
> 
> protected
> Real :;
> 
> Number
>     :
>       ( (Digit)+ ('.' | 'e') ) => (Digit)+ ( '.' (Digit)* (Exponent)? 
> |
> Exponent) { _ttype = Real; }
>     | '.' { _ttype = DOT; } ( (Digit)+ (Exponent)? { _ttype = Real; } 
> )?
>     | (Digit)+ { _ttype = Integer; }
>     | "0x" ('a'..'f' | Digit)* { _ttype = HexLiteral; } // "0x" is 
> valid hex literal
>     ;
> 
> k is set to 2.
> 
> I haven't tested it so I don't know.
> Furthermore, since I'm a newbie, I am not even sure what the two 
> Integer and Real rule is supposed to do either.  What does it do?  
> It's empty?
> 
> Jiho

Interger/Real:
--------------

I'm guessing that the author is trying to define those token types. He
should have used a tokens block e.g.
	tokens
	{
		INTEGER;
		REAL;
		......
	}


Examples of lexing numbers:
---------------------------

The KCSParse/csharp_v1 grammar has a more extensive set of rules for
dealing with numeric tokens.

Find it in the examples/csharp directory of a 2.7.6 distro or on the web
site in file sharing.


Micheal

-----------------------
The best way to contact me is via the list/forum. My time is very
limited.



More information about the antlr-interest mailing list