[antlr-interest] Lexer question (Update on Input from Dominik)
Jiho Han
jhan at InfinityInfo.com
Thu Jul 27 07:45:50 PDT 2006
Perhaps, I don't understand what tokens section does.
Is
tokens
{
AND = "and";
}
different from,
AND: "and";
? And what is an imaginary token? I read the meta language document
but I can't seem to grasp them.
And what does,
tokens
{
INTEGER :;
}
really mean? There is the token defined but it doesn't represent
anything?
-----Original Message-----
From: antlr-interest-bounces at antlr.org
[mailto:antlr-interest-bounces at antlr.org] On Behalf Of Micheal J
Sent: Thursday, July 27, 2006 10:54 AM
To: antlr-interest at antlr.org
Subject: RE: [antlr-interest] Lexer question (Update on Input from
Dominik)
> I was looking at ms sql grammar by Tomasz Jastrzebski (on
> antlr.org) and this is what I found:
>
> protected
> Integer :;
>
> protected
> Real :;
>
> Number
> :
> ( (Digit)+ ('.' | 'e') ) => (Digit)+ ( '.' (Digit)* (Exponent)?
> |
> Exponent) { _ttype = Real; }
> | '.' { _ttype = DOT; } ( (Digit)+ (Exponent)? { _ttype = Real; }
> )?
> | (Digit)+ { _ttype = Integer; }
> | "0x" ('a'..'f' | Digit)* { _ttype = HexLiteral; } // "0x" is
> valid hex literal
> ;
>
> k is set to 2.
>
> I haven't tested it so I don't know.
> Furthermore, since I'm a newbie, I am not even sure what the two
> Integer and Real rule is supposed to do either. What does it do?
> It's empty?
>
> Jiho
Interger/Real:
--------------
I'm guessing that the author is trying to define those token types. He
should have used a tokens block e.g.
tokens
{
INTEGER;
REAL;
......
}
Examples of lexing numbers:
---------------------------
The KCSParse/csharp_v1 grammar has a more extensive set of rules for
dealing with numeric tokens.
Find it in the examples/csharp directory of a 2.7.6 distro or on the web
site in file sharing.
Micheal
-----------------------
The best way to contact me is via the list/forum. My time is very
limited.
More information about the antlr-interest
mailing list