[antlr-interest] Token vs Rule

[Ric Klaren]

On Thu, Apr 01, 2004 at 08:15:11PM -0000, srandm wrote:
> What is the difference between putting the token in the tokens 
> section vs declaring it as a rule outside of the tokens section.  
> They seem to produce different result which I am not quite sure I 
> understand:

Stuff in the tokens section gets tested in rules that have the option
testLiterals turned on. A token defined in a rule will always be matched
and passed on the the parser (unless it's a protected rule)

Usually you turn of testLiterals globally in your lexer grammar then enable
it for a number of rules that match keywords or the likes (check for
instance the tincy example) here the ID rule from it:

ID
options {
        testLiterals = true;
        paraphrase = "an identifier";
}
        :       ('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'_'|'0'..'9')*
        ;

ID matches all stuff resembling an identifier, but before the ID token is
returned to the parser a hash table is checked that's generated from the
tokens section and if a match is found that token is returned.	 

Cheers,

Ric
-- 
-----+++++*****************************************************+++++++++-------
    ---- Ric Klaren ----- j.klaren at utwente.nl ----- +31 53 4893722  ----
-----+++++*****************************************************+++++++++-------
 Time what is time - I wish I knew how to tell You why - It hurts to know -
          Aren't we machines - Time what is time - Unlock the door
               - And see the truth - Then time is time again
                From: 'Time what is Time' by Blind Guardian



 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/