[antlr-interest] Token vs Rule

[Mark Buckle]

Erm, Is there anyway for the parser to communicate with the lexer to pass
calling context down to this rule which will enable
run-time setting of the testLiterals=true option ?
Mark

-----Original Message-----
From: Ric Klaren [mailto:klaren at cs.utwente.nl] 
Sent: 02 April 2004 10:37
To: antlr-interest at yahoogroups.com
Subject: Re: [antlr-interest] Token vs Rule


On Thu, Apr 01, 2004 at 08:15:11PM -0000, srandm wrote:
> What is the difference between putting the token in the tokens
> section vs declaring it as a rule outside of the tokens section.  
> They seem to produce different result which I am not quite sure I 
> understand:

Stuff in the tokens section gets tested in rules that have the option
testLiterals turned on. A token defined in a rule will always be matched and
passed on the the parser (unless it's a protected rule)

Usually you turn of testLiterals globally in your lexer grammar then enable
it for a number of rules that match keywords or the likes (check for
instance the tincy example) here the ID rule from it:

ID
options {
        testLiterals = true;
        paraphrase = "an identifier";
}
        :       ('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'_'|'0'..'9')*
        ;

ID matches all stuff resembling an identifier, but before the ID token is
returned to the parser a hash table is checked that's generated from the
tokens section and if a match is found that token is returned.	 

Cheers,

Ric
-- 
-----+++++*****************************************************+++++++++
-----+++++-------
    ---- Ric Klaren ----- j.klaren at utwente.nl ----- +31 53 4893722  ----
-----+++++*****************************************************+++++++++
-----+++++-------
 Time what is time - I wish I knew how to tell You why - It hurts to know -
          Aren't we machines - Time what is time - Unlock the door
               - And see the truth - Then time is time again
                From: 'Time what is Time' by Blind Guardian



 
Yahoo! Groups Links



 

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

This e-mail may contain information that is privileged, confidential or
otherwise protected from disclosure. It must not be used by, or its 
contents copied or disclosed to, persons other than the intended 
recipient. However, the contents of this e-mail may be intercepted, 
monitored or recorded by Insurance Technology Solutions Limited for 
the purposes of ensuring compliance with its policies and procedures. 
Any liability (in negligence or otherwise) arising from any third party
acting, or refraining from acting, on any information contained in this
e-mail is excluded.

Any views expressed in this message are those of the individual
sender and do not necessarily represent the views of
Insurance Technology Solutions Limited.

If you have received this e-mail in error please notify the
originator of the message.

Scanning of this message and addition of this footer is performed
by SurfControl E-mail Filter software in conjunction with 
virus detection software.



 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/