[antlr-interest] Getting token extents for grammar rules

[Ric Klaren]

Hi,

On Tue, May 11, 2004 at 10:22:41AM -0700, Paul J. Lucas wrote:
> 	For better or worse, suppose I'm building my own parse-tree
> 	data structure where every grammar rule is of the form:
> 
> 		someRule
> 		{
> 			enterRule( SOME_RULE_ID );
> 		}
> 			: /* call of subrules, tokens, etc. */
> 				{
> 					leaveRule( SOME_RULE_ID );
> 				}
> 			;
> 
> 	and each lexer rule is of the form:
> 
> 		SOME_TOKEN
> 			: 'xxxx'
> 				{
> 					giveToken( SOME_TOKEN );
> 				}
> 			;
> 
> 	where giveToken() gives the current token to a class that
> 	accumulates all tokens parsed.  The enterRule() and leaveRule()
> 	methods carve up the sequence of tokens such that each rule has
> 	the extent of tokens comprising it, i.e., token[i]...token[j].
> 	This works fine... mostly.

This is feedback from the parser to the lexer in general this does not
work. Unless enterRule/leaveRule keep into account the parser entering
guessing mode and the odd extra character lookahead at times. 

You could tag all the tokens with unique ID's and have the
enterRule/leaveRule's mark the exact token that is 'seen'/tagged in the
rule in your token collector (you could use LA(1) for this at the spots
where you call leave/enterRule).

Cheers,

Ric
-- 
-----+++++*****************************************************+++++++++-------
    ---- Ric Klaren ----- j.klaren at utwente.nl ----- +31 53 4893755  ----
-----+++++*****************************************************+++++++++-------
  Quidquid latine dictum sit, altum viditur.
                 (Whatever is said in Latin sounds profound.)



 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/