[antlr-interest] Getting token extents for grammar rules
Ric Klaren
klaren at cs.utwente.nl
Thu May 13 02:59:46 PDT 2004
Hi,
On Tue, May 11, 2004 at 10:22:41AM -0700, Paul J. Lucas wrote:
> For better or worse, suppose I'm building my own parse-tree
> data structure where every grammar rule is of the form:
>
> someRule
> {
> enterRule( SOME_RULE_ID );
> }
> : /* call of subrules, tokens, etc. */
> {
> leaveRule( SOME_RULE_ID );
> }
> ;
>
> and each lexer rule is of the form:
>
> SOME_TOKEN
> : 'xxxx'
> {
> giveToken( SOME_TOKEN );
> }
> ;
>
> where giveToken() gives the current token to a class that
> accumulates all tokens parsed. The enterRule() and leaveRule()
> methods carve up the sequence of tokens such that each rule has
> the extent of tokens comprising it, i.e., token[i]...token[j].
> This works fine... mostly.
This is feedback from the parser to the lexer in general this does not
work. Unless enterRule/leaveRule keep into account the parser entering
guessing mode and the odd extra character lookahead at times.
You could tag all the tokens with unique ID's and have the
enterRule/leaveRule's mark the exact token that is 'seen'/tagged in the
rule in your token collector (you could use LA(1) for this at the spots
where you call leave/enterRule).
Cheers,
Ric
--
-----+++++*****************************************************+++++++++-------
---- Ric Klaren ----- j.klaren at utwente.nl ----- +31 53 4893755 ----
-----+++++*****************************************************+++++++++-------
Quidquid latine dictum sit, altum viditur.
(Whatever is said in Latin sounds profound.)
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/antlr-interest/
<*> To unsubscribe from this group, send an email to:
antlr-interest-unsubscribe at yahoogroups.com
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
More information about the antlr-interest
mailing list