[antlr-interest] Re: keywords appended to identifiers, sugges tion s?

mzukowski at yci.com mzukowski at yci.com
Thu Jun 12 13:19:10 PDT 2003


Instead of halting you could make your lexer have a token buffer.  Halting
in ANTLR is not supported.  But you could modify your lexer to generate
multiple tokens, stuffing them into a buffer.  Then you have to manually
subclass your generated lexer and override nextToken() to first check the
buffer for a token to return.

Personally I like the tokenstream approach.  There's no reason why you can't
expose the keywords table through a lexer method and use that in your
stream.  Then you're not maintaining two tables.

Monty

-----Original Message-----
From: gdave [mailto:dave.green at valley.net] 
Sent: Thursday, June 12, 2003 12:58 PM
To: antlr-interest at yahoogroups.com
Subject: [antlr-interest] Re: keywords appended to identifiers, suggestion
s?


--- In antlr-interest at yahoogroups.com, "Jim O'Connor" 
<Jim.OConnor at m...> wrote:
> Hi Dave,
>   There is one solution.  There may be others.  A lexer is a
CharScanner.
> CharScanner has a method testLiteralsTable().
> 
> class L extends Lexer;
> 
......  rest of solution snipped......

I should have written in my first note that I was hoping to find a 
grammar based solution.  To get something running, I wrote a proxy 
TokenStream which I placed between the Lexer and the Parser which 
would notice trailing KEYWORDS appended to NAME tokens and split them 
into two tokens before sending them along.

This works, but leaves me maintaining a list of Keywords in a class 
separate from the rest of my grammar which feels awkward.  I was 
hoping that I was just missing some utility of the EBNF and predicate 
syntax that would let me lookahead to notice trailing KEYWORDS.

Actually I have tried experiments with Semantic predicates and 
although the "infinite" lookahead ability allows me to see a KEYWORD 
in the future, I have not found a production to place with the 
predicate that halts before consuming the KEYWORD characters.

anyone have any grammar ideas?
Dave


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 




More information about the antlr-interest mailing list