[antlr-interest] Reuse of same token in multiple lexer modes

Terence Parr parrt at cs.usfca.edu
Tue Oct 9 10:23:52 PDT 2012


hi. Since the parser needs to have a unique token type, ANTLR does not allow you to redefine token roles. You can of course use a lexer command to change the token type of something after you match it with -> type(OPEN_PAREN).

 I should also note that it sounds like what you really want is a recursive lexer rule, given that you are doing a push in the lexer mode as well.
Ter
On Oct 9, 2012, at 5:30 AM, Kai Burjack (HBT) wrote:

> Hello Terrence,
> 
> first of all, many thanks for ANTLR and ANTLRv4 in particular, with its (among other things) greatly improved error reporting!
> 
> I have a question about lexer modes. I want to write an "island grammar", as it is called in your ANTLR4 beta2 book, and have found that it does not seem to be possible to reference the same token rule in multiple lexer modes.
> 
> Simplified example grammar:
> 
> lexer grammar MyLexer;
> 
> OPEN_PAREN : '(' -> pushMode(PAR) ;
> 
> mode PAR ;
> 
> OPEN_PAREN : '(' -> pushMode(PAR) ;
> CLOSE_PAREN : ')' -> popMode ;
> ...other tokens that are otherwise not allowed outside of parenthese...
> 
> What I am trying to do there is to "know" when I am in parenthesis in order to allow more tokens (such as '>' for freemarker template language style) that were otherwise not allowed outside of that mode.
> 
> The "Tool" runs through this grammar fine, but the generated Java code contains errors due to non-existing identifier "PAR".
> 
> Can you help me on this one, please?
> 
> Thanks.
> 
> Best Regards,
> Kai
> 
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address



More information about the antlr-interest mailing list