[antlr-interest] ANTLR 3 Lexical States

Jim Idle jimi at temporal-wave.com
Fri Jan 25 07:34:58 PST 2008


It will be a lot more readable, and generate better code if you do this 
instead:

1) Create fragment rules for your tokens that have the same pattern. In 
fact you can just use the tokens {} section to create the token types, 
but then ANTLR will give you warnings that there is no token called XYZ 
when you try to use this type in the lexer. As I hate warnings, I use 
fragment tokens. You won't use them for matching (usually) so they don't 
actually have to match the pattern they represent, but it is probably 
good to document this if they don't!!

2) Use one pattern match for all the tokens that clash, then change the 
type according to the context:


fragment STRING 		: LETTERS 	;
fragment SPECIAL_STRING : LETTERS	;

STRINGS:
		LETTERS

			{
				switch (state)
				{
					case States.NORMAL:

							$type = STRING;
							Break;
				
					case States.SPECIAL:
							$type = SPECIAL_STRING;
...
				}	
			}
;


And so on.

Jim

> -----Original Message-----
> From: Bertalan Fodor (LilyPondTool) [mailto:lilypondtool at organum.hu]
> Sent: Friday, January 25, 2008 2:21 AM
> To: Antlr Interest
> Subject: [antlr-interest] ANTLR 3 Lexical States
> 
> My Antlr grammar I'm migrating to Antlr 3 heavily uses lexical states,
> that is, the Lexer has lots of semantic predicates to distinguish
> between alternatives like this
> STRING: {inState(States.NORMAL)}? LETTER+
> SPECIAL_STRING: {inState(States.SPECIAL)}? LETTER+
> 
> The states are set during the parse process, like this
> special_handling: "\special" { setState(States.SPECIAL); }
> SPECIAL_STRING;
> 
> It worked perfectly well in Anltr 2. However, now I'm a bit afraid 
that
> the Antlr 3 style lexing will make this not work.
> 
> What do you think?
> 
> Thank you,
> 
> Bertalan Fodor
> 
> --
> LilyPondTool is the editor for LilyPond files.
> See http://lilypondtool.organum.hu
> 




More information about the antlr-interest mailing list