[antlr-interest] Controlling Lexer from Parser

David-Sarah Hopwood david-sarah at jacaranda.org
Thu Dec 3 10:12:06 PST 2009


Gokulakannan Somasundaram wrote:
> Hi,
>    I have seen some relevant articles in the FAQ. But i want to know,
> whether the following approach will always work for me.
> 
> I am trying to parse a SQL grammar, in which the SQL Keywords are sometime
> allowed as table names / column names.
> a)  Say when i am expecting a table_name /column name from parser, i set a
> global variable called x.
> b) i check this x to set the token type of that particular token.
> 
> This will succeed only if the parser completes executing the parsing actions
> before trying to make tokens out of the inputstream. Is it always the case
> with ANTLR?

No; unless you use a different TokenStream class, the whole stream will be
lexed before any token is parsed.

In any case, lexer->parser feedback is almost certainly not the best way to
solve your problem above. (Lexer->parser feedback is horribly complicated,
in any parser generator but especially in ANTLR, and should be used only as
a last resort in my experience.)

To solve the problem that keywords can sometimes be used as identifiers,
define a parser rule 'keyword' whose alternatives are all of the keywords,
and then define

  id : ID | keyword ;

This might introduce some ambiguities depending on the language you're
parsing, but you can usually solve those using predicates.

If you want to change the token type of a keyword to ID when it is used
as an identifier, then define

  id : ID
     | k=keyword { if ($k.tree != null) $k.tree.getToken().setType(ID); }
     ;

-- 
David-Sarah Hopwood  ⚥  http://davidsarah.livejournal.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 292 bytes
Desc: OpenPGP digital signature
Url : http://www.antlr.org/pipermail/antlr-interest/attachments/20091203/cc2112ba/attachment.bin 


More information about the antlr-interest mailing list