[antlr-interest] Controlling Lexer from Parser

[Johannes Luber]

> At 23:07 3/12/2009, Gokulakannan Somasundaram wrote:
> >I am trying to parse a SQL grammar, in which the SQL Keywords are 
> >sometime allowed as table names / column names.
> >a)  Say when i am expecting a table_name /column name from 
> >parser, i set a global variable called x.
> >b) i check this x to set the token type of that particular token.
> >
> >This will succeed only if the parser completes executing the 
> >parsing actions before trying to make tokens out of the 
> >inputstream. Is it always the case with ANTLR? I see no reason 
> >why this should not work, but i want to make sure. (The Lexer 
> >and  Parser are in different grammar files)
> 
> No.  In fact it is never the case with ANTLR -- the lexer runs to 
> completion and generates the entire token stream before any parser 
> rules are executed.

That is no longer completely true. Ter changed the default token stream to be unbuffered, so lexing happens always when calling NextToken(). That being said, without special care the lexer still won't change its lexing strategy without extra input from the parser.

Johannes
> 
> There are two common strategies for doing what you're trying to do 
> (both documented in the wiki):
>    1. Lex any letter sequence as an ID, then use semantic 
> predicates to treat specific IDs as keywords in the parser if they 
> have the right contents.
>    2. Lex keywords as individual keyword tokens (eg. SELECT) and 
> anything else as an ID, then define a parser rule "id" that 
> accepts ID or any of the keywords, and use that in any context you 
> want an identifier.
> 
> (The second is my preferred method, but either one will work.)
> 
> In either case, you can either leave the token type alone (eg. if 
> directly executing actions in the parser) or change the type as 
> needed (eg. if building an AST for later processing).
> 
> 
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe:
> http://www.antlr.org/mailman/options/antlr-interest/your-email-address

-- 
Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 -
sicherer, schneller und einfacher! http://portal.gmx.net/de/go/atbrowser