[antlr-interest] Tokenising for context specific reserved words

Loring Craymer lgcraymer at yahoo.com
Fri Jul 18 14:22:16 PDT 2008


Cute--I found it on citeseer.  They are solving a slightly different problem--multi-valued types, not just binary alternatives--and having the parser do a table lookup.  I think that the binary approach is probably sufficient--either you want the generic answer, or you are in a context where you want the specific answer (a keyword--possibly one of a set of possible keywords)--and can be fitted into ANTLR without having to go to a GLR parser.
--Loring



----- Original Message ----
> From: Terence Parr <parrt at cs.usfca.edu>
> To: Loring Craymer <lgcraymer at yahoo.com>
> Cc: Johannes Luber <jaluber at gmx.de>; Jim Idle <jimi at temporal-wave.com>; antlr-interest <antlr-interest at antlr.org>
> Sent: Friday, July 18, 2008 1:58:00 PM
> Subject: Re: [antlr-interest] Tokenising for context specific reserved words
> 
> 
> On Jul 18, 2008, at 1:59 AM, Loring Craymer wrote:
> 
> > Actually, I just thought of a hybrid approach:  support subtypes for  
> > double-quoted literals and have the subtype field set by the lexer.  
> > Then, a doubly quoted literal reference in a grammar is handled by  
> > looking at the subtype field, while an ID reference looks at the  
> > type field.  Except for the extra field (which probably costs  
> > nothing--minimum allocation sizes are usually larger than tokens),  
> > the only added runtime overhead would be in the lexer to set the  
> > subtype field.
> 
> There was an interesting paper called "Schrodinger's tokens"... if you  
> are a physics or quantum physics buff, you get the reference ;)
> 
> Ter



      



More information about the antlr-interest mailing list