[antlr-interest] Tokenising for context specific reserved words

Terence Parr parrt at cs.usfca.edu
Fri Jul 18 13:58:00 PDT 2008


On Jul 18, 2008, at 1:59 AM, Loring Craymer wrote:

> Actually, I just thought of a hybrid approach:  support subtypes for  
> double-quoted literals and have the subtype field set by the lexer.   
> Then, a doubly quoted literal reference in a grammar is handled by  
> looking at the subtype field, while an ID reference looks at the  
> type field.  Except for the extra field (which probably costs  
> nothing--minimum allocation sizes are usually larger than tokens),  
> the only added runtime overhead would be in the lexer to set the  
> subtype field.

There was an interesting paper called "Schrodinger's tokens"... if you  
are a physics or quantum physics buff, you get the reference ;)

Ter


More information about the antlr-interest mailing list