[antlr-interest] Non-reserved keywords (again)

Martin Probst mail at martin-probst.com
Thu Aug 11 07:15:13 PDT 2005


Hi,

> I have search long and far to see if anyone had had the same problem I'm
> having with a language that does not reserve keywords, like SQL and in
> my case Fortran. I have found many people with the problem, yet no
> useful solutions. 

One solution is usually to use a stateful lexer, e.g. if you have a
keyword called "if" that may also be an identifier you have to keep
track in the lexer if your in a place where an identifier or where a
keyword should be lexed. Every lexed token then moves the lexer into
another state. That can be quite cumbersome (the XQuery spec for example
provides a lexer automaton that has over 16 states, most of them with
well over 10 different actions), but usually leads to something usable.

The second advise you found probably works too, but you might need to
use a lot of syntatic predicates.

Martin



More information about the antlr-interest mailing list