[antlr-interest] A proposal for keywords

[Loring Craymer]

Handling keywords in grammars is an awkward problem.  Languages handle
keywords in one of two ways:  1.)  keywords are uniquely named, or 2.)
keyword names can be used for other language elements (variable names,
etc.).  ANTLR 2 preferentially supports option 1; PCCTS allowed either, but
directly supported neither (option 1 could be supported by adding symbol
table lookup in lexer actions; option 2 could be supported by predicate
hoisting).

 

It strikes me that the difference is solely a matter of when types are bound
to tokens.  For option 1, types are bound to tokens in the lexer.  For
option 2, literal types should be bound in the parser:  that is, the lexer
binds the generic type to the token (TEXT or NUMBER, for example) and the
dynamically looks up the next token in the literals table whenever
attempting to match a literal.  That is, "if" would be first typed as TEXT
but matched (and retyped) as LITERAL_if when matching an occurrence of "if"
in the parser.

 

I was concerned that this might not work with the LL(*) DFAs of ANTLR 3,
until I realized that the predicate hoisting mechanism provides almost all
of the support needed.  (Some sort of type patching table may also be
required; "if" might be matched by a state that allowed either LITERAL_if or
TEXT as the type for that token; for a first implementation, the type
patching may not be necessary since tree walkers could also do dynamic
lookup when matching literals.  Patching, though, seems preferable over the
longer term.)

 

Comments, anyone?  As far as I can see, the only downside to providing an
option to select a keyword mechanism is that we will need to find a
replacement topic for ANTLR workshop discussions-I can remember being in
discussions on this topic at every one of the past workshops!

 

--Loring

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20060523/4d1e656d/attachment.html