[antlr-interest] token context information

[Pete Forman]

At 2004-06-18 04:36 -0700, Mr Torsten Curdt wrote:
>It's all about: I have two tokens, both are almost
>the same. The lexer will give me a warning because
>they are ambiguous. Whether it is token A or token
>B depends on the previous token.

Generally lexers should be concentrating on converting sequences of
characters into tokens, with little or no account of context.  A
question to ask is whether inspecting a LITERAL itself can always
classify it as a KEY or VALUE.  If not then it may be better for the
lexer to only return LITERALs and to add actions like setType(KEY)
and setType(VALUE) in the parser rules.  The parser should run checks
for valid keys and values over and above the constraints of literal.

The alternative is for the lexer to do some of the parsing.  The rule
for LITERAL would look ahead for a colon (and perhaps value after
that) to see whether to setType(KEY).  The rule for COLON would set a
state to indicate that the next LITERAL should setType(VALUE).  This
might improve efficiency at the expense of muddier code.

A question to guide your choice is what happens when you read

   foo : bar

where foo is okay as a literal but fails to meet the rules for a key.
Can that be interpreted as some other grammatical construct or is that
an error that should issue a diagnostic and bail out?

-- 
Pete Forman                -./\.-  Disclaimer: This post is originated
WesternGeco                  -./\.-   by myself and does not represent
pete.forman at westerngeco.com    -./\.-   opinion of Schlumberger, Baker
http://petef.port5.com           -./\.-   Hughes or their divisions.



 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/