[antlr-interest] token context information
Mr Torsten Curdt
tcurdt at yahoo.com
Fri Jun 18 07:59:19 PDT 2004
> >It's all about: I have two tokens, both are almost
> >the same. The lexer will give me a warning because
> >they are ambiguous. Whether it is token A or token
> >B depends on the previous token.
>
> Generally lexers should be concentrating on
> converting sequences of
> characters into tokens, with little or no account of
> context. A
> question to ask is whether inspecting a LITERAL
> itself can always
> classify it as a KEY or VALUE.
Not from their content itself. Only through
their context/position ...that's probably
the reason why the token definition might
not be right. Maybe I need to have more fine
grained definition. Maybe I need to split up
a general LITERAL and provide WORD and NUMBER
tokens. Not sure. ...it just feels a bit to
overcomplicated for a simple
key : value
matching where I want to have some restrictions
on the key syntax and almost free form values.
> If not then it may
> be better for the
> lexer to only return LITERALs and to add actions
> like setType(KEY)
> and setType(VALUE) in the parser rules. The parser
> should run checks
> for valid keys and values over and above the
> constraints of literal.
That's interesting! Can I still use the ANTLR
machinery to check whether the LITERAL is a KEY
or a VALUE?
> The alternative is for the lexer to do some of the
> parsing.
...which breaks SoC :-/
> The rule
> for LITERAL would look ahead for a colon (and
> perhaps value after
> that) to see whether to setType(KEY). The rule for
> COLON would set a
> state to indicate that the next LITERAL should
> setType(VALUE). This
> might improve efficiency at the expense of muddier
> code.
:-/
> A question to guide your choice is what happens when
> you read
>
> foo : bar
>
> where foo is okay as a literal but fails to meet the
> rules for a key.
> Can that be interpreted as some other grammatical
> construct or is that
> an error that should issue a diagnostic and bail
> out?
Well, it should give me a VALUE COLON VALUE instead
of a KEY COLON VALUE and the parser should bail out.
Or what do you mean?
Thanks!
--
Torsten
__________________________________
Do you Yahoo!?
Read only the mail you want - Yahoo! Mail SpamGuard.
http://promotions.yahoo.com/new_mail
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/antlr-interest/
<*> To unsubscribe from this group, send an email to:
antlr-interest-unsubscribe at yahoogroups.com
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
More information about the antlr-interest
mailing list