[antlr-interest] token context information

Mr Torsten Curdt tcurdt at yahoo.com
Fri Jun 18 07:59:19 PDT 2004


> >It's all about: I have two tokens, both are almost
> >the same. The lexer will give me a warning because
> >they are ambiguous. Whether it is token A or token
> >B depends on the previous token.
> 
> Generally lexers should be concentrating on
> converting sequences of
> characters into tokens, with little or no account of
> context.  A
> question to ask is whether inspecting a LITERAL
> itself can always
> classify it as a KEY or VALUE.

Not from their content itself. Only through
their context/position ...that's probably
the reason why the token definition might
not be right. Maybe I need to have more fine
grained definition. Maybe I need to split up
a general LITERAL and provide WORD and NUMBER
tokens. Not sure. ...it just feels a bit to
overcomplicated for a simple

 key : value

matching where I want to have some restrictions
on the key syntax and almost free form values.

> If not then it may
> be better for the
> lexer to only return LITERALs and to add actions
> like setType(KEY)
> and setType(VALUE) in the parser rules.  The parser
> should run checks
> for valid keys and values over and above the
> constraints of literal.

That's interesting! Can I still use the ANTLR
machinery to check whether the LITERAL is a KEY
or a VALUE?

> The alternative is for the lexer to do some of the
> parsing.

...which breaks SoC :-/

> The rule
> for LITERAL would look ahead for a colon (and
> perhaps value after
> that) to see whether to setType(KEY).  The rule for
> COLON would set a
> state to indicate that the next LITERAL should
> setType(VALUE).  This
> might improve efficiency at the expense of muddier
> code.

:-/

> A question to guide your choice is what happens when
> you read
> 
>    foo : bar
> 
> where foo is okay as a literal but fails to meet the
> rules for a key.
> Can that be interpreted as some other grammatical
> construct or is that
> an error that should issue a diagnostic and bail
> out?

Well, it should give me a VALUE COLON VALUE instead
of a KEY COLON VALUE and the parser should bail out.

Or what do you mean?

Thanks!
--
Torsten


		
__________________________________
Do you Yahoo!?
Read only the mail you want - Yahoo! Mail SpamGuard.
http://promotions.yahoo.com/new_mail 


 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/
 



More information about the antlr-interest mailing list