[antlr-interest] token context information

tcurdt tcurdt at yahoo.com
Wed Jun 16 02:46:36 PDT 2004


This is maybe a pretty general question but...

I've run into the following problem. I have
a file with key value pairs like this:

 key1 : value1
 key2 : value2


Now what I'd like have is a token stream
of

 KEY COLON VALUE
 KEY COLON VALUE

but the KEY and the VALUE tokens are pretty
much the same. Except that the KEY token is
more restricted on particular chars. So the
only way I got this working was to have it

 LITERAL COLON LITERAL
 LITERAL COLON LITERAL

Which is bad and gives me some headaches due
to the character restrictions in the key.

If the lexer would have the context information
as the parser has he would "know" that after
COLON there is supposed to be a VALUE and only
needs to test that. But I don't know how to do
that with a grammar.

I also tried with fixed KEY tokens. So all
KEYs are known by the grammar. But as soon
as I write:

key1 : key1

The lexer recognizes the KEY1 token twice

 KEY1 COLON KEY1

But I still wanna have

 KEY1 COLON VALUE

What am I missing here?
Any pointers appreciated!

TIA
--
Torsten




 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/
 



More information about the antlr-interest mailing list