[antlr-interest] token context information

Monty Zukowski monty at codetransform.com
Wed Jun 16 09:28:07 PDT 2004


Look at "lexical states" in the documentation.  
http://www.antlr.org/doc/lexer.html#Lexical_States

Monty

On Jun 16, 2004, at 2:46 AM, tcurdt wrote:

> This is maybe a pretty general question but...
>
> I've run into the following problem. I have
> a file with key value pairs like this:
>
>  key1 : value1
>  key2 : value2
>
>
> Now what I'd like have is a token stream
> of
>
>  KEY COLON VALUE
>  KEY COLON VALUE
>
> but the KEY and the VALUE tokens are pretty
> much the same. Except that the KEY token is
> more restricted on particular chars. So the
> only way I got this working was to have it
>
>  LITERAL COLON LITERAL
>  LITERAL COLON LITERAL
>
> Which is bad and gives me some headaches due
> to the character restrictions in the key.
>
> If the lexer would have the context information
> as the parser has he would "know" that after
> COLON there is supposed to be a VALUE and only
> needs to test that. But I don't know how to do
> that with a grammar.
>
> I also tried with fixed KEY tokens. So all
> KEYs are known by the grammar. But as soon
> as I write:
>
> key1 : key1
>
> The lexer recognizes the KEY1 token twice
>
>  KEY1 COLON KEY1
>
> But I still wanna have
>
>  KEY1 COLON VALUE
>
> What am I missing here?
> Any pointers appreciated!
>
> TIA
> --
> Torsten
>
>
>
>
>
> Yahoo! Groups Links
>
>
>
>
>
>
>
>
Monty Zukowski

ANTLR & Java Consultant -- http://www.codetransform.com
ANSI C/GCC transformation toolkit -- 
http://www.codetransform.com/gcc.html
Embrace the Decay -- http://www.codetransform.com/EmbraceDecay.html



 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/
 



More information about the antlr-interest mailing list