[antlr-interest] token context information
Monty Zukowski
monty at codetransform.com
Wed Jun 16 09:28:07 PDT 2004
Look at "lexical states" in the documentation.
http://www.antlr.org/doc/lexer.html#Lexical_States
Monty
On Jun 16, 2004, at 2:46 AM, tcurdt wrote:
> This is maybe a pretty general question but...
>
> I've run into the following problem. I have
> a file with key value pairs like this:
>
> key1 : value1
> key2 : value2
>
>
> Now what I'd like have is a token stream
> of
>
> KEY COLON VALUE
> KEY COLON VALUE
>
> but the KEY and the VALUE tokens are pretty
> much the same. Except that the KEY token is
> more restricted on particular chars. So the
> only way I got this working was to have it
>
> LITERAL COLON LITERAL
> LITERAL COLON LITERAL
>
> Which is bad and gives me some headaches due
> to the character restrictions in the key.
>
> If the lexer would have the context information
> as the parser has he would "know" that after
> COLON there is supposed to be a VALUE and only
> needs to test that. But I don't know how to do
> that with a grammar.
>
> I also tried with fixed KEY tokens. So all
> KEYs are known by the grammar. But as soon
> as I write:
>
> key1 : key1
>
> The lexer recognizes the KEY1 token twice
>
> KEY1 COLON KEY1
>
> But I still wanna have
>
> KEY1 COLON VALUE
>
> What am I missing here?
> Any pointers appreciated!
>
> TIA
> --
> Torsten
>
>
>
>
>
> Yahoo! Groups Links
>
>
>
>
>
>
>
>
Monty Zukowski
ANTLR & Java Consultant -- http://www.codetransform.com
ANSI C/GCC transformation toolkit --
http://www.codetransform.com/gcc.html
Embrace the Decay -- http://www.codetransform.com/EmbraceDecay.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/antlr-interest/
<*> To unsubscribe from this group, send an email to:
antlr-interest-unsubscribe at yahoogroups.com
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
More information about the antlr-interest
mailing list