[antlr-interest] Catch all characters token

[Pete Forman]

At 2004-07-07 20:01 +0100, Steve Jones wrote:
>The problem is complicated by the fact I need to return three tokens, eg:
>
>BEGIN
>SOME_TEXT
>END
>
>If I use greedy then then I will only produce one token that contains the
>terminals BEGIN and END as well as the interposing text.

It looks like one token's worth of info to me.

Some solutions to consider:

1) Return one token from lexer, parser puts three tokens into the AST.
    textBlock: SOME_TEXT {## = #([BEGIN, "Begin"], ##, [END, "End"]);};

2) One token from lexer, use a TokenStream to make the three tokens.
    http://www.antlr.org/doc/streams.html

3) Switch to a second lexer on BEGIN.
    That is probably too heavyweight for your simple SOME_TEXT.

-- 
Pete Forman                -./\.-  Disclaimer: This post is originated
WesternGeco                  -./\.-   by myself and does not represent
pete.forman at westerngeco.com    -./\.-   opinion of Schlumberger, Baker
http://petef.port5.com           -./\.-   Hughes or their divisions.



 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
    antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/