[antlr-interest] Catch all characters token

[Monty Zukowski]

On Jul 8, 2004, at 1:38 AM, Pete Forman wrote:

> At 2004-07-07 20:01 +0100, Steve Jones wrote:
>> The problem is complicated by the fact I need to return three tokens, 
>> eg:
>>
>> BEGIN
>> SOME_TEXT
>> END
>>
>> If I use greedy then then I will only produce one token that contains 
>> the
>> terminals BEGIN and END as well as the interposing text.
>
> It looks like one token's worth of info to me.
>
> Some solutions to consider:
>
> 1) Return one token from lexer, parser puts three tokens into the AST.
>     textBlock: SOME_TEXT {## = #([BEGIN, "Begin"], ##, [END, "End"]);};
>
> 2) One token from lexer, use a TokenStream to make the three tokens.
>     http://www.antlr.org/doc/streams.html
>
> 3) Switch to a second lexer on BEGIN.
>     That is probably too heavyweight for your simple SOME_TEXT.
>

Nice summary, Pete.

Monty




 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
    antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/