[antlr-interest] Catch all characters token
Monty Zukowski
monty at codetransform.com
Thu Jul 8 09:34:05 PDT 2004
On Jul 8, 2004, at 1:38 AM, Pete Forman wrote:
> At 2004-07-07 20:01 +0100, Steve Jones wrote:
>> The problem is complicated by the fact I need to return three tokens,
>> eg:
>>
>> BEGIN
>> SOME_TEXT
>> END
>>
>> If I use greedy then then I will only produce one token that contains
>> the
>> terminals BEGIN and END as well as the interposing text.
>
> It looks like one token's worth of info to me.
>
> Some solutions to consider:
>
> 1) Return one token from lexer, parser puts three tokens into the AST.
> textBlock: SOME_TEXT {## = #([BEGIN, "Begin"], ##, [END, "End"]);};
>
> 2) One token from lexer, use a TokenStream to make the three tokens.
> http://www.antlr.org/doc/streams.html
>
> 3) Switch to a second lexer on BEGIN.
> That is probably too heavyweight for your simple SOME_TEXT.
>
Nice summary, Pete.
Monty
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/antlr-interest/
<*> To unsubscribe from this group, send an email to:
antlr-interest-unsubscribe at yahoogroups.com
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
More information about the antlr-interest
mailing list