[antlr-interest] Lexer rule for strings with quoted strings within

Ron Hunter-Duvar ron.hunter-duvar at oracle.com
Thu Mar 25 08:43:22 PDT 2010


It's funny that you say that backtracking will not work in lexers. I 
don't know if it would help in the particular case the OP asked about, 
but I've found that it not only works, it's the only way to get Antlr 
lexers to work the way lexers are generally expected to work based on 
experience with FSA-based lexers. With backtracking turned on, rules 
such as for floating point and integer tokens can be kept separate (as 
long as they're in the right order). Without backtracking the first 
possible match is the only one that will ever be tried (if the float 
rule is first, only floats can be lexed, never integers; if the integer 
rule is first, floats can never be lexed), so it becomes necessary to 
merge all such rules into one big ugly rule with lots of predicates and 
set type calls.

Ron


Jim Idle wrote:
> 1) Backtracking will not work in lexers
> 2) Don't advise people to use backtrack mode
> 3) That rule will give warnings anyway
>
>
> BTW Guys, don't forget antlr.markmail.org !!!
>
> Jim
>
>   
>> -----Original Message-----
>> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
>> bounces at antlr.org] On Behalf Of Bart Kiers
>> Sent: Thursday, March 25, 2010 6:51 AM
>> To: Anders Sollander
>> Cc: antlr-interest at antlr.org
>> Subject: Re: [antlr-interest] Lexer rule for strings with quoted
>> strings within
>>
>> On Thu, Mar 25, 2010 at 2:01 PM, Anders Sollander <
>> Anders.Sollander at mathworks.de> wrote:
>>
>>     
>>> Hi,
>>>
>>> I've been trying to write a lexer rule for strings with strings in
>>>       
>> them,
>>     
>>> like
>>>
>>>        "This has a ""quoted string"" within"
>>>
>>> Is there a simple lexer rule for this, or do I need some kind of look
>>> ahead?
>>>
>>>
>>>       
>> You could enable the `backtrack` option:
>>
>> String
>> options { backtrack = true; }
>>   :  '"' ('""' | ~'"')+ '"'
>>   ;
>>
>> Regards,
>>
>> Bart.
>>
>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
>> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
>> email-address
>>     
>
>
>
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>
>   

-- 
Ron Hunter-Duvar | Software Developer V | 403-272-6580
Oracle Service Engineering
Gulf Canada Square 401 - 9th Avenue S.W., Calgary, AB, Canada T2P 3C5

All opinions expressed here are mine, and do not necessarily represent
those of my employer.



More information about the antlr-interest mailing list