[antlr-interest] Lexer rule for strings with quoted strings within
Ron Hunter-Duvar
ron.hunter-duvar at oracle.com
Thu Mar 25 08:43:22 PDT 2010
It's funny that you say that backtracking will not work in lexers. I
don't know if it would help in the particular case the OP asked about,
but I've found that it not only works, it's the only way to get Antlr
lexers to work the way lexers are generally expected to work based on
experience with FSA-based lexers. With backtracking turned on, rules
such as for floating point and integer tokens can be kept separate (as
long as they're in the right order). Without backtracking the first
possible match is the only one that will ever be tried (if the float
rule is first, only floats can be lexed, never integers; if the integer
rule is first, floats can never be lexed), so it becomes necessary to
merge all such rules into one big ugly rule with lots of predicates and
set type calls.
Ron
Jim Idle wrote:
> 1) Backtracking will not work in lexers
> 2) Don't advise people to use backtrack mode
> 3) That rule will give warnings anyway
>
>
> BTW Guys, don't forget antlr.markmail.org !!!
>
> Jim
>
>
>> -----Original Message-----
>> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
>> bounces at antlr.org] On Behalf Of Bart Kiers
>> Sent: Thursday, March 25, 2010 6:51 AM
>> To: Anders Sollander
>> Cc: antlr-interest at antlr.org
>> Subject: Re: [antlr-interest] Lexer rule for strings with quoted
>> strings within
>>
>> On Thu, Mar 25, 2010 at 2:01 PM, Anders Sollander <
>> Anders.Sollander at mathworks.de> wrote:
>>
>>
>>> Hi,
>>>
>>> I've been trying to write a lexer rule for strings with strings in
>>>
>> them,
>>
>>> like
>>>
>>> "This has a ""quoted string"" within"
>>>
>>> Is there a simple lexer rule for this, or do I need some kind of look
>>> ahead?
>>>
>>>
>>>
>> You could enable the `backtrack` option:
>>
>> String
>> options { backtrack = true; }
>> : '"' ('""' | ~'"')+ '"'
>> ;
>>
>> Regards,
>>
>> Bart.
>>
>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
>> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
>> email-address
>>
>
>
>
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>
>
--
Ron Hunter-Duvar | Software Developer V | 403-272-6580
Oracle Service Engineering
Gulf Canada Square 401 - 9th Avenue S.W., Calgary, AB, Canada T2P 3C5
All opinions expressed here are mine, and do not necessarily represent
those of my employer.
More information about the antlr-interest
mailing list