[antlr-interest] [Antlr 3] lexer problem

[Sven Efftinge]

Yes, I know. But I'd expect that the lexer tracks back when it can  
not complete the optional ('.' DIGIT)? part.
So it just consumes 42 (because it is a valid FLOAT, too).
The parser behaves like this, but the lexer not.
I'd expected that the following two grammars would successfully parse  
'42.foo'

// float as lexer rule
start : FLOAT ('.' foo)?;
FLOAT : DIGIT ('.' DIGIT)?;
DIGIT : '0'..'9'+;

// float as parser rule
start : float ('.' foo)?;
float : DIGIT ('.' DIGIT)?;
DIGIT : '0'..'9'+;

Only the second one works...

On May 15, 2007, at 10:45 , Alex Shneyderman wrote:

> On 5/15/07, Sven Efftinge <sven at efftinge.de> wrote:
>>
>> Hi,
>>
>>
>> I faced the following behaviour.
>> For the grammar
>>
>> start  : FLOAT ('.' 'foo')?;
>
> this is the parser rule. so it happens after lexer produced the tokens
>
>>
>> FLOAT  : '0'..'9'+ ('.' '0'..'9'+)?;
>
> now this is your lexer rule. Lexer gets to scan your char input first.
> So it scans 47 then sees '.' it assumes '0'..'9'+ after the dot should
> come to produce the FLOAT but instead gets 'foo'. So you get an
> exception. When you provide 47.42.foo as inpit everythings lexer cares
> about gets matched to produce FLOAT, so no problems in this input.

--

Sven Efftinge

Am Sophienhof 33
24941 Flensburg

mail : sven at efftinge.de
fon   : +49 176 21769757
web : http://www.efftinge.de
blog : http://effi-blog.blogspot.com