[antlr-interest] Lexer bug? (with test cases!) [LAST MESSAGE]

[Terence Parr]

On Oct 24, 2007, at 7:43 PM, Austin Hastings wrote:

> Terence Parr wrote:
>>
>> It appears it's not backtracking because only 1 extra char is  
>> enough, but it cannot be statically determined if we assume any  
>> char can follow.  I think you could make the subrule more  
>> complicated and it would require 2 extra then 3 then 4 etc...
>
> I think that inserting a catch{} block for optional subrules would  
> go a long way toward eliminating this particular aspect of the  
> problem. When you don't match correctly, throw an exception, catch  
> it, and reset. The consumption of the input becomes temporary until  
> the token completes, but that's straightforward.

Lex backtracks just like that.  Can you people not see that as  
backtracking?  I'm dumbfounded at this point.

>>> This is because the clause ('.' DIGIT+)? must either match a dot  
>>> followed by at least one digit (minimum two characters) or match  
>>> nothing at all.  Seeing the dot and then assuming the following  
>>> character is a DIGIT without actually checking it first just  
>>> isn't right.
>>
>> Isn't right in that it won't do what is natural.  I couldnt' make  
>> antlr do the right thing easily in this case.
>
> Freudian slip? ;)

No, i meant every word.

> The original question was regarding Number = DIGIT+ ('.' DIGIT+)?  
> and how LL(*) was deciding that '.' implied the entire chain, when  
> it didn't. Perhaps another year would help, but I don't see how  
> this counts as working.

It works per my algorithm.  Don't like it? Use another tool.  I know  
i'm a prof, but i'm getting tired of schooling you boys.  ANY char  
can follow.  No analysis possible.


>
> Besides, isn't that what backtrack=... is for? :-) Maybe there's a  
> lexer::backtrack switch needed? Or maybe k needs to be * instead of 1.

See my prev email. mimicking lex is easy.  Or, i can finally add my  
option to emit lexer errors i normally hush.

>> Have you built a lex like creature before?  I have back in 1988.   
>> I believe I'm remembering how it operates correctly.
>>
> I don't think it matters, does it? You're in the recursive-descent  
> business, not the state-machine business. You sell chicken, they  
> sell hamburgers.

Yes, everyone i ssaying lex does it right though. sure. backtracking.  
so backtrack with antlr to do what you want.

Ok, i'm done now.
Ter