[antlr-interest] Looping Lexer

[Terence Parr]

On Jun 11, 2007, at 4:57 PM, Stephen Gargan wrote:
> The question remains though, what is wrong with my grammar that  
> would cause it to loop like this? It compiles fine, though that is  
> not to say it is valid ;), I'm new to all of this and would not be  
> surprised if it were my fault (indeed I'm hoping this is the case).  
> Is there anything obviously wrong in my grammar that would cause it  
> to loop like this? If so is there anything I should look for as  
> tell-tale to avoid this problem? Might there be a way for the lexer  
> to know it is stuck in a loop and complain?

No empty lexer rules. ;)  * and ? are bad in the lexer if they are  
entire production.  I need to make a warning for this.

Ter
>
> grammar simple;
>
> paths    :     (path)*;
> path    :    VOLUME | NAME NEWLINE;
> VOLUME    :    'volume' NUMBER;
> NAME    :    (WORD? WS?)*;
> NUMBER    :     ('0'..'9')+ ;
> WORD     :    ('a'..'z'|'A'..'Z'|'_')*;
> WS     :     ( '\t' | ' ')+     { $channel = HIDDEN; } ;
> NEWLINE :    '\r'? '\n' ;
>
> The following snippet of code runs using the files generated by  
> antlrworks  and gets stuck in the lexer. The test input is
>
> volume1
> this is a test volume2
>
> public class LexerTest {
>
>     public static void main(String[] args) throws Exception{
>         simpleLexer lex = new simpleLexer(new ANTLRFileStream("/tmp/ 
> antlrworks/__Test___input.txt"));
>         System.out.println("Created Lexer");
>
>         Token t = null;
>         do{
>             t = lex.nextToken();
>             System.out.println(t.toString());
>         }
>         while(t != null);
>
>         System.out.println("done"); // never gets here
>
>     }
> }
>
>
> I am working on reproducing the grammar that caused the repeated  
> emission of the same token. I'll post again when I have failing  
> this way again. Anyone know what might be wrong?
>
> Thanks in advance,
>
> Stephen.
>
>
>