[antlr-interest] Lexer consumes input but doesn't emit all tokens

[Glenn McGregor]

On 8/7/2012 3:05 PM, Андрей Асеев wrote:
> It would be, at example, if you choose incorrect rule in ANTLRWorks
> interpreter rule box. :)
>
> Or show there your gramar rule you use to parse test input.

If i change the input to 'test:ack :', it parses just fine, and returns 
appropriate tokens.


The grammar is about 500 lines long, and I tried to show just the 
hopefully relevant entries.

But it the rule in AntlrWorks starts at my

start_program
     :    program EOF! ;


I can post my grammar somewhere if it becomes necessary to pursue this.


The output of the interpreter shows  (in bad ascii art)

<grammar Tal>
     start_program
         program
             <epsilon>
         t

with the altered input, I get

<grammar Tal>
     start_program
         program
             statement
                 label_statement
                     string_literal
                         test:ack
                     :
<EOF>

Thanks

Glenn

>
>> Given the partial grammar from a much larger...
>>
>>
>> tokens { COLON = ':' }
>>
>> fragment
>> ALPHA_NUM
>>        :    'A'..'Z' | 'a'..'z' | '0'..'9';
>>
>> NAME_LITERAL
>>        :    '\\'? ALPHA_NUM ( ( ':' | '_' | '-' | ALPHA_NUM )* ALPHA_NUM )? ;
>>
>> ANY    :    . ;
>>
>>
>>
>> I would like the input
>>
>> test:ack:
>>
>> to arrive as two tokens, a NAME_LITERAL of 'test:ack', and a COLON.
>>
>> Instead, this input disappears entirely, but parses successfully.
>>
>> Any suggestions?
>>
>> Glenn McGregor