[antlr-interest] Adding a Space Leads to Mismatch

[Quintin Beukes]

I have tried to skip whitespace and have used tokens. The above
grammar is mostly just in debug state.

If I can narrow down the problem even further. The lexer keeps
predicting the "If " to be ALPHANUMERICSPACE, so the lexer fails. I
can actually not see why it would even do that, because this string
can never even match ALPHANUMERICSPACE.

Input:
(If )

grammar DebugA;

tokens {
  IF = 'If';
  OB = '(';
  CB = ')';
}

fieldRules
  : rule
    EOF
  ;

rule
  : OB ruleExpression CB NEWLINE
  ;

ruleExpression
  : IF ALPHANUMERIC
  ;

ALPHANUMERIC
  : ('a'..'z' | 'A'..'Z' | '0'..'9')+
  ;

ALPHANUMERICSPACE
  : ('a'..'z' | 'A'..'Z' | '0'..'9')+ (' '+ ('a'..'z' | 'A'..'Z' | '0'..'9')+)*
  ;

WS
  : (' '|'\t')+ {skip();}
  ;

NEWLINE
  : '\r'? '\n'
  ;


Quintin Beukes

On Thu, Feb 9, 2012 at 9:30 PM, Jim Idle <jimi at temporal-wave.com> wrote:
> Don't use 'strings' in your parser, create real tokens and list the
> keywords and punctuation in the lexer before the generic rule. Also, it
> does not look like you need the spaces, so try skipping them:
>
> LPAREN: '(' ;
> ...
> KEYWORD: 'keyword';
> ....
> ALPHANUMERICSPACE: 'A'..'Z'+ ... etc
>
> WS: (' '|'\t')+ { skip(); } ;  // Then remove WS refs in your parser
>
>
> Jim
>
>> -----Original Message-----
>> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
>> bounces at antlr.org] On Behalf Of Quintin Beukes
>> Sent: Thursday, February 09, 2012 11:20 AM
>> To: antlr-interest at antlr.org
>> Subject: Re: [antlr-interest] Adding a Space Leads to Mismatch
>>
>> I debugged the Lexer, and it seems that it's predictions for the next
>> token always seems to match against ALPHANUMERICSPACE.
>>
>> How can I resolve such a prediction error? Even if just pointing me to
>> the wiki.
>>
>> thanks,
>> Quintin Beukes
>>
>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
>> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
>> email-address
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address