[antlr-interest] Is it a BUG in the Lexer-Rule-Checker?

[Nigel Sheridan-Smith]

> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> bounces at antlr.org] On Behalf Of T.Lindig
> Sent: Friday, 28 January 2005 7:04 AM
> To: antlr-interest at antlr.org
> Subject: [antlr-interest] Is it a BUG in the Lexer-Rule-Checker?
> 
> Hi,
> 
> don't know, if this is the right place for bug reports, but i can't
> found a better one.
> 
> I working with ANTLR 2.7.4
> 
> the folowing Lexer-Grammer brings a "lexical nondeterminism" warning.
> 
>  >>>>>>>>>> begin <<<<<<<<<<<<<<<<<
> class COMALexer extends Lexer;
> 
> options {
>    k=3;
> }
> 
> PREPROCESSOR:
>    "#test"
>    (PREPRO_INLINE_WS)?   //here we get the warning
>    (SL_COMMENT)?
> ;
> 
> protected
> PREPRO_INLINE_WS :
>    (
>      INLINE_SPACE
>    | ML_COMMENT
>    )+
> ;
> 
> protected
> INLINE_SPACE
>    :   ' ' | '\t' | '\f'
>    ;
> 
> // multiple-line comments
> ML_COMMENT
>    :
>      "/*" ( options{greedy=false;} :  . )*
>      "*/"
>      { $setType(Token.SKIP); }
>    ;
> 
> // Single-line comments
> SL_COMMENT
>    :  "//" ( ~('\n'|'\r'))*
>    { $setType(Token.SKIP); }
> ;
> 
> 
> // Whitespace -- ignored
> WS:
>    ( INLINE_SPACE )+
>    { $setType(Token.SKIP); }
> ;
> 
>  >>>>>>>>>>>>>>>>>>>>>>> end <<<<<<<<<<<<<<<<<<<<<<
> 
> 
> ANTLR Parser Generator   Version 2.7.4   1989-2004 jGuru.com
> CommentTest.g:9:3: warning:lexical nondeterminism between alts 1 and 2
> of block upon
> CommentTest.g:9:3:     k==1:'/'
> CommentTest.g:9:3:     k==2:<end-of-token>,'/'
> CommentTest.g:9:3:     k==3:<end-of-token>,'\t','\u000c','
> ','#','*','/','e','s','t'
> 
> 
> but if i substitute the PREPRO_INLINE_WS token in the PREPROCESSOR rule
> like here:
> 
> PREPROCESSOR:
>    "#test"
>    ( INLINE_SPACE | ML_COMMENT )*
>    (SL_COMMENT)?
> ;
> 
> I will get no warning. know also ANTLR see, that this code is correct.
> 
> so, is it a bug in ANTLR or did i looked over something?
> 


No bug... you might get a SL_COMMENT that is part of the PREPROCESSOR token
or part of its own token. 

Nigel

--
Nigel Sheridan-Smith
PhD research student

Faculty of Engineering
University of Technology, Sydney
Phone: 02 9514 7946
Fax: 02 9514 2435