[antlr-interest] Why does syntactic predicate not take effect?

[Britta Kiera]

Don't know if the first reply was sent. Trying it a second time.

> At 23:32 10/11/2008, Britta Kiera wrote:
> >The lexer was supposed to generate a NAMES token for the feature
> >name sequence. The definition below shows an approach that I tried
> >to accomplish this. This approach didn't work. The lexer never
> >generated a NAMES token although I tried
> >to enforce this using a syntactic predicate. I solved this problem
> >in the parser but I'd like to understand why the
> >syntactic predicate does not take effect. Can somebody explain this
> >to me?
>
> Are you using the interpreter or the debugger (or a "real" compiled program)? 
> Because the interpreter doesn't evaluate predicates.
I'm not using ANTLRWorks. I'm using the ANTLR IDE Eclipse plugin with
the ANTLR 3.1.1 runtime to generate the lexer code. Then I run the main
method of the generated lexer using the Eclipse "Run" command. The main
method of the grammar that I sent with my first mail contains a short test.

> >NAMES:
> >    ;
>
> You need to make this a fragment rule.  Otherwise you've got a top-level lexer
> rule which can successfully match nothing at all, which is Bad.  (Since that way
> lies infinite loops.)
In the grammar below NAMES has been made a fragment. But still it produces the same
output as before that doesn't contain a NAMES token:

Token:  WHITE(99) >   <
Token:  IDENT( 0) >plugins<
Token:  DOT( 0) >.<
Token:  IDENT( 0) >navigation<
Token:  DOT( 0) >.<
Token:  IDENT( 0) >XRefs<
Token:  WHITE(99) > <
Token:  IDENT( 0) >Outline<
Token:  WHITE(99) > <
Token:  IDENT( 0) >GoTo<
Token:  WHITE(99) > <
Token:  LB( 0) >{<
Token:  RB( 0) >}<
Token:  -1( 0) >null<

Regards,
Nukiti

========================= modified ANTLR lexer start ======================
lexer grammar SimpleLex;

options {
    language = Java;
}

@header {
package test.antlr;

import java.io.StringReader;
}

@members {
    public static void main(String args[]) throws Exception {
        String      input = "   plugins.navigation.XRefs Outline GoTo {}";
        CharStream  cs    = new ANTLRStringStream(input);
        SimpleLex   lex   = new SimpleLex(cs);
       
        Token t;
        do {
            String type = "?";
            t=lex.nextToken();
            switch(t.getType()) {
                case IDENT: type = "IDENT"; break;
                case NAMES: type = "NAMES"; break;
                case DOT  : type = "DOT"  ; break;
                case WHITE: type = "WHITE"; break;
                case LB   : type = "LB"   ; break;
                case RB   : type = "RB"   ; break;
                default   : type = Integer.toString(t.getType()); break;
            }
            System.out.printf("Token: %6s(%2d) >%s<\n", type, t.getChannel(), t.getText());
        }
        while(t.getType() != -1);
    }
}

IDENT
    : (ID (WS ID)+)=> ID (WS ID)+ {$type = NAMES;}
    | ID
    ;
    
WHITE
    : WS { $channel = HIDDEN; }
    ;

LB  : '{' ;
RB  : '}' ;
DOT : '.' ;

fragment NAMES :;
fragment WS    : (' '|'\r'|'\t'|'\u000C'|'\n')+;
fragment ID    : LETTER (LETTER|DIGIT)*;
fragment DIGIT : '0'..'9';
fragment LETTER: 'A'..'Z' | 'a'..'z' | '_';