[antlr-interest] skip not honored when filter=true

[Fady Moussallam]

Hello,

I am using ANTLR 3.2.

With a lexer grammar such as this one:

        lexer grammar Z;
        options {
            filter=true;
        }
        IS_KEYWORD                : 'IS' {skip();};

The skip() is ignored and the token gets inserted in the token stream.

If I change to filter=false, skip is honored.

The problem might come from the nextToken() method which seems to be
overriden in the generated lexer code when filter=true.

The nextToken() method in org.antlr.runtime.Lexer has code like this:

        try {
            mTokens();
            if ( state.token==null ) {
                emit();
            }
            else if ( state.token==Token.SKIP_TOKEN ) {
                continue;
            }
            return state.token;
        }

While the generated equivalent code (in Z.java) looks like this:

        try {
            int m = input.mark();
            state.backtracking=1; 
            state.failed=false;
            mTokens();
            state.backtracking=0;
            
            if ( state.failed ) {
                input.rewind(m);
                input.consume(); 
            }
            else {
                emit();
                return state.token;
            }
        }
So whatever the state.token value, the token is emitted.

Or I messed up and there some other explanation?

Thanks for your help.

Fady