[antlr-interest] Multiple lexer tokens per rule

[Junkman]

Try this to get you started:
-----------------------------------------------------
@lexer::members {

    // Queue to hold additional tokens
    private java.util.Queue<Token> tokenQueue = new
java.util.LinkedList<Token>();

    // Include queue in reset().
    public void reset() {
        super.reset();
        tokenQueue.clear();
    }

    // Queued tokens are returned before matching a new token.
    public Token nextToken() {
        if (tokenQueue.peek() != null)
            return tokenQueue.poll();
        return super.nextToken();
    }

}

MATCHED_TOKEN:  ...
    {
        // Add additional tokens to the queue.
        tokenQueue( new CommonToken( ... ) );
    }

-----------------------------------------------------

MATCHED_TOKEN is returned first, and additional tokens queued by
MATCHED_TOKEN's action are returned subsequently before matching new
tokens in the input stream.

Instantiate the additional token accordingly if you need input stream
context - see Lexer.emit().



Ken Williams wrote:
> 
> On 6/3/10 4:18 PM, "Jim Idle" <jimi at temporal-wave.com> wrote:
> 
>> Add to an array or collection then get nextToken to remove from the
>> collection. It si slower to do this so it isn't the default way.
> 
> Yeah, that's what the book says. =)
> 
> It seems like there are some subtleties involved, though - there's a lot of
> bookkeeping in nextToken() that looks kind of scary (e.g. the
> current-line-number stuff, the default-channel stuff, etc.), and if I
> override it I'm really not confident I'll do it correctly.  I'm also unsure
> how mTokens(), emit(), and nextToken() cooperate with their member
> variables.
> 
> I tried this simple-minded implementation, and started getting out-of-bounds
> exceptions:
> 
> @lexer::members {
>     List<Token> tokBuf = new ArrayList<Token>();
>     public Token nextToken() {
>         while (tokBuf.isEmpty()) {
>             emit();
>         }
>         return tokBuf.remove(0);
>     }
>     public void emit(Token token) {
>         tokBuf.add(token);
>     }
> }
> 
> 
> So if someone does have a working example, I'd love to see it!
>