[antlr-interest] Returning Multiple Tokens

FranklinChen at cmu.edu FranklinChen at cmu.edu
Mon Jul 5 14:31:57 PDT 2004


Jason <jasonriz at yahoo.com> writes:
> Hello,
> 
> There are certain circumstances under which I'd like
> to return more than a single token to the parser.

One of my solutions is like yours.  Concretely, I stash Tokens into a
LinkedList in the lexer, then I wrap the lexer and LinkedList in
another, which I pass to the parser (code below).

The other solution I have used in more complicated situations is to
rethink the design, and pass a single complex Token up to the parser,
and there use another parser to parse that Token, and insert the parse
tree from that parser into the main parse tree, and then process the
full parse tree.



import java.util.*;
import antlr.*;

/**
 * Use queue of tokens before the existing token stream.
 */
public class InsertedTokenStream implements TokenStream {
    private final TokenStream stream;
    private final LinkedList insertedTokens;

    public InsertedTokenStream(TokenStream stream, LinkedList insertedTokens) {
        this.stream = stream;
        this.insertedTokens = insertedTokens;
    }

    public Token nextToken() throws TokenStreamException {
        if (insertedTokens.size() == 0) {
            return stream.nextToken();
        }
        else {
            return (Token) insertedTokens.removeFirst();
        }
    }
}

-- 
Franklin


 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
    antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/
 



More information about the antlr-interest mailing list