[antlr-interest] Returning Multiple Tokens

[Jason]

--- FranklinChen at cmu.edu wrote:
> Jason <jasonriz at yahoo.com> writes:
> > Hello,
> > 
> > There are certain circumstances under which I'd
> like
> > to return more than a single token to the parser.
> 
> One of my solutions is like yours.  Concretely, I
> stash Tokens into a
> LinkedList in the lexer, then I wrap the lexer and
> LinkedList in
> another, which I pass to the parser (code below).
> 
> The other solution I have used in more complicated
> situations is to
> rethink the design, and pass a single complex Token
> up to the parser,
> and there use another parser to parse that Token,
> and insert the parse
> tree from that parser into the main parse tree, and
> then process the
> full parse tree.
> 
> 
> 
> import java.util.*;
> import antlr.*;
> 
> /**
>  * Use queue of tokens before the existing token
> stream.
>  */
> public class InsertedTokenStream implements
> TokenStream {
>     private final TokenStream stream;
>     private final LinkedList insertedTokens;
> 
>     public InsertedTokenStream(TokenStream stream,
> LinkedList insertedTokens) {
>         this.stream = stream;
>         this.insertedTokens = insertedTokens;
>     }
> 
>     public Token nextToken() throws
> TokenStreamException {
>         if (insertedTokens.size() == 0) {
>             return stream.nextToken();
>         }
>         else {
>             return (Token)
> insertedTokens.removeFirst();
>         }
>     }
> }
> 

Franklin,

Thanks for the code.  I hadn't ever looked at the
TokenStream interface and didn't realize how simple it
was.  This is essentially, what I had in mind, but, of
course, has the practical advantage that I don't have
to modify the generated code.  Thanks again.

-exits


		
__________________________________
Do you Yahoo!?
Yahoo! Mail - 50x more storage than other providers!
http://promotions.yahoo.com/new_mail


 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
    antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/