[antlr-interest] Embedded token stream technique

Monty Zukowski monty at codetransform.com
Thu Apr 29 11:24:27 PDT 2004


I've used a similar approach before, this will work.

Monty

On Apr 29, 2004, at 11:09 AM, FranklinChen at cmu.edu wrote:

> I asked earlier in several messages about a variety of lexer
> techniques I tried in order to do what I need to do... I never got any
> feedback about any of it, but I think I have come up with an
> interesting hack, and am interested whether it introduces unforeseen
> problems:
>
> Say I am parsing and currently use a lexer that has a token FLOAT,
> which is FLOAT_INTEGER DOT FLOAT_FRACTION (assume all protected).  Now
> assume that my parser wants to get the subparts of the token.  (Never
> mind that in this toy example this makes no sense: in real examples,
> where the token is actually recursive, I do want access to subparts of
> a token, for reparsing in the right context.)
>
> I've implemented a trick as follows (relevant excerpt
> follows). Comments?
>
>
> header {
>     import java.util.LinkedList;
> }
>
> class FloatParser extends Parser;
>
> options {
>     buildAST = true;
> }
>
> tokens {
>     NUMBER;
> }
>
> number
>     :
>         i:FLOAT_INTEGER
>         DOT!
>         f:FLOAT_FRACTION
>         {
>           #number = #([NUMBER], #number);
>         }
>     ;
>
> class FloatLexer extends Lexer;
>
> {
>     /** Queue of tokens. */
>     protected LinkedList insertedTokens = new LinkedList();
> }
>
> // Want to push tokens back on the stream in order to treat as
> // "integer" followed by "fraction".
> FLOAT
>     :
>         i:FLOAT_INTEGER
>         {
>             LinkedList queue = new LinkedList();
>             queue.add(i);
>         }
>         d:DOT
>         {
>             queue.add(d);
>         }
>         f:FLOAT_FRACTION
>         {
>             queue.add(f);
>         }
>
>         {
>             // Have to manually kick off the token stream.
>             Token token = (Token) queue.removeFirst();
>             $setToken(token);
>             System.err.println("*** inserting tokens: " + queue);
>             insertedTokens.addAll(queue);
>         }
>     ;
>
> protected
> FLOAT_INTEGER
>     :
>         ('0'..'9')+
>     ;
>
> protected
> FLOAT_FRACTION
>     :
>         ('0'..'9')+
>     ;
>
> protected
> DOT
>     :
>         '.'
>     ;
>
>
> import java.io.InputStream;
> import java.io.Reader;
>
> import antlr.Token;
> import antlr.TokenStreamException;
> import antlr.InputBuffer;
> import antlr.LexerSharedInputState;
>
>
> /**
>  * Use queue of tokens before the existing token stream.
>  */
> public class InsertedFloatLexer extends FloatLexer {
>     public InsertedFloatLexer(InputStream in) {
>         super(in);
>     }
>
>     public InsertedFloatLexer(Reader in) {
>         super(in);
>     }
>
>     public InsertedFloatLexer(InputBuffer ib) {
>         super(ib);
>     }
>
>     public InsertedFloatLexer(LexerSharedInputState state) {
>         super(state);
>     }
>
>     public Token nextToken() throws TokenStreamException {
>         if (insertedTokens.size() == 0) {
>             return super.nextToken();
>         }
>         else {
>             return (Token) insertedTokens.removeFirst();
>         }
>     }
> }
>
>
> import antlr.TokenStream;
> import antlr.Token;
> import antlr.TokenStreamException;
>
> public class FloatDriver {
>     public static void main(String[] args) {
>         try {
>             InsertedFloatLexer lexer = new 
> InsertedFloatLexer(System.in);
>             FloatParser parser = new FloatParser(lexer);
>
>             parser.start();
>         }
>         catch (Exception e) {
>             System.err.println(e.getMessage());
>         }
>     }
> }
>
>
>
> -- 
> Franklin
>
>
>
> Yahoo! Groups Links
>
>
>
>
>
>
>
>
Monty Zukowski

ANTLR & Java Consultant -- http://www.codetransform.com
ANSI C/GCC transformation toolkit -- 
http://www.codetransform.com/gcc.html
Embrace the Decay -- http://www.codetransform.com/EmbraceDecay.html



 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/
 



More information about the antlr-interest mailing list