[antlr-interest] TokenRewriteStream

[Terence Parr]

On Mar 9, 2009, at 12:13 PM, Filipe David Manana wrote:

> Like this:
>
>         ANTLRInputStream input = new ANTLRInputStream(inputStream);
>
>         this.lexer = new JavaScriptLexer(input);
>
>         TokenRewriteStream tokens = new TokenRewriteStream(lexer);
>
>         tokens.insertAfter(tokens.getTokens().size() - 1, ";");
>
>         this.lexer = new JavaScriptLexer(new  
> ANTLRStringStream(tokens.toString()));
>

you are clearly reusing tokens still. make new token stream attached  
to this.lexer
Ter
>         this.parser = new JavaScriptParser(tokens);
>
> ?
> It doesn't work either :(
>
> cheers
>
> On Sun, Mar 8, 2009 at 8:14 PM, Terence Parr <parrt at cs.usfca.edu>  
> wrote:
> Oh wait...you are parsing original stream. rewrite stream only give  
> modified txt not tokens. create new lexer attached to  
> tokens.toString() output.
> T
>
> On Mar 8, 2009, at 12:00 PM, Filipe David Manana wrote:
>
> For the input "var a = 1<EOF>", even if I insert the ";" token in  
> the middle of the stream, the parser gives me the error that  
> encountered an unexpected <EOF> after the "1" literal. Clearly it  
> should report some other error. This is why I state that it ignores  
> the modified token stream.
>
> E.g. :
>
>        TokenRewriteStream tokens = new TokenRewriteStream(lexer);
>
>        // debug
>        System.err.println("token stream: " + tokens.toString());
>        tokens.insertAfter(tokens.getTokens().size() - 2, ";");
>
>        this.parser = new JavaScriptParser(tokens);
>
>        // debug
>        System.err.println("token stream: " + tokens.toString());
>        System.err.println("token index: " + tokens.index());
>
> running my parser, a feeding it with "var a = 1<EOF>" :
>
> fdmanana at core2duo:~/projects/test$ java -jar test.jar -
> var a = 1token stream:
> token stream: var a = ;1
> token index: 0
> Error(s) found while parsing standard input
>
>    line 0:-1 no viable alternative at input '<EOF>'
> fdmanana at core2duo:~/projects/test$
>
> If I provide " var a = ;1<EOF>" I get the expected error:  "line 1:8  
> no viable alternative at input ';'".
> Therefore this forces me to believe the parser/lexer ignores the  
> modifed token stream.
>
> Any suggestion?
>
> thanks
>
>
> On Sun, Mar 8, 2009 at 7:52 PM, Filipe David Manana  
> <fdmanana at ieee.org> wrote:
> Hi Terence,
>
> Nop, I tried it and it doesn't work either.
> It seems the parser gets only the original, unmodified, token stream.
>
> Any other suggestion?
>
> thanks
>
>
> On Sun, Mar 8, 2009 at 6:03 PM, Terence Parr <parrt at cs.usfca.edu>  
> wrote:
> maybe index should be size-1?
> T
> On Mar 8, 2009, at 3:48 AM, Filipe David Manana wrote:
>
> Hello,
>
> I am trying to add a token into the token stream before the parser's  
> starting rule (method) is invoked.
> I am doing it using a TokenRewriteStream and like this:
>
>        ANTLRInputStream input = new ANTLRInputStream(inputStream);
>        this.lexer = new JavaScriptLexer(input);
>        TokenRewriteStream tokens = new TokenRewriteStream(lexer);
>
>        // debug
>        System.err.println("token stream: " + tokens.toString());
>
>        tokens.insertAfter(tokens.getTokens().size(), ";");
>
>        this.parser = new JavaScriptParser(tokens);
>
>        // debug
>        System.err.println("token stream: " + tokens.toString());
>        System.err.println("token index: " + tokens.index());
>
> The problem is that the parser seems to ignore the appended token.
> The original token stream is:   "var a = 1<EOF>"
> The output of the previous println statements is:
>
> token stream:
> token stream: var a = 1;
>
> What am I doing wrong?
> Or is there a better way to do it?
>
> thank you
>
>
> -- 
> Filipe David Manana,
> fdmanana at ieee.org
> PGP key - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC569452B
>
> "Reasonable men adapt themselves to the world.
> Unreasonable men adapt the world to themselves.
> That's why all progress depends on unreasonable men."
>
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>
>
>
>
> -- 
> Filipe David Manana,
> fdmanana at ieee.org
> PGP key - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC569452B
>
> "Reasonable men adapt themselves to the world.
> Unreasonable men adapt the world to themselves.
> That's why all progress depends on unreasonable men."
>
>
>
>
> -- 
> Filipe David Manana,
> fdmanana at ieee.org
> PGP key - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC569452B
>
> "Reasonable men adapt themselves to the world.
> Unreasonable men adapt the world to themselves.
> That's why all progress depends on unreasonable men."
>
>
>
>
>
> -- 
> Filipe David Manana,
> fdmanana at ieee.org
> PGP key - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC569452B
>
> "Reasonable men adapt themselves to the world.
> Unreasonable men adapt the world to themselves.
> That's why all progress depends on unreasonable men."
>