[antlr-interest] TokenRewriteStream

[Terence Parr]

Oh wait...you are parsing original stream. rewrite stream only give  
modified txt not tokens. create new lexer attached to  
tokens.toString() output.
T
On Mar 8, 2009, at 12:00 PM, Filipe David Manana wrote:

> For the input "var a = 1<EOF>", even if I insert the ";" token in  
> the middle of the stream, the parser gives me the error that  
> encountered an unexpected <EOF> after the "1" literal. Clearly it  
> should report some other error. This is why I state that it ignores  
> the modified token stream.
>
> E.g. :
>
>         TokenRewriteStream tokens = new TokenRewriteStream(lexer);
>
>         // debug
>         System.err.println("token stream: " + tokens.toString());
>         tokens.insertAfter(tokens.getTokens().size() - 2, ";");
>
>         this.parser = new JavaScriptParser(tokens);
>
>         // debug
>         System.err.println("token stream: " + tokens.toString());
>         System.err.println("token index: " + tokens.index());
>
> running my parser, a feeding it with "var a = 1<EOF>" :
>
> fdmanana at core2duo:~/projects/test$ java -jar test.jar -
> var a = 1token stream:
> token stream: var a = ;1
> token index: 0
> Error(s) found while parsing standard input
>
>     line 0:-1 no viable alternative at input '<EOF>'
> fdmanana at core2duo:~/projects/test$
>
> If I provide " var a = ;1<EOF>" I get the expected error:  "line 1:8  
> no viable alternative at input ';'".
> Therefore this forces me to believe the parser/lexer ignores the  
> modifed token stream.
>
> Any suggestion?
>
> thanks
>
>
> On Sun, Mar 8, 2009 at 7:52 PM, Filipe David Manana  
> <fdmanana at ieee.org> wrote:
> Hi Terence,
>
> Nop, I tried it and it doesn't work either.
> It seems the parser gets only the original, unmodified, token stream.
>
> Any other suggestion?
>
> thanks
>
>
> On Sun, Mar 8, 2009 at 6:03 PM, Terence Parr <parrt at cs.usfca.edu>  
> wrote:
> maybe index should be size-1?
> T
> On Mar 8, 2009, at 3:48 AM, Filipe David Manana wrote:
>
>> Hello,
>>
>> I am trying to add a token into the token stream before the  
>> parser's starting rule (method) is invoked.
>> I am doing it using a TokenRewriteStream and like this:
>>
>>         ANTLRInputStream input = new ANTLRInputStream(inputStream);
>>         this.lexer = new JavaScriptLexer(input);
>>         TokenRewriteStream tokens = new TokenRewriteStream(lexer);
>>
>>         // debug
>>         System.err.println("token stream: " + tokens.toString());
>>
>>         tokens.insertAfter(tokens.getTokens().size(), ";");
>>
>>         this.parser = new JavaScriptParser(tokens);
>>
>>         // debug
>>         System.err.println("token stream: " + tokens.toString());
>>         System.err.println("token index: " + tokens.index());
>>
>> The problem is that the parser seems to ignore the appended token.
>> The original token stream is:   "var a = 1<EOF>"
>> The output of the previous println statements is:
>>
>> token stream:
>> token stream: var a = 1;
>>
>> What am I doing wrong?
>> Or is there a better way to do it?
>>
>> thank you
>>
>>
>> -- 
>> Filipe David Manana,
>> fdmanana at ieee.org
>> PGP key - http://pgp.mit.edu:11371/pks/lookup? 
>> op=get&search=0xC569452B
>>
>> "Reasonable men adapt themselves to the world.
>> Unreasonable men adapt the world to themselves.
>> That's why all progress depends on unreasonable men."
>>
>>
>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
>> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>
>
>
>
> -- 
> Filipe David Manana,
> fdmanana at ieee.org
> PGP key - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC569452B
>
> "Reasonable men adapt themselves to the world.
> Unreasonable men adapt the world to themselves.
> That's why all progress depends on unreasonable men."
>
>
>
>
> -- 
> Filipe David Manana,
> fdmanana at ieee.org
> PGP key - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC569452B
>
> "Reasonable men adapt themselves to the world.
> Unreasonable men adapt the world to themselves.
> That's why all progress depends on unreasonable men."
>