[antlr-interest] TokenRewriteStream

Filipe David Manana fdmanana at ieee.org
Mon Mar 9 13:33:12 PDT 2009


Ah!
Clearly I was missing that.
It's working now.

many thanks and keep up the excellent work with ANTLR


On Mon, Mar 9, 2009 at 9:06 PM, Terence Parr <parrt at cs.usfca.edu> wrote:

>
> On Mar 9, 2009, at 12:13 PM, Filipe David Manana wrote:
>
>  Like this:
>>
>>        ANTLRInputStream input = new ANTLRInputStream(inputStream);
>>
>>        this.lexer = new JavaScriptLexer(input);
>>
>>        TokenRewriteStream tokens = new TokenRewriteStream(lexer);
>>
>>        tokens.insertAfter(tokens.getTokens().size() - 1, ";");
>>
>>        this.lexer = new JavaScriptLexer(new
>> ANTLRStringStream(tokens.toString()));
>>
>>
> you are clearly reusing tokens still. make new token stream attached to
> this.lexer
> Ter
>
>         this.parser = new JavaScriptParser(tokens);
>>
>> ?
>> It doesn't work either :(
>>
>> cheers
>>
>> On Sun, Mar 8, 2009 at 8:14 PM, Terence Parr <parrt at cs.usfca.edu> wrote:
>> Oh wait...you are parsing original stream. rewrite stream only give
>> modified txt not tokens. create new lexer attached to tokens.toString()
>> output.
>> T
>>
>> On Mar 8, 2009, at 12:00 PM, Filipe David Manana wrote:
>>
>> For the input "var a = 1<EOF>", even if I insert the ";" token in the
>> middle of the stream, the parser gives me the error that encountered an
>> unexpected <EOF> after the "1" literal. Clearly it should report some other
>> error. This is why I state that it ignores the modified token stream.
>>
>> E.g. :
>>
>>       TokenRewriteStream tokens = new TokenRewriteStream(lexer);
>>
>>       // debug
>>       System.err.println("token stream: " + tokens.toString());
>>       tokens.insertAfter(tokens.getTokens().size() - 2, ";");
>>
>>       this.parser = new JavaScriptParser(tokens);
>>
>>       // debug
>>       System.err.println("token stream: " + tokens.toString());
>>       System.err.println("token index: " + tokens.index());
>>
>> running my parser, a feeding it with "var a = 1<EOF>" :
>>
>> fdmanana at core2duo:~/projects/test$ java -jar test.jar -
>> var a = 1token stream:
>> token stream: var a = ;1
>> token index: 0
>> Error(s) found while parsing standard input
>>
>>   line 0:-1 no viable alternative at input '<EOF>'
>> fdmanana at core2duo:~/projects/test$
>>
>> If I provide " var a = ;1<EOF>" I get the expected error:  "line 1:8 no
>> viable alternative at input ';'".
>> Therefore this forces me to believe the parser/lexer ignores the modifed
>> token stream.
>>
>> Any suggestion?
>>
>> thanks
>>
>>
>> On Sun, Mar 8, 2009 at 7:52 PM, Filipe David Manana <fdmanana at ieee.org>
>> wrote:
>> Hi Terence,
>>
>> Nop, I tried it and it doesn't work either.
>> It seems the parser gets only the original, unmodified, token stream.
>>
>> Any other suggestion?
>>
>> thanks
>>
>>
>> On Sun, Mar 8, 2009 at 6:03 PM, Terence Parr <parrt at cs.usfca.edu> wrote:
>> maybe index should be size-1?
>> T
>> On Mar 8, 2009, at 3:48 AM, Filipe David Manana wrote:
>>
>> Hello,
>>
>> I am trying to add a token into the token stream before the parser's
>> starting rule (method) is invoked.
>> I am doing it using a TokenRewriteStream and like this:
>>
>>       ANTLRInputStream input = new ANTLRInputStream(inputStream);
>>       this.lexer = new JavaScriptLexer(input);
>>       TokenRewriteStream tokens = new TokenRewriteStream(lexer);
>>
>>       // debug
>>       System.err.println("token stream: " + tokens.toString());
>>
>>       tokens.insertAfter(tokens.getTokens().size(), ";");
>>
>>       this.parser = new JavaScriptParser(tokens);
>>
>>       // debug
>>       System.err.println("token stream: " + tokens.toString());
>>       System.err.println("token index: " + tokens.index());
>>
>> The problem is that the parser seems to ignore the appended token.
>> The original token stream is:   "var a = 1<EOF>"
>> The output of the previous println statements is:
>>
>> token stream:
>> token stream: var a = 1;
>>
>> What am I doing wrong?
>> Or is there a better way to do it?
>>
>> thank you
>>
>>
>> --
>> Filipe David Manana,
>> fdmanana at ieee.org
>> PGP key - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC569452B
>>
>> "Reasonable men adapt themselves to the world.
>> Unreasonable men adapt the world to themselves.
>> That's why all progress depends on unreasonable men."
>>
>>
>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
>> Unsubscribe:
>> http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>>
>>
>>
>>
>> --
>> Filipe David Manana,
>> fdmanana at ieee.org
>> PGP key - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC569452B
>>
>> "Reasonable men adapt themselves to the world.
>> Unreasonable men adapt the world to themselves.
>> That's why all progress depends on unreasonable men."
>>
>>
>>
>>
>> --
>> Filipe David Manana,
>> fdmanana at ieee.org
>> PGP key - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC569452B
>>
>> "Reasonable men adapt themselves to the world.
>> Unreasonable men adapt the world to themselves.
>> That's why all progress depends on unreasonable men."
>>
>>
>>
>>
>>
>> --
>> Filipe David Manana,
>> fdmanana at ieee.org
>> PGP key - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC569452B
>>
>> "Reasonable men adapt themselves to the world.
>> Unreasonable men adapt the world to themselves.
>> That's why all progress depends on unreasonable men."
>>
>>
>


-- 
Filipe David Manana,
fdmanana at ieee.org
PGP key - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC569452B

"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20090309/83c09d00/attachment.html 


More information about the antlr-interest mailing list