[antlr-interest] TokenRewriteStream

[Filipe David Manana]

Like this:

        ANTLRInputStream input = new ANTLRInputStream(inputStream);

        this.lexer = new JavaScriptLexer(input);

        TokenRewriteStream tokens = new TokenRewriteStream(lexer);

        tokens.insertAfter(tokens.getTokens().size() - 1, ";");

        this.lexer = new JavaScriptLexer(new
ANTLRStringStream(tokens.toString()));

        this.parser = new JavaScriptParser(tokens);

?
It doesn't work either :(

cheers

On Sun, Mar 8, 2009 at 8:14 PM, Terence Parr <parrt at cs.usfca.edu> wrote:

> Oh wait...you are parsing original stream. rewrite stream only give
> modified txt not tokens. create new lexer attached to tokens.toString()
> output.
> T
> On Mar 8, 2009, at 12:00 PM, Filipe David Manana wrote:
>
>  For the input "var a = 1<EOF>", even if I insert the ";" token in the
>> middle of the stream, the parser gives me the error that encountered an
>> unexpected <EOF> after the "1" literal. Clearly it should report some other
>> error. This is why I state that it ignores the modified token stream.
>>
>> E.g. :
>>
>>        TokenRewriteStream tokens = new TokenRewriteStream(lexer);
>>
>>        // debug
>>        System.err.println("token stream: " + tokens.toString());
>>        tokens.insertAfter(tokens.getTokens().size() - 2, ";");
>>
>>        this.parser = new JavaScriptParser(tokens);
>>
>>        // debug
>>        System.err.println("token stream: " + tokens.toString());
>>        System.err.println("token index: " + tokens.index());
>>
>> running my parser, a feeding it with "var a = 1<EOF>" :
>>
>> fdmanana at core2duo:~/projects/test$ java -jar test.jar -
>> var a = 1token stream:
>> token stream: var a = ;1
>> token index: 0
>> Error(s) found while parsing standard input
>>
>>    line 0:-1 no viable alternative at input '<EOF>'
>> fdmanana at core2duo:~/projects/test$
>>
>> If I provide " var a = ;1<EOF>" I get the expected error:  "line 1:8 no
>> viable alternative at input ';'".
>> Therefore this forces me to believe the parser/lexer ignores the modifed
>> token stream.
>>
>> Any suggestion?
>>
>> thanks
>>
>>
>> On Sun, Mar 8, 2009 at 7:52 PM, Filipe David Manana <fdmanana at ieee.org>
>> wrote:
>> Hi Terence,
>>
>> Nop, I tried it and it doesn't work either.
>> It seems the parser gets only the original, unmodified, token stream.
>>
>> Any other suggestion?
>>
>> thanks
>>
>>
>> On Sun, Mar 8, 2009 at 6:03 PM, Terence Parr <parrt at cs.usfca.edu> wrote:
>> maybe index should be size-1?
>> T
>> On Mar 8, 2009, at 3:48 AM, Filipe David Manana wrote:
>>
>>  Hello,
>>>
>>> I am trying to add a token into the token stream before the parser's
>>> starting rule (method) is invoked.
>>> I am doing it using a TokenRewriteStream and like this:
>>>
>>>        ANTLRInputStream input = new ANTLRInputStream(inputStream);
>>>        this.lexer = new JavaScriptLexer(input);
>>>        TokenRewriteStream tokens = new TokenRewriteStream(lexer);
>>>
>>>        // debug
>>>        System.err.println("token stream: " + tokens.toString());
>>>
>>>        tokens.insertAfter(tokens.getTokens().size(), ";");
>>>
>>>        this.parser = new JavaScriptParser(tokens);
>>>
>>>        // debug
>>>        System.err.println("token stream: " + tokens.toString());
>>>        System.err.println("token index: " + tokens.index());
>>>
>>> The problem is that the parser seems to ignore the appended token.
>>> The original token stream is:   "var a = 1<EOF>"
>>> The output of the previous println statements is:
>>>
>>> token stream:
>>> token stream: var a = 1;
>>>
>>> What am I doing wrong?
>>> Or is there a better way to do it?
>>>
>>> thank you
>>>
>>>
>>> --
>>> Filipe David Manana,
>>> fdmanana at ieee.org
>>> PGP key - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC569452B
>>>
>>> "Reasonable men adapt themselves to the world.
>>> Unreasonable men adapt the world to themselves.
>>> That's why all progress depends on unreasonable men."
>>>
>>>
>>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
>>> Unsubscribe:
>>> http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>>>
>>
>>
>>
>>
>> --
>> Filipe David Manana,
>> fdmanana at ieee.org
>> PGP key - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC569452B
>>
>> "Reasonable men adapt themselves to the world.
>> Unreasonable men adapt the world to themselves.
>> That's why all progress depends on unreasonable men."
>>
>>
>>
>>
>> --
>> Filipe David Manana,
>> fdmanana at ieee.org
>> PGP key - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC569452B
>>
>> "Reasonable men adapt themselves to the world.
>> Unreasonable men adapt the world to themselves.
>> That's why all progress depends on unreasonable men."
>>
>>
>


-- 
Filipe David Manana,
fdmanana at ieee.org
PGP key - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC569452B

"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20090309/a93515b7/attachment.html