[antlr-interest] TokenRewriteStream

[Filipe David Manana]

For the input "var a = 1<EOF>", even if I insert the ";" token in the middle
of the stream, the parser gives me the error that encountered an unexpected
<EOF> after the "1" literal. Clearly it should report some other error. This
is why I state that it ignores the modified token stream.

E.g. :

        TokenRewriteStream tokens = new TokenRewriteStream(lexer);

        // debug
        System.err.println("token stream: " + tokens.toString());
        tokens.insertAfter(tokens.getTokens().size() - 2, ";");

        this.parser = new JavaScriptParser(tokens);

        // debug
        System.err.println("token stream: " + tokens.toString());
        System.err.println("token index: " + tokens.index());

running my parser, a feeding it with "var a = 1<EOF>" :

fdmanana at core2duo:~/projects/test$ java -jar test.jar -
var a = 1token stream:
token stream: var a = ;1
token index: 0
Error(s) found while parsing standard input

    line 0:-1 no viable alternative at input '<EOF>'
fdmanana at core2duo:~/projects/test$

If I provide " var a = ;1<EOF>" I get the expected error:  "line 1:8 no
viable alternative at input ';'".
Therefore this forces me to believe the parser/lexer ignores the modifed
token stream.

Any suggestion?

thanks


On Sun, Mar 8, 2009 at 7:52 PM, Filipe David Manana <fdmanana at ieee.org>wrote:

> Hi Terence,
>
> Nop, I tried it and it doesn't work either.
> It seems the parser gets only the original, unmodified, token stream.
>
> Any other suggestion?
>
> thanks
>
>
> On Sun, Mar 8, 2009 at 6:03 PM, Terence Parr <parrt at cs.usfca.edu> wrote:
>
>> maybe index should be size-1?T
>> On Mar 8, 2009, at 3:48 AM, Filipe David Manana wrote:
>>
>> Hello,
>>
>> I am trying to add a token into the token stream before the parser's
>> starting rule (method) is invoked.
>> I am doing it using a TokenRewriteStream and like this:
>>
>>         ANTLRInputStream input = new ANTLRInputStream(inputStream);
>>         this.lexer = new JavaScriptLexer(input);
>>         TokenRewriteStream tokens = new TokenRewriteStream(lexer);
>>
>>         // debug
>>         System.err.println("token stream: " + tokens.toString());
>>
>>         tokens.insertAfter(tokens.getTokens().size(), ";");
>>
>>         this.parser = new JavaScriptParser(tokens);
>>
>>         // debug
>>         System.err.println("token stream: " + tokens.toString());
>>         System.err.println("token index: " + tokens.index());
>>
>> The problem is that the parser seems to ignore the appended token.
>> The original token stream is:   "var a = 1<EOF>"
>> The output of the previous println statements is:
>>
>> token stream:
>> token stream: var a = 1;
>>
>> What am I doing wrong?
>> Or is there a better way to do it?
>>
>> thank you
>>
>>
>> --
>> Filipe David Manana,
>> fdmanana at ieee.org
>> PGP key - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC569452B
>>
>> "Reasonable men adapt themselves to the world.
>> Unreasonable men adapt the world to themselves.
>> That's why all progress depends on unreasonable men."
>>
>>
>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
>> Unsubscribe:
>> http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>>
>>
>>
>
>
> --
> Filipe David Manana,
> fdmanana at ieee.org
> PGP key - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC569452B
>
> "Reasonable men adapt themselves to the world.
> Unreasonable men adapt the world to themselves.
> That's why all progress depends on unreasonable men."
>
>


-- 
Filipe David Manana,
fdmanana at ieee.org
PGP key - http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC569452B

"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20090308/44afbf6d/attachment.html