[antlr-interest] Re: catching an exception in the parser and cont inuing - will this work?

Fri Jan 24 09:47:39 PST 2003

The single spot you are asking about is the consume() method.  Everything
goes through there.  Look at the generated code.  

Monty

-----Original Message-----
From: davidjpenton2002 <djplists at arrowsash.com>
[mailto:djplists at arrowsash.com]
Sent: Friday, January 24, 2003 9:23 AM
To: antlr-interest at yahoogroups.com
Subject: [antlr-interest] Re: catching an exception in the parser and
cont inuing - will this work?

Thanks for your help.

I had hoped that the "bad" token could be handled by a catch block 
somewhere.  The "somewhere" part may be the catch.  It appears to me 
that the normal behavior of an antlr parser is to report an 
unexpected token, and then to forge ahead.  That's the behavior I 
observe if I place a token (or rather a string that will match a 
known lexer rule) in the wrong spot in my input.  The parser reports 
it as an unexpected token, but continues to parse.  That is the 
behavior that I want to use by returning a "special" token unknown to 
any parser rule. I do not (yet) understand just where and how the 
normal processing happens in the parser code - I'm busy looking 
through it to find out, especially as regards exception handling.  
Any hints on this would help, although I am also re-reading the docs 
on exceptions.

The main thing that would make life easier for me is if I can 
identify a single spot to handle the information about source of 
tokens.  That is, it would be a pain in the neck if such processing 
had to be associated with each parser rule separately.  I'm not sure 
if I am being clear about that.

The other two strategies that you suggest (hidden token stream; 
embedding file information in the token) may also be fruitful for me, 
so I'll look into both. Thanks for the excellent insights.

- Dave Penton -

--- In antlr-interest at yahoogroups.com, mzukowski at y... wrote:
> What I do in the gcc project is have tokens aware of the file they 
came from
> as well.  Then it is pretty easy to tell whether a new file has been
> included or not.
> 
> There are other ways too, like Ter's fieldguide on whitespace 
preservation
> by using a hidden token stream.
> 
> You'll have to be more specific about what you mean by making the 
parser
> "aware" of the token change.  There are other ways to do this.  
> 
> Your way might work, but exceptions are expensive.  Wait, it won't 
work
> because it will screw up lookahead.  No rule will know about your 
bad token
> and that would be "bad".
> 
> If you don't like having the filename in the token, I would instead 
put that
> "bad" token in a hidden token stream.  You can override consume() 
to check
> the hidden token stream and set your state.  Or it can wait in the 
hidden
> stream and everytime you check that state you could pull in the 
tokens from
> the hidden stream and set state before returning it.
> 
> Monty
> 
> -----Original Message-----
> From: davidjpenton2002 <djplists at a...>
> [mailto:djplists at a...]
> Sent: Friday, January 24, 2003 7:32 AM
> To: antlr-interest at yahoogroups.com
> Subject: [antlr-interest] catching an exception in the parser and
> continuing - will this work?
> 
> 
> I am handling "include files" in my lexer, as in the java example 
> provided with antlr.  This means that fact the lexer is in an 
> included file is transparent to the parser.
> 
> However, it turns out that I want the parser to be made aware of 
the 
> fact that the lexer is in an included file. (Kindly assume that I 
> have a sound design reason for this ;-) Given the amount of 
lookahead 
> that may occur, I am sure that the only safe way to do this is by 
> putting the correct "notification" on the stream of tokens passed 
to 
> the parser.
> 
> Is it possible to have the lexer pass a back a "bad" token (i.e. 
one 
> that the parser can never recognize), catch a resultant exception, 
> and then (here's the important part) force the parser to take up 
> right where is was when it got the "bad" token?  It would amount to 
> skipping over the "bad" token, except that the exception handling 
> code would set a flag or two that would be available later to 
parser 
> semantic actions.
> 
> I have looked at the exception handling documentation, and sifted 
> through my generated parser.  It is not clear to me if my strategy 
> will work.
> 
> BTW, I am still using antlr 2.7.1 - I want to try to finish what I 
am 
> doing before trying the switch to 2.7.2.
> 
> Thanks.
> 
> - Dave Penton -
> 
> 
>  
> 
> Your use of Yahoo! Groups is subject to 
http://docs.yahoo.com/info/terms/

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 