[antlr-interest] Why ANTLR doesn't check existence of lexical symbols?

[Greg Lindholm]

--- mzukowski at yci.com wrote:
> Parsers use symbols for AST node types as well.  Just because the
> parser
> knows about a symbol doesn't mean the lexer has to generate it.
> 
I don't realize these were used for AST node types. 
But, doesn't Antlr know the difference between an AST node type and
something that it's suppose to match from a token stream?

> I'm having a hard time understanding how someone can mentally add a
> new
> token to a parser rule but not put that token into a lexer.
> 
I don't make coding mistakes on purpose. It's more a problem of typos,
forgetting and just plain mistakes.  The point is, a friendly tools
should warn you of any obvious errors that it is able to detect.

> At the very least these are trivially caught by your unit tests which
> excersize your grammar rules.
> 
IMHO I don't believe the words "trivial", "unit tests" and "grammar"
belong in the same sentence :)   Unless you mean "trival unit tests" or
"trivial grammar" :)  

Greg

> Monty
> 
> -----Original Message-----
> From: Greg Lindholm [mailto:glindholm at yahoo.com] 
> Sent: Wednesday, June 11, 2003 1:25 PM
> To: antlr-interest at yahoogroups.com
> Subject: Re: [antlr-interest] Why ANTLR doesn't check existence of
> lexical
> symbols?
> 
> 
> Hi Ter,
> 
> It seems to me that there are 2 case;
> 1) the lexer is in the same file or
> 2) the parser does an import of the token symbols
> 
> In either case if Antlr encounters undefined tokens in a parser and
> has to
> generate new symbols I think you got a problem that at the very least
> deserves a warning message.
> 
> Cause if the lexer doesn't know about the symbol it's never going to
> create
> a token of that type.
> 
> Or am I missing something?
> 
> 
> --- Terence Parr <parrt at jguru.com> wrote:
> > 
> > On Wednesday, June 11, 2003, at 02:11  AM, Hrvoje Nezic wrote:
> > 
> > > Hi,
> > >
> > > If some lexical symbol is referenced in parser grammar,
> > > but is not actually defined in lexer, ANTLR doesn't generate
> error 
> > > or warning messages, so this can be detected only at runtime on 
> > > parser testing. I find it very inconvenient, because you have to 
> > > check existence of token symbols manually.
> > > Is there any reason why ANTLR behaves like this, and
> > > is there any workaround?
> > 
> > The problem is that ANTLR can be hooked up to any TokenStream
> object.
> >  
> > Further, the lexer may not be defined in the same grammar file.
> > ANTLR 
> > cannot answer this question, I guess is the answer (though not the
> > one 
> > you are hoping for) ;)
> > 
> > Terence
> > --
> > Co-founder, http://www.jguru.com
> > Creator, ANTLR Parser Generator: http://www.antlr.org Co-founder, 
> > http://www.peerscope.com link sharing, pure-n-simple Lecturer in
> Comp. 
> > Sci., University of San Francisco
> > 
> > 
> >  
> > 
> > Your use of Yahoo! Groups is subject to 
> > http://docs.yahoo.com/info/terms/
> > 
> > 
> 
> 
> __________________________________
> Do you Yahoo!?
> Yahoo! Calendar - Free online calendar with sync to Outlook(TM).
> http://calendar.yahoo.com
> 
>  
> 
> Your use of Yahoo! Groups is subject to
> http://docs.yahoo.com/info/terms/ 
> 
> 
>  
> 
> Your use of Yahoo! Groups is subject to
> http://docs.yahoo.com/info/terms/ 
> 
> 


__________________________________
Do you Yahoo!?
Yahoo! Calendar - Free online calendar with sync to Outlook(TM).
http://calendar.yahoo.com

 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/