[antlr-interest] Where is the EOF documentation?

George S. Cowan cowang at comcast.net
Mon Jan 5 09:48:34 PST 2009


Is there any documentation for the special EOF token? 
 
When the parser reaches the end of file, special things seem to happen. But
I can't find anywhere this is documented. I'm in the middle of my second
pass through the Definitive ANTLR Reference, and I don't recall it occurring
in any of the grammars there, or in fact, being mentioned at all. It isn't
in the index either. 
 
I have seen it mentioned as a magic fix for certain grammars on the website,
e.g.,  <http://www.antlr.org/wiki/pages/viewpage.action?pageId=4554943>
http://www.antlr.org/wiki/pages/viewpage.action?pageId=4554943. 
 
In a search from the ANTLR home page, I found several email list entries
that mentioned EOF frustrations and solutions, but I am still not clear
about when it is needed and when it is not. (When trying to find emails from
a search on the home page, watch out for the "" port in the URLS, it
must be manually removed in order to get to the emails' new location.)
 
Here are the the facts that I have gleaned from reading and experiments:
 
1. EOF is not usually required at the end of your top-level rule in a
grammar, but sometimes it is. When is not clear.
 
2. Some rules cannot be unit tested because directly calling them against a
valid input stream puts an EOF at the end of the input steam and some rules
trip over the EOF. When is not clear.
 
3. EOF is only available in the parser, not the lexer. So, for instance, we
can't make it part of whitespace, or make it ok for a line comment to end
with an EOF as well as a newline.
 
 
I begin to suspect that EOF is not truly a token on the token steam, but
just something to use in the parser when certain special undocumented
handling is wanted. I also suspect that the effect of an end of file is
different depending on DFAs, hoisting, and maybe other things not seen in
the grammar, but going on in the implementation. 
 
A satisfactory response would be, "How true, but a future release will clean
up some of this, and better to wait and document that."
 
Frustrated, but hoping for clarity, 
George
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20090105/d99e7215/attachment.html 


More information about the antlr-interest mailing list