[antlr-interest] Explanation about EOF needed

Jim Idle jimi at temporal-wave.com
Sun Dec 7 09:07:45 PST 2008


On Sun, 2008-12-07 at 18:35 +0300, `VL wrote:

> Hello.

Having a newline at the end of a file defines a text file in UNIX, not
having one is basically creating a special condition that you have to
rejig your grammar to accommodate. I find that it generally isn't worth
it and just append a newline to the input source so that it will always
have one. This is much simpler.

Jim

> 
> I'm trying to include comments information into my grammar and have
> some problems.
> Below is the minimal example i have:
> 
> ========== Cut =================
> grammar simple;
> 
> // the whole thing: all stuff till the end of file
> prog :   commented_kw+ EOF;
> 
> // Keyword with optional comment before it or just comment or empty line
> commented_kw: COMMENT* varspec* NL;
> 
> // key=val
> varspec: TKN '=' (TKN);
> 
> // token for keys and values
> TKN:  ('a'..'z'|'A'..'Z'| '0'..'9')+;
> 
> // newline separator
> NL : '\n' ;
> 
> // whitespace are thrown away
> WS:     (' '|'\t'| '\r')+ { $channel=HIDDEN; };
> 
> // comment goes till end of line, including it
> COMMENT : (' '|'\t')* ( '#' (~'\n')*  ) NL;
> ========== Cut =================
> 
> Here is the input (generic configuration file):
> 
> ========== Cut =================
> 
> 
> # This is one-line comment (global for file)
> 
> # This is many one-line comments (global)
> # This is many one-line comments (global)
> # This is many one-line comments (global)
> # This is many one-line comments (global)
> 
> # This is comment to var
> var=value
> 
> # This is comment to many vars
> var=value var=value var=value
> 
> var=value
> # After-var comment
> 
> # Pre-Var 1
> # Pre-Var 2
> # Pre-Var 3
> var=value
> # After-Var 1
> # After-Var 2
> 
> 
> # LAST LINE - no newline at the END
> ========== Cut =================
> 
> Everything goes fine, except i'm getting
> MismatchedTokenException(0!=0) on the last line.
> 
> Well, using such grammar it is expected error, since my comment is
> defined to end with NL
> and it can't be found.
> 
> I tried to modify COMMENT lexer rule to end with (NL|EOF). No effect.
> But i have one more
> required NL, in commented_kw rule, i changed it to (NL|EOF), and got
> error about unreachable
> alternative in 'prog' rule. I tried a lot of more variants, trying to
> move COMMENT rule into parser
> level and more, but i can't understand what's going on.
> 
> I'm satisfied with first variant of grammar, but it gives me an error.
> How should i modify it to
> make it work? Where is my error when i tried to achive my goals?
> 
> Thanks.
> 
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20081207/71a224f4/attachment.html 


More information about the antlr-interest mailing list