[antlr-interest] Optional tokens

[shmuel siegel]

hammett wrote:

>> I will repeat John's question. If a NEWLINE can occur anywhere, where 
>> does a statement end? If it can only occur (and must occur) at a 
>> statement end then defactoring and putting it at the end of each of 
>> your statement types will should solve your problem.
>
>
> Allow me to rephrase my problem: a NEWLINE is meaningless anywhere but 
> at statements end, which means that it must be ignored as a token 
> anywhere but at statements end.
>
> I believe aspl VBScript grammar solved that nicely. Gonna use their 
> approach.
>
I don't know VBScript but I do know the problems with parsing javascript 
and the solutions are not elegant. In javascript, semicolons are 
optional but do mean EOS if present. EOL is usually  meaningless except 
when occuring between incompatable tokens and then it means semicolon. 
So basically, when the parser doesn't understand what it gets it tells 
the lexer "try again with a semicolon". Unless of course EOL follows 
break, continue, or return in which case it is unconditionally a 
semicolon. Not a nice language.
The simplest way to handle this is for the lexer to know the token pairs 
that are acceptable to the parser and replace EOL by semicolon whenever 
not doing so would create an illegal pair.
I hope this gives you some insight into the problem that you are facing. 
Alternatively, somebody can suggest a clean method for dealing with 
javascripts virtual semicolon.