[antlr-interest] Optional tokens

[shmuel siegel]

hammett wrote:
> ----- Original Message ----- From: "John B. Brodie" <jbb at acm.org>
> 
>> Do you really expect that:
>>
>>    x ; , ; y ; = ; 1 ; , ; 2
>>
>> would parse to the same AST as:
>>
>>    x, y = 1, 2
> 
> 
> Call me a dreamer! :-D
> 
>> you did say that SEMI is to be totally ignored anywhere except at the
>> end of a statement...
>>
>> and so just where does a statement end anyway? (answering this may
>> lead you to the answer of your original question)
> 
> 
> I'm more concerned with NEWLINE... answering your question, I did:
> 
> statement:
>    :
>    (
>      declarationStmt
>      |
>      compoundStmt
>      |
>      expressionStmt
>    )
>    (SEMI!|NEWLINE!)
>    ;
> 
> But having NEWLINE as a real token obligates me to deal with it 
> everywhere, that's the bottom line problem...

I will repeat John's question. If a NEWLINE can occur anywhere, where 
does a statement end? If it can only occur (and must occur) at a 
statement end then defactoring and putting it at the end of each of your 
statement types will should solve your problem.