[antlr-interest] Optional tokens

Ric Klaren ric.klaren at gmail.com
Mon May 23 04:07:30 PDT 2005


Hi,

On 5/23/05, hammett <hammett at uol.com.br> wrote:
> I'm struggling with my grammar where I need to express that SEMI or NEWLINE
> is required to signal the end of my statements while should be totally
> ignored elsewhere.
> 
> So, a possible AST could be
> 
> NEWLINE
> TypeDefinition
> NEWLINE
>   MethodDefintion
>   NEWLINE
>     Statements either SEMI or NEWLINE must be used to end each statement
>       But NEWLINEs on the code should be ignored

Bit of a fundamental question do you need/want the newlines in your
ast? In general you'd want to reduce complexity of the stuff you're
parsing the further you go down in your 'parser-stack'. E.g. in lexer
and parser you deal with hopefully most of the irregularity of your
language and spit out a AST that's more geared towards easily
processing the input. Sometimes you'd need extra tree transformations
to come to a really regular AST that you can process very easily
(hardly any ambiguities by introducing imaginary tokens, reordering
etc..)

Unless you of course do some transformations that require keeping
around the whitespace and such, then disregard my comment :)

Cheers,

Ric


More information about the antlr-interest mailing list