[antlr-interest] Re: Antlr 3.0 spaces between tokens

[John D. Mitchell]

>>>>> "matthew" == matthew ford <Matthew.Ford at forward.com.au> writes:
[...]

> Perhaps for Antlr 3.0 we can have a better means of handling white space.

Did you read Ter's blog entry on lexing from last Sunday?

> Antlr provides an ignore whitespace capability that is appealing WS : ( '
> ' | '\t' | '\n' { newline(); } | '\r' )+ { $setType(Token.SKIP); } ;but
> every time I try and use it I come across a situation where I really
> want/need the white space in the parser.

> So I end up having the lexer pass it back to the parser.  (or have switch
> in the lexer that the parser uses to control the return of whitespace.  I
> know this is a no-no but it has worked for me in some cases)

> The parser usually only needs to know about the whitespace in a few rules
> but now has (WS)* all over the place to handle whitespace every where.

> Basically what I would like to have the lexer pass all the whitespace
> back to the parser) and then in the parser be able to say a) for this
> rule ignore white space.  or b) for this rule whitespace is important

In fact, the v3 lexer will pass on everything that's not explicitly elided
by the user-specified filter rule.

In the lexer, you can assign different tokens to different "channels".

In the parser, you set a default set of channels to pay attention to.

Ter's blog is unclear on this point but the basic idea is that there'll be
some syntax in the parser so that you can have specific parser rules pay
attention to more/less/different channels than the default set.  This will
handle cases like you mention where WS is significant in certain rules and
ignored everywhere else.

Hope this helps,
		John


 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
    antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/