[antlr-interest] Re: Antlr 3.0 spaces between tokens

[matthew ford]

Sounds interesting.
Although I would need to see more details.  I am not a fan of the token
stream switching that is implemented at the moment.
The interaction with lookahead worries me.
matthew

----- Original Message ----- 
From: "John D. Mitchell" <johnm-antlr at non.net>
To: <antlr-interest at yahoogroups.com>
Sent: Friday, November 12, 2004 3:52 AM
Subject: [antlr-interest] Re: Antlr 3.0 spaces between tokens


>
> >>>>> "matthew" == matthew ford <Matthew.Ford at forward.com.au> writes:
> [...]
>
> > Perhaps for Antlr 3.0 we can have a better means of handling white
space.
>
> Did you read Ter's blog entry on lexing from last Sunday?
>
> > Antlr provides an ignore whitespace capability that is appealing WS :
( '
> > ' | '\t' | '\n' { newline(); } | '\r' )+ { $setType(Token.SKIP); } ;but
> > every time I try and use it I come across a situation where I really
> > want/need the white space in the parser.
>
> > So I end up having the lexer pass it back to the parser.  (or have
switch
> > in the lexer that the parser uses to control the return of whitespace.
I
> > know this is a no-no but it has worked for me in some cases)
>
> > The parser usually only needs to know about the whitespace in a few
rules
> > but now has (WS)* all over the place to handle whitespace every where.
>
> > Basically what I would like to have the lexer pass all the whitespace
> > back to the parser) and then in the parser be able to say a) for this
> > rule ignore white space.  or b) for this rule whitespace is important
>
> In fact, the v3 lexer will pass on everything that's not explicitly elided
> by the user-specified filter rule.
>
> In the lexer, you can assign different tokens to different "channels".
>
> In the parser, you set a default set of channels to pay attention to.
>
> Ter's blog is unclear on this point but the basic idea is that there'll be
> some syntax in the parser so that you can have specific parser rules pay
> attention to more/less/different channels than the default set.  This will
> handle cases like you mention where WS is significant in certain rules and
> ignored everywhere else.
>
> Hope this helps,
> John
>
>
>
> Yahoo! Groups Links
>
>
>
>
>
>
>



 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
    antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/