[antlr-interest] Whats the diff between 'protected' and'Token.SKIP'

Nigel Sheridan-Smith nbsherid at secsme.org.au
Thu Mar 17 23:45:45 PST 2005


> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> bounces at antlr.org] On Behalf Of Prashant Deva
> Sent: Friday, 18 March 2005 5:36 PM
> To: antlr-interest at antlr.org
> Subject: Re: [antlr-interest] Whats the diff between 'protected'
> and'Token.SKIP'
> 
> > protected rules do not produce tokens--they must be called by public
> rules
> > which do produce tokens.
> Well it seems the token doesn't get created when you set the toke type
> as skip , too.
> Here is the code from a generated lexer -
> if ( _createToken && _token==null && _ttype!=Token.SKIP ) {
> 			_token = makeToken(_ttype);
> 			_token.setText(new String(text.getBuffer(), _begin,
> text.length()-_begin));
> 		}
> 
> So as you can see , in both the cases the token doesn't get created.
> Then how are they different.
> 


Both protected and non-protected rules can lead to the creation of tokens.
However, protected rules do not create tokens of their own, but contribute
to other tokens.

e.g.

A: 'c' B 'd';
protected B: 'z';

When the lexer reads the input stream, it constructs an "A" token if it sees
'c' followed by 'z' followed by 'd'. However, it will never construct a "B"
token on its own, consisting of only the 'z', because it is a protected
rule.

Setting "skip" on a (constructed) token prevents it from being forwarded on
to the parser. Consequently, it is often used for comments and white-space,
since these are irrelevant to many parsers. I have no idea what setting skip
on a protected rule does, though. Probably just changes the skip flag on the
encapsulating rule(s).

Nigel

--
Nigel Sheridan-Smith
PhD research student

Faculty of Engineering
University of Technology, Sydney
Phone: 02 9514 7946
Fax: 02 9514 2435




More information about the antlr-interest mailing list