[antlr-interest] Whats the diff between 'protected' and 'Token.SKIP'

[Monty Zukowski]

Prashant Deva wrote:
>>protected rules do not produce tokens--they must be called by public rules
>>which do produce tokens.
> 
> Well it seems the token doesn't get created when you set the toke type
> as skip , too.
> Here is the code from a generated lexer -
> if ( _createToken && _token==null && _ttype!=Token.SKIP ) {
> 			_token = makeToken(_ttype);
> 			_token.setText(new String(text.getBuffer(), _begin, text.length()-_begin));
> 		}
> 
> So as you can see , in both the cases the token doesn't get created.
> Then how are they different.
> 
> PRASHANT
> 
> 
> On Thu, 17 Mar 2005 21:33:47 -0800, Loring Craymer
> <Loring.G.Craymer at jpl.nasa.gov> wrote:
> 
>>protected rules do not produce tokens--they must be called by public rules
>>which do produce tokens.
>>
>>--Loring
>>
>>
>>>-----Original Message-----
>>>From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
>>>bounces at antlr.org] On Behalf Of Prashant Deva
>>>Sent: Thursday, March 17, 2005 8:44 PM
>>>To: antlr-interest at antlr.org
>>>Subject: [antlr-interest] Whats the diff between 'protected' and
>>>'Token.SKIP'
>>>
>>>Whats the diff between marking a token as 'SKIP' and marking it as
>>>'protected'.
>>>Eg-
>>>Whats the diff between the 2 rules -
>>>WS   : ('\t' | '\r' | ' ') {_ttype=Token.SKIP;} ;
called by nextToken() and doesn't produce a token.

>>>
>>>protected WS   : ('\t' | '\r' | ' ') ;
not called by nextToken() directly -- must be called by another rule to 
be used.


The difference is this--protected rules don't get included in the ANTLR 
synthesized nextToken() rule.  That is the entry point for the lexer to 
produce tokens.


Monty