[antlr-interest] Matching compound keywords in the lexer

Franck Mangin franck102 at yahoo.com
Fri Nov 25 21:33:43 PST 2011


Thanks all for the replies.

As far as I know just the tokens won't work, the parser complains that "no *rule* defines token xxx" - which is why I hadn't tried fragments, I 
assumed I would get the same complaint, but that actually works 
perfectly.

Franck


________________________________
 From: Benjamin S Wolf <jokeserver at gmail.com>
To: Jim Idle <jimi at temporal-wave.com> 
Cc: Bart Kiers <bkiers at gmail.com>; franck102 <franck102 at yahoo.com>; "antlr-interest at antlr.org" <antlr-interest at antlr.org> 
Sent: Friday, November 25, 2011 11:16 PM
Subject: Re: [antlr-interest] Matching compound keywords in the lexer
 
You can do it either way, I guess. I use just the tokens def myself.

On Fri, Nov 25, 2011 at 2:07 PM, Jim Idle <jimi at temporal-wave.com> wrote:
> You don't need the tokens def. just the lexer fragment.
>
> Jim
>
> On Nov 25, 2011, at 12:54, Bart Kiers <bkiers at gmail.com> wrote:
>
>> Hi Franck,
>>
>> On Fri, Nov 25, 2011 at 9:47 PM, franck102 <franck102 at yahoo.com> wrote:
>>
>>> ...
>>
>> containOperator : CONTAINS_TEXT | CONTAINS_MATCH
>>>
>>> CONTAINS_TEXT
>>>       :       'contains' WS+ ( 'match' { $type=CONTAINS_MATCH }
>>>       |                              'text' )
>>>       ;
>>>
>>> // CONTAINS_MATCH:;  // causes "token definitions can never be matched"
>>> error
>>>
>>>
>> Add CONTAINS_MATCH to your @tokens{...} and create an empty fragment rule
>> called CONTAINS_MATCH to silence the warning:
>>
>> -----------------------------
>>
>> tokens {
>>  CONTAINS_MATCH;
>> }
>>
>> ...
>>
>> CONTAINS_TEXT
>>  :  'contains' WS+ ( 'match' CONTAINS_MATCH
>>                    | 'text' )
>>  ;
>>
>> ...
>>
>> fragment CONTAINS_MATCH : ;
>>
>> -----------------------------
>>
>> Regards,
>>
>> Bart.
>>
>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
>> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>


More information about the antlr-interest mailing list