[antlr-interest] Lexer alt does not generate corresponding token?

Gerald Rosenberg gerald at certiv.net
Sun Oct 21 16:50:03 PDT 2012

Interesting.  This appears to generate correctly with COMMENT_BLOCK and 
COMMENT_LINE declared as tokens.

     | SL_COMMENT -> channel(HIDDEN), type(COMMENT_LINE)

Curious to know if it runs afoul of the same issue.

On 10/21/2012 4:09 PM, Terence Parr wrote:
> yep, we did that on purpose. If you ever set a token type with a lexer command, is very risky to reference the surrounding lexer rule in a parser.
> @Sam: can remember exactly how this a bit us?
> Ter
> On Oct 21, 2012, at 2:23 PM, Gerald Rosenberg wrote:
>> When generating a lexer containing the rule
>>     : SL_COMMENT (SL_COMMENT)+ -> channel(HIDDEN), type(COMMENT_BLOCK)
>>     | SL_COMMENT -> channel(HIDDEN)
>>     ;
>> no 'COMMENT_LINE' token is produced in the associated token vocab. Splitting the alts into separate rules produces the expected set of tokens.  Bug or am I missing something?
>> Using v4b2.  Test lexer attached.
>> <TestLexer.g4>
>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
>> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address

More information about the antlr-interest mailing list