[antlr-interest] insert token in lexer?

shmuel siegel antlr at shmuelhome.mine.nu
Thu Jun 1 00:23:02 PDT 2006


Kay Roepke wrote:
> 
> On 30. May 2006, at 20:30 Uhr, Putrycz, Erik wrote:
> 
>> Is it possible to insert a token in a lexer rule?
>>
>> On a specific rule, I'd like to change the type of the token with a
>> $setType(...) and I'd also would like to insert another token.
>>
>> I know I could do that with a TokenStream but I'd rather avoid writing
>> that in Java if possible...
> 
> I'm afraid that it is not possible without doing Java coding.
> nextToken() always returns the token in _returnToken. So there can be 
> only one token at a time.
> Surely you could provide some nextToken replacement in subclass that 
> does it differently, and provide some
> hooks for adding more than one token. Then let nextToken loop over the 
> accumulated tokens until it runs
> out and only then let it call another lexer rule. That way you could add 
> as many tokens as you want.
> 
> But question is, why do you want to do that? It sounds a bit far out ;)
> 
> HTH,
> 
> -k
> 
It is one way of implementing javascript's virtual semicolon


-- 
No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.1.394 / Virus Database: 268.8.0/352 - Release Date: 5/30/2006



-- 
No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.1.394 / Virus Database: 268.8.0/352 - Release Date: 5/30/2006



More information about the antlr-interest mailing list