[antlr-interest] Lexing an interesting syntax

[Ola Bini]

Jim Idle wrote:
>> -----Original Message-----
>> From: Ola Bini [mailto:ola.bini at gmail.com]
>> Sent: Wednesday, January 02, 2008 8:13 AM
>> To: antlr-interest at antlr.org
>> Subject: [antlr-interest] Lexing an interesting syntax
>>
>> Hi,
>>
>> Just started work on a lexer for an Io-based language. I want the 
>>     
> lexing
>   
>> to handle the same constructs as Io, and mostly it's really easy. I 
>>     
> hit
>   
>> one little snag though. I have a solution, but it's incredibly ugly. 
>>     
> So
>   
>> I'm wondering how this can be done in the Antlr way.
>>
>> To make it easy, the lexing is only on identifiers, where any
>> combination of the letter "s" and ":" is valid, "=", ":=" and "::=" is
>> valid. That's all.
>> With these constraints, I need:
>>
>> * "s:" to lex into "s:"
>> * "s:=" to lex into "s" and ":="
>> * "s::=" to lex into "s:" and ":="
>> * "s::::=" to lex into "s:::" and ":="
>>     
>
> Don't try to do so much of this in the lexer is the answer. Allow a 
> separate token, COLON and either make the operator ":=" in the lexer or 
> perhaps even parse that in the parser. But adding a bit more to your 
> requirements by guessing ;-), then for the input:
>   
Thanks. That might actually be the best solution. It's a bit ugly, since 
Identifier tokens would be good to always be able to rely on from the 
lexer, but it seems like a much better approach than mine, at least.

Cheers

-- 
 Ola Bini (http://ola-bini.blogspot.com) 
 JRuby Core Developer
 Developer, ThoughtWorks Studios (http://studios.thoughtworks.com)
 Practical JRuby on Rails (http://apress.com/book/view/9781590598818)

 "Yields falsehood when quined" yields falsehood when quined.