[antlr-interest] Lexing an interesting syntax
Ola Bini
ola.bini at gmail.com
Wed Jan 2 09:14:47 PST 2008
Jim Idle wrote:
>> -----Original Message-----
>> From: Ola Bini [mailto:ola.bini at gmail.com]
>> Sent: Wednesday, January 02, 2008 8:13 AM
>> To: antlr-interest at antlr.org
>> Subject: [antlr-interest] Lexing an interesting syntax
>>
>> Hi,
>>
>> Just started work on a lexer for an Io-based language. I want the
>>
> lexing
>
>> to handle the same constructs as Io, and mostly it's really easy. I
>>
> hit
>
>> one little snag though. I have a solution, but it's incredibly ugly.
>>
> So
>
>> I'm wondering how this can be done in the Antlr way.
>>
>> To make it easy, the lexing is only on identifiers, where any
>> combination of the letter "s" and ":" is valid, "=", ":=" and "::=" is
>> valid. That's all.
>> With these constraints, I need:
>>
>> * "s:" to lex into "s:"
>> * "s:=" to lex into "s" and ":="
>> * "s::=" to lex into "s:" and ":="
>> * "s::::=" to lex into "s:::" and ":="
>>
>
> Don't try to do so much of this in the lexer is the answer. Allow a
> separate token, COLON and either make the operator ":=" in the lexer or
> perhaps even parse that in the parser. But adding a bit more to your
> requirements by guessing ;-), then for the input:
>
Thanks. That might actually be the best solution. It's a bit ugly, since
Identifier tokens would be good to always be able to rely on from the
lexer, but it seems like a much better approach than mine, at least.
Cheers
--
Ola Bini (http://ola-bini.blogspot.com)
JRuby Core Developer
Developer, ThoughtWorks Studios (http://studios.thoughtworks.com)
Practical JRuby on Rails (http://apress.com/book/view/9781590598818)
"Yields falsehood when quined" yields falsehood when quined.
More information about the antlr-interest
mailing list