[antlr-interest] Too many uses for escape character giving me lexer troubles.

[Jeremy D. Frens]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Eric Deplagne wrote:
> On Wed, 14 Mar 2007 21:37:07 -0400, Jeremy D. Frens wrote:
>> Terence Parr wrote:
>>> On Mar 13, 2007, at 6:51 PM, Jeremy D. Frens wrote:
>>>> In my language (http://nolatte.sf.net/), the backslash character is the
>>>> escape character, and it gets used for (at least) two different tasks.
>>>> Here's a stripped down grammar:
>>>>
>>>> atom        :  WORD | IDENTIFIER ;
>>>> WORD        :  ( ('a'..'z') | ( '\\' '{' ) )+ ;
>>>> IDENTIFIER    :   '\\' ('a'..'z')+ ;
>>>>
>>>> The key is that the backslash gets used for two purposes: as a real
>>>> escape character (to escape '{' in a WORD) and as the beginning of an
>>>> IDENTIFIER.  The problem comes in when my grammar tries to scan and/or
>>>> parse something like this:
>>>>
>>>>   abc\xyz
>>>>
>>>> This should be two tokens: a WORD "abc" and an IDENTIFIER "\xyz".
>>>> However, since the backslash is allowed at all in a WORD, the lexer
>>>> consumes it, and then it gets confused by the 'x'.
>>> try putting ID before WORD
>> Same problem.  Three more observations:
> 
>   I would simply not do that at lexer level.
> 
>   What would the following give ?:
> 
>     atom : word | identifier;
>     word : ( LOWCASE | BACKSLASH OBRACE )+;
>     identifier : BACKSLASH LOWCASE+
>     BACKSLASH : '\\';
>     OBRACE : '{';
>     LOWCASE : 'a'..'z';
> 

In case anyone cares, I opted for this approach.  "word" is recognized
by the parser; it looks for "SYLLABLE"s which might be raw text, escaped
curly braces, or some other strange metacharacters in the language.

word	:	syllable+
		-> ^(WORD syllable+)
	;
fragment
syllable
	:	( SYLLABLE | EQUALS | AMPERSAND )
	;

SYLLABLE can be "raw text" *xor* "\{", "\}", "\\".  (Check out
http://tinyurl.com/3c8dbl , if you're *really* interested in what the
grammar looks like.  It's the whole grammar file as found in my SVN
repository.)

I decided *not* to go with a lexer solution which re-set the type
because it just didn't feel as clean to me.

jdf

- --
* Jeremy D. Frens * Professor, Computer Science * jdfrens at calvin.edu *
         ``I would put exclamation points at the end of each
           of these sentences!  This one!  And that one!''
                          -- Elaine, _Seinfeld_

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGCGtOOcBu2deY79IRAlRgAJ9HnPIAoIazgExfehrXVj4QpwzeswCfRMfz
qogWzzfD5Szg0vkk09rTJcc=
=85qZ
-----END PGP SIGNATURE-----