[antlr-interest] Too many uses for escape character giving me lexer troubles.

[Jeremy D. Frens]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Terence Parr wrote:
> On Mar 13, 2007, at 6:51 PM, Jeremy D. Frens wrote:
>> In my language (http://nolatte.sf.net/), the backslash character is the
>> escape character, and it gets used for (at least) two different tasks.
>> Here's a stripped down grammar:
>>
>> atom        :  WORD | IDENTIFIER ;
>> WORD        :  ( ('a'..'z') | ( '\\' '{' ) )+ ;
>> IDENTIFIER    :   '\\' ('a'..'z')+ ;
>>
>> The key is that the backslash gets used for two purposes: as a real
>> escape character (to escape '{' in a WORD) and as the beginning of an
>> IDENTIFIER.  The problem comes in when my grammar tries to scan and/or
>> parse something like this:
>>
>>   abc\xyz
>>
>> This should be two tokens: a WORD "abc" and an IDENTIFIER "\xyz".
>> However, since the backslash is allowed at all in a WORD, the lexer
>> consumes it, and then it gets confused by the 'x'.
> 
> try putting ID before WORD

Same problem.  Three more observations:

Interpreting in ANTLRWorks as a WORD, I get a MismatchedTokenException
(complaining about getting an 'x' instead of a '{').

Interpreting in ANTLRWorks as an atom, I get what appears to be a valid
AST, although the leaf node had "abc\xyz" in it as if that's the text of
the leaf token.

Running the generated Java code, the lexer actually returns *just* "z"
as a WORD for the "abc\xyz" input.

At first, I was thinking that the problem was one with follow sets, but
now I'm not so sure.  Shouldn't a simple lookahead of k=2 detect when to
stop the current WORD and start an IDENTIFIER?  I originally thought it
might be follow sets because if a follow set only contained single
characters, then "\" becomes ambiguous: it's sometimes part of a WORD,
it sometimes follows a word.  But it occurs to me now that follow sets
don't normally enter into a lexer.

jdf

- --
* Jeremy D. Frens * Professor, Computer Science * jdfrens at calvin.edu *
   ``It just as easily could have gone the other way.''
           -- Chicago Cubs manager Don Zimmer on
               his team's 4-4 record

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFF+KNDOcBu2deY79IRAnB/AKC0/qSCCkbnJ0EHJggYiLRwIUO3pgCcDouC
UdNn3O8HG7Yeowa5Auad2Tw=
=1iMw
-----END PGP SIGNATURE-----