[antlr-interest] lexing tips needed... (or, could I use predicate in lexer rule?)

Lloyd Dupont ld at galador.net
Sun Jul 15 04:09:46 PDT 2007

MessageInteresting idea.. I am thinking about it!
  ----- Original Message ----- 
  From: Micheal J 
  To: antlr-interest at antlr.org 
  Sent: Sunday, July 15, 2007 11:29 AM
  Subject: Re: [antlr-interest] lexing tips needed... (or,could I use predicate in lexer rule?)

  You may be trying to do too much in the lexer Lloyd. Seems you can simply change the lexer to emit say NUM_FLOAT_DOT for floats that end with the decimal point. The lexer seems to handle your other NUM_FLOAT examples just fine. 

  You can then deal with the NUM_FLOAT_DOT ID sequence in the parser.

    :  DIGITS 
            '.' { $type = NUM_FLOAT_DOT; } 
            ( fraction_digits=DIGITS { $type = NUM_FLOAT; } )? 
            ( exponent_part=EXPONENT_PART { $type = NUM_FLOAT; } )? 
            ( float_suffix=FLOAT_TYPE_SUFFIX { $type = NUM_FLOAT; } )?

    | ..................... other match

  The best way to contact me is via the list/forum. My time is very limited. 

    -----Original Message-----
    From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-bounces at antlr.org] On Behalf Of Lloyd Dupont
    Sent: 15 July 2007 02:02
    To: antlr-interest at antlr.org
    Subject: [antlr-interest] lexing tips needed... (or,could I use predicate in lexer rule?)

    For my parser I reaped of the lexer rules from the Java exemple.
    Particularly you've got this lexer token:
     | ............ other match

    Now there is a problem with that!....
    in the language I target everything is an object.

    "2."  is NUM_FLOAT["2."] 
    "2.1"  is NUM_FLOAT["2.1"]
    "2.ToString()" is NUM_INT["2"] DOT["."] ID["ToString"] LPAREN["("] RPAREN[")]

    So I was trying to disambiguate the lexer with a construct like that (trying predicate in the lexer):
     : (
      | ( DIGITS '.' ID )=> DIGITS
      |    ..................... other match

    but this doesn't seem to work,
    1. it told me the NUM_INT rule is now not accessible anymore (I guess the NUM_FLOAT rule absorbs it with my second alternative)
    2. but more importantly an input such as "2.a" return "2." followed by MismatchTokenException.

    can I use predicate in lexer rule?
    how could I disambiguate the NUM_FLOAT lexer rule?
    I.e. being both able to read "2.3" (NUM_FLOAT) and "2.ToString()" (NUM_INT DOT ID LPAREN RPAREN)?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20070715/7aeca1e9/attachment.html 

More information about the antlr-interest mailing list