[antlr-interest] lexing tips needed... (or, could I use predicate in lexer rule?)

[Lloyd Dupont]

MessageInteresting idea.. I am thinking about it!
thanks....
  ----- Original Message ----- 
  From: Micheal J 
  To: antlr-interest at antlr.org 
  Sent: Sunday, July 15, 2007 11:29 AM
  Subject: Re: [antlr-interest] lexing tips needed... (or,could I use predicate in lexer rule?)


  You may be trying to do too much in the lexer Lloyd. Seems you can simply change the lexer to emit say NUM_FLOAT_DOT for floats that end with the decimal point. The lexer seems to handle your other NUM_FLOAT examples just fine. 

  You can then deal with the NUM_FLOAT_DOT ID sequence in the parser.

  NUM_INT
    :  DIGITS 
       (
            '.' { $type = NUM_FLOAT_DOT; } 
            ( fraction_digits=DIGITS { $type = NUM_FLOAT; } )? 
            ( exponent_part=EXPONENT_PART { $type = NUM_FLOAT; } )? 
            ( float_suffix=FLOAT_TYPE_SUFFIX { $type = NUM_FLOAT; } )?
       )?

    | ..................... other match
  ;


  Micheal
  -----------------------
  The best way to contact me is via the list/forum. My time is very limited. 

    -----Original Message-----
    From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-bounces at antlr.org] On Behalf Of Lloyd Dupont
    Sent: 15 July 2007 02:02
    To: antlr-interest at antlr.org
    Subject: [antlr-interest] lexing tips needed... (or,could I use predicate in lexer rule?)


    For my parser I reaped of the lexer rules from the Java exemple.
    Particularly you've got this lexer token:
    NUM_FLOAT
     : DIGITS '.' (DIGITS)? (EXPONENT_PART)? (FLOAT_TYPE_SUFFIX)?
     | ............ other match
     ;

    Now there is a problem with that!....
    in the language I target everything is an object.

    so:
    "2."  is NUM_FLOAT["2."] 
    "2.1"  is NUM_FLOAT["2.1"]
    "2.ToString()" is NUM_INT["2"] DOT["."] ID["ToString"] LPAREN["("] RPAREN[")]

    So I was trying to disambiguate the lexer with a construct like that (trying predicate in the lexer):
    NUM_FLOAT
     : (
       ( DIGITS '.' EXPONENT_PART )=> DIGITS '.' (EXPONENT_PART) (FLOAT_TYPE_SUFFIX)?
      | ( DIGITS '.' ID )=> DIGITS
      | DIGITS '.' (DIGITS)? (EXPONENT_PART)? (FLOAT_TYPE_SUFFIX)?
      )
      |    ..................... other match
    ;

    but this doesn't seem to work,
    1. it told me the NUM_INT rule is now not accessible anymore (I guess the NUM_FLOAT rule absorbs it with my second alternative)
    2. but more importantly an input such as "2.a" return "2." followed by MismatchTokenException.

    can I use predicate in lexer rule?
    how could I disambiguate the NUM_FLOAT lexer rule?
    I.e. being both able to read "2.3" (NUM_FLOAT) and "2.ToString()" (NUM_INT DOT ID LPAREN RPAREN)?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20070715/7aeca1e9/attachment.html