[antlr-interest] Antlr dropping tokens?

[Terence Parr]

Hi Jon,

$a* would allow $a to be an empty list whereas $a+ with throw an  
exception if the list were in.  In general you must match the  
cardinality on the right side with cardinality on the left :)
Ter
On Jan 22, 2008, at 10:13 AM, Jon Schewe wrote:

> What is difference between $a* and $a+?
>
> Terence Parr wrote:
>> same error as last person with this problem a day or so ago.
>>
>> use $a+ or $a*
>>
>> Ter
>> On Jan 21, 2008, at 7:27 AM, Jon Schewe wrote:
>>
>>> Here's a piece of my grammar that I'm testing with junit.  It  
>>> appears
>>> that some tokens are getting dropped.
>>> The input is this:
>>> b[1 ... (param1 - 5 - 1) * 4]
>>>
>>> The resulting tree is this:
>>> (b (SUM (PRODUCT (NUMBER 1))) (SUM (PRODUCT (SUM (PRODUCT param1)
>>> (PRODUCT (NUMBER -1) (NUMBER 5)) (PRODUCT (NUMBER -1) (NUMBER  
>>> 1))))))
>>>
>>> Where did the "* 4" go?  I didn't think ANTLR could drop tokens like
>>> that.  This is using antlr 3.0.1.
>>>
>>> The grammar fragment is below:
>>> /**
>>> * Name used in a variable declaration.
>>> */
>>> nameDecl :
>>> IDENT -> IDENT
>>> | IDENT LBRACK lb=subscriptAddExpr[false] ELLIPSIS
>>> ub=subscriptAddExpr[false] RBRACK -> ^(IDENT $lb $ub)
>>> ;
>>> subscriptAddExpr[boolean negate]
>>> :
>>> subscriptMultExpr[negate] (PLUS subscriptMultExpr[negate] | MINUS
>>> subscriptMultExpr[!negate])* -> ^(SUM subscriptMultExpr+)
>>>
>>> ;
>>>
>>> /**
>>> * @param negate if true, negate all expressions by multiplying by -1
>>> */
>>> subscriptMultExpr[boolean negate]
>>> :
>>> a+=subscriptAtom (a+=subscriptMultHelp)* -> {negate}? ^(PRODUCT
>>> ^(NUMBER NUM_INT["-1"]) $a)
>>>                                          ->           ^(PRODUCT $a)
>>> ;
>>>
>>> subscriptMultHelp : PRODUCT subscriptAtom -> subscriptAtom ;
>>>
>>> /**
>>> * Base type that can be inside a subscript.
>>> */
>>> subscriptAtom
>>> :
>>> IDENT
>>> | numint
>>> | subscriptParExpression
>>> ;
>>>
>>> subscriptParExpression
>>> :
>>> LPAREN subscriptAddExpr[false] RPAREN -> subscriptAddExpr
>>> ;
>>>
>>>
>>> /**
>>> * A finite integer number.  May be negative.
>>> */
>>> numint
>>> :
>>> MINUS NUM_INT -> ^(NUMBER ^(MINUS NUM_INT))
>>> | NUM_INT -> ^(NUMBER NUM_INT)
>>> ;
>>>
>>> // ----------- Lexer ---------------------
>>> // Operators
>>> LPAREN          :   '('     ;
>>> RPAREN          :   ')'     ;
>>> LBRACK          :   '['     ;
>>> RBRACK          :   ']'     ;
>>> ELLIPSIS        :   '...'   ;
>>> EQ              :   '='     ;
>>> MINUS           :   '-'     ;
>>> PLUS            :   '+'     ;
>>> SEMI            :   ';'     ;
>>> LCURLY          :   '{'     ;
>>> RCURLY          :   '}'     ;
>>> LE              :   '<='    ;
>>> COLON           :   ':'     ;
>>> COMMA           :   ','     ;
>>> PRODUCT         :   '*'     ;
>>>
>>> // Keywords
>>> IN              :   'in'    ;
>>>
>>> // Functions
>>> SUMMATION       :   'SUM'   ;
>>> LOOP            :   'LOOP'  ;
>>> TAN             :   'tan'   ;
>>> COS             :   'cos'   ;
>>> SIN             :   'sin'   ;
>>> LOG             :   'log'   ;
>>> LOG10           :   'log10' ;
>>> EXP             :   'exp'   ;
>>> POW             :   'pow'   ;
>>>
>>>
>>> /** Single-line comments */
>>> SL_COMMENT
>>> : '//' ~( '\n'|'\r' )* '\r'? '\n' { $channel=HIDDEN; }
>>>   ;
>>>
>>> /** multiple-line comments */
>>> ML_COMMENT
>>>   :    '/*'
>>>       ( options {greedy=false;} : . )*
>>>       '*/'
>>>       {$channel=HIDDEN;}
>>>   ;
>>>
>>> IDENT :
>>> ('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'_'|'0'..'9')*
>>> ;
>>>
>>> // a numeric literal
>>> NUM_INT
>>> : ('0'..'9')+ EXPONENT?
>>> ;
>>>
>>> NUM_FLOAT
>>>   :     DIGITS '.' DIGITS? EXPONENT?
>>>   | '.' DIGITS EXPONENT?
>>>   ;
>>>
>>> fragment
>>> DIGITS : ('0'..'9')+ ;
>>>
>>>
>>> // a protected method to assist in matching floating point numbers
>>> fragment
>>> EXPONENT
>>> : ('e'|'E') ('+'|'-')? ('0'..'9')+
>>> ;
>>>
>>> // Whitespace -- ignored
>>> WS    :    (    ' '
>>>       |    '\t'
>>>       |    '\f'
>>>           // handle newlines
>>>       |    (    '\r\n'  // Evil DOS
>>>           |    '\n'    // Unix (the right way)
>>>           )
>>>       )+
>>>       { $channel=HIDDEN; }
>>>   ;
>>>
>>>
>>>
>>> -- 
>>> Jon Schewe | http://mtu.net/~jpschewe
>>> If you see an attachment named signature.asc, this is my digital
>>> signature.
>>> See http://www.gnupg.org for more information.
>>>
>>> For I am convinced that neither death nor life, neither angels
>>> nor demons, neither the present nor the future, nor any
>>> powers, neither height nor depth, nor anything else in all
>>> creation, will be able to separate us from the love of God that
>>> is in Christ Jesus our Lord. - Romans 8:38-39
>>>
>>>
>
> -- 
> Jon Schewe | http://mtu.net/~jpschewe
> If you see an attachment named signature.asc, this is my digital
> signature.
> See http://www.gnupg.org for more information.
>
> For I am convinced that neither death nor life, neither angels nor  
> demons, neither the present nor the future, nor any powers, neither  
> height nor depth, nor anything else in all creation, will be able to  
> separate us from the love of God that is in Christ Jesus our Lord. -  
> Romans 8:38-39
>