[antlr-interest] Semantic Predicates in a Lexer

"Paul Bouché (NSN)" paul.bouche at nsn.com
Fri Mar 20 16:10:08 PDT 2009


Hi Gavin,
> Given the input "3", both NAME and NUMBER are viable output rules 
> since both consume the exact same input.  ANTLR should therefore 
> choose whichever one is listed first (which ought to be NUMBER in this 
> case, unless you've listed the rules out-of-order).
Yes it should, but since there is a gated sem pred for NAME it does not. 
The redundant gated sem pred at the NUMBER rule fixed this problem in my 
actual work, but is cumbersome to add to all products NAME pertains to. 
So no ANTLR does not follow the token which is defined first rule when 
there are gated semantic predicates present - bug or feature? If feature 
I'd like to know why.
>   If you don't want to be at the mercy of this sort of thing then you 
> should modify the NAME rule so that a NAME is not permitted to begin 
> with a digit.
I could do that, but why? NAMEs shall begin with a digit, i.e. '3a' is a 
name, and it is not ambigious. This works fine without sem preds.
> But given your example input and what you've said thus far, I think 
> you're overcomplicating things.
Really?
>   If there are no whitespace limitations between the name fragments 
> and the colon (ie. whitespace is permitted and ignored), then what you 
> really should do is to remove the NAME rule entirely and just have the 
> lexer emit SIMPLENAME, COLON, and NUMBER.  Then up in the parser you 
> can define a 'name' rule that recognises SIMPLENAME COLON SIMPLENAME 
> (or whatever) as a single logical unit.
Yeah possible, this is what I do in the case where I want it. Did I 
mention that I solved my problem by emitting more than one token in the 
special case of names without colons? But to do this in all cases would 
be a problem because I'd like to add them as one token to the AST, so 
doing that would have meant more rewrite work in the AST parser.

Just good to know that gated semantic predicates in lexer rules should 
be met with caution ;-)

BR,
Paul


-- 
Paul Bouché
Voice: +49 30 590080-1284
 
Nokia Siemens Networks GmbH & Co. KG, An den Treptowers 1, 12435 Berlin, Germany
Sitz der Gesellschaft: München / Registered office: Munich
Registergericht: München / Commercial registry: Munich, HRA 88537
WEEE-Reg.-Nr.: DE 52984304

Persönlich haftende Gesellschafterin / General Partner: Nokia Siemens Networks Management GmbH
Geschäftsleitung / Board of Directors: Lydia Sommer, Olaf Horsthemke
Vorsitzender des Aufsichtsrats / Chairman of supervisory board: Lauri Kivinen
Sitz der Gesellschaft: München / Registered office: Munich
Registergericht: München / Commercial registry: Munich, HRB 163416



More information about the antlr-interest mailing list