[antlr-interest] fragment: simple (or naive) usage does not work

[Shmuel Siegel]

Works for me in the Antlrworks interpreter. Version 1.0b9
It parses all five lines. And produces a parse tree. Zip of the parse
tree is attached.

I used a combined grammar.

grammar boo;

INT          : 'int' ;
SEMI         : ';' ;
WS           :  (  ' '| '\t'| '\r' | '\n' )+ {$channel=HIDDEN;} ;

IDENTIFIER   :
    ('a'..'z'|'A'..'Z'|'_')+ ;

NUMBER : DIGIT+ (BASE ZNUM+)? ;
fragment ZNUM : DIGIT|'z'|'Z' ;
fragment BASE : 'b' | 'h';
fragment DIGIT : '0'..'9';


source_text :
   { System.out.println("Weird lexer"); }
   int_defs+
   numbers+
   ;

int_defs :
   INT            { System.out.print("int "); }
   id=IDENTIFIER  { System.out.print($id.text); }
   SEMI           { System.out.println(";"); }
   ;

numbers :
   n=NUMBER         { System.out.println($n.text); }
   ;



-----Original Message-----
From: antlr-interest-bounces at antlr.org
[mailto:antlr-interest-bounces at antlr.org] On Behalf Of Martin d'Anjou
Sent: Tuesday, February 27, 2007 9:27 PM
To: antlr-interest at antlr.org
Subject: [antlr-interest] fragment: simple (or naive) usage does not
work

Hi,

This is my input:
int id;
int int_id;
int _int_id;
45b32
6h87z

I have to parse those pesky numbers at the botom. So I wrote the
following 
lexer:

lexer grammar DUMMY_Lexer;
INT          : 'int' ;
SEMI         : ';' ;
WS           :  (  ' '| '\t'| '\r' | '\n' )+ {$channel=HIDDEN;} ;

IDENTIFIER   :
    ('a'..'z'|'A'..'Z'|'_')+ ;

NUMBER : DIGIT+ (BASE ZNUM+)? ;
fragment ZNUM : DIGIT|'z'|'Z' ;
fragment BASE : 'b' | 'h';
fragment DIGIT : '0'..'9';

And of course the parser:

parser grammar DUMMY_Parser;
options {
   tokenVocab=DUMMY_Lexer;
}

source_text :
   { System.out.println("Weird lexer"); }
   int_defs+
   numbers+
   ;

int_defs :
   INT            { System.out.print("int "); }
   id=IDENTIFIER  { System.out.print($id.text); }
   SEMI           { System.out.println(";"); }
   ;

numbers :
   n=NUMBER         { System.out.println($n.text); }
   ;


Alas, I get:
line 4:0 required (...)+ loop did not match anything at input '45b32'

If I move ZNUM inside NUMBER, like this:

NUMBER : DIGIT+ (BASE (DIGIT|'z'|'Z')+)? ;

then it works. What's up with fragment lexer rules?

Thanks,
Martin

-------------- next part --------------
A non-text attachment was scrubbed...
Name: boo.zip
Type: application/x-zip-compressed
Size: 6791 bytes
Desc: boo.zip
Url : http://www.antlr.org/pipermail/antlr-interest/attachments/20070228/b99b6e39/attachment-0001.bin