[antlr-interest] Re: ANTLR and runtime token assigning - URGENT

[antlrlist]

> 1. How can I set "SEPARATOR" token value from my grammar file in
produced CSVLexer/CSVParser?
> I mean, I wrote class that uses this Parser/Lexer but I want the
class' user to have ability to set SEPARATOR to ';'
> instead of ",". How is it possible to define variable SEPARATOR in
Parser/Lexer that will be used to process input
> without running antlr.Tool again?

The only way I could think about is accepting both "," and ";", like this:

SEPARATOR : ';' | ',';

Now the lexer recognizes both entries like SEPARATOR at the same time.
If you don't want to recognize them simultaneously use a class member:

class CSVLexer extends Lexer;
...
{
   protected char sepChar = ',';
   ... // your gets and sets here
}

SEPARATOR : { sepChar==',' }? ','
          | { sepChar==';' }? ';'
          ;

> I wonder why ANTLR hard-code tokens instead of exposing them in
easy-to-modify variables...

I had the same thoughts until I begun to understand the code that
ANTLR generates, and how it is generated. The idea is that you can't
generate all those "switchs" if you don't have fixed values...

> 2. In my grammar I tried to put
> RECORD  : (~('"'|SEPARATOR|'\r'|'\n'))+;
> instead of
> RECORD  : (~('"'|','|'\r'|'\n'))+;
> 
> but antlr.Tool throws error:
> csv.g:119:13: This subrule cannot be inverted.  Only subrules of the
form:
>      (T1|T2|T3...) or
>      ('c1'|'c2'|'c3'...)
>  may be inverted (ranges are also allowed).
>  Exiting due to errors.
> 
> How can I do that?
> 

If you're going to have ';' or ',' as SEPARATORS, then write:

RECORD  : (~('"'|','|';'|'\r'|'\n'))+;

Sorry, but you just can't have such a "customizable separator
character". You have to specify the character set you're going to use
on each rule; if not, then ANTLR just won't be able to generate the
right code. The analizer's code is based on the "constantness" of your
tokens.

If you want to have such a customizable analizer, you'll have to
create it manually; A lexer class only needs to implement the
TokenStream interface.

Cheers!


Enrique


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/