[antlr-interest] TokenStreamRewriteEngine question

[Terence Parr]

On Mar 12, 2006, at 8:51 AM, Scott Amort wrote:

> Hi All,
>
> I am using a TokenStreamRewriteEngine to discard unwanted  
> whitespace and
> comments, while still retaining the original file contents for  
> debug and
> error messages.  However, I have noticed that within my lexer, I
> 'prediscard' a number of other characters, such as double-quotes,
> backslashes, etc.  These latter types are necessary to define certain
> tokens, but I don't want them actually passed on to the parser, so I
> have lexer defines like:
>
> TAG
> : '\\'! IDENT
> ;
>
> Where IDENT is an alphanumeric identifier.  What I have noticed,
> however, is that the backslash character never makes it to the rewrite
> engine, and so, is missing from the output of originalToStream.

Yep.  don't mod the token stream...v3 doesn't have this problem...

> A possible solution to this is to not have my lexer do as much
> 'parsing', and just be concerned with more basic token types, but  
> once I

a good strategy...

> do that I get a wide variety of non-determinism errors.  There are

can u use syn preds?

Ter