[antlr-interest] TokenStreamRewriteEngine question
Terence Parr
parrt at cs.usfca.edu
Sun Mar 12 11:02:55 PST 2006
On Mar 12, 2006, at 8:51 AM, Scott Amort wrote:
> Hi All,
>
> I am using a TokenStreamRewriteEngine to discard unwanted
> whitespace and
> comments, while still retaining the original file contents for
> debug and
> error messages. However, I have noticed that within my lexer, I
> 'prediscard' a number of other characters, such as double-quotes,
> backslashes, etc. These latter types are necessary to define certain
> tokens, but I don't want them actually passed on to the parser, so I
> have lexer defines like:
>
> TAG
> : '\\'! IDENT
> ;
>
> Where IDENT is an alphanumeric identifier. What I have noticed,
> however, is that the backslash character never makes it to the rewrite
> engine, and so, is missing from the output of originalToStream.
Yep. don't mod the token stream...v3 doesn't have this problem...
> A possible solution to this is to not have my lexer do as much
> 'parsing', and just be concerned with more basic token types, but
> once I
a good strategy...
> do that I get a wide variety of non-determinism errors. There are
can u use syn preds?
Ter
More information about the antlr-interest
mailing list