[antlr-interest] tracking token position when original file is pre-processed

[Jim Idle]

Easiest is to have the preprocessor mark the input stream like cpp does:

# 555 "myfile.c"

And then add a reference to the file into the token or similar.

You can also incorporate the preprocessor in to your lexer and stack input streams if it isn't in need of a parser to do the pre-processing.


Jim

> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> bounces at antlr.org] On Behalf Of Bob Frankel
> Sent: Wednesday, January 06, 2010 5:24 PM
> To: antlr-interest at antlr.org
> Subject: [antlr-interest] tracking token position when original file is
> pre-processed
> 
> my language has a simple pre-processor that expands text of the form
> ${<env-var-name>} as a first phase of translation; the expanded stream
> is then input to my ANTLRInputStream, where it proceeds onward to the
> lexer/parser in the usual fashion.  said another way, neither the lexer
> nor the parser is aware of the ${...} construct.
> 
> needless to say, character-position information (eg., token start/stop)
> are relative to the expanded stream and not the original file; this
> creates an problem, of course, when error indicators are not correctly
> positioned in the original source file (as i'm doing through some
> editor
> integration inside eclipse).
> 
> is there some pattern and/or (simple!) example that illustrates a
> technique for managing this situation; is there some way (say) i might
> embedded the equivalent of #line directives in the expanded stream
> which
> are then stripped further downstream while adjusting token offsets???
> 
> 
> 
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
> email-address