[antlr-interest] Inconsistent token line number and character position info in AST Walker?

[Randall Barnhart]

I was curious if anyone else has seen inconsistent results when trying to access the line number and character position in line of a token in the AST?  Do I always need to regenerate the grammar code every time I regenerate the tree walker code?
 
My setup is as follows: I am running with ANTLR 3.0.1.  We have a JFace text editor that contains the source code.  We grab the source code out as a java String object.  And the following is my initialization code:
 
public static ArrayList<PredicateError> compilePredicate(String text, IDocument srcDoc, ArrayList<Signatures> prjSignatures, DLPredicateConcept predConcept) {  clearAndResetCompiler();          // Create a lexer attached to that input stream  ANTLRStringStream txtStream = new ANTLRStringStream(text);        iorLPLexer lexer = new iorLPLexer(txtStream);        // Create a stream of tokens pulled from the lexer        CommonTokenStream tokens = new CommonTokenStream(lexer);
        // Create a parser attached to the token stream        iorLPParser parser = new iorLPParser(tokens);        iorLPParser.system_return r = null;        try        {         // Invoke the program rule in get return value         r = parser.system();        }
 
And here is a snippet of my AST walker code:
 
varAssign returns [CLPredTail varAssignConstr] : ^(VARASSIGN ID expr)   {    if(currPredRules.size() > 0)    {     String currRuleName = (String)currPredRules.peek();     PredRuleImpl ruleImpl = checker.getPredRuleImplByName(currRuleName);     if(ruleImpl != null)      ruleImpl.addVarValue($expr.start, $ID.text, $expr.text, $expr.exprType);     else      System.err.println("Did not find valid matching rule implementation in symbol table.");    }    else    {     // Var assign outside of predicate in stmt block    }        if($expr.exprValue != null)    {     if($expr.exprType.equals(IORLPCompilerHelper.NUMERIC_TYPE))     {      if($expr.exprConstraint != null)      {       if(CLPredNumConstant.class.isInstance($expr.exprConstraint))       {        CLPredNumConstAssign predConstr = new CLPredNumConstAssign((Double)$expr.exprValue, $ID.text);        int ruleVarAssignLN = $varAssign.start.getLine();        int ruleVarAssignCP = $varAssign.start.getCharPositionInLine();        int ruleVarAssignSI = $varAssign.start.getTokenStartIndex();        int varAssignLN = $VARASSIGN.getLine();        int varAssignCP = $VARASSIGN.getCharPositionInLine();        int varAssignSI = $VARASSIGN.getTokenStartIndex();        int idLN = $ID.getLine();        int idCP = $ID.getCharPositionInLine();        int idSI = $ID.getTokenStartIndex();        int exprLN = $expr.start.getLine();        int exprCP = $expr.start.getCharPositionInLine();        int exprSI = $expr.start.getTokenStartIndex();        int docLineNo = $varAssign.start.getLine();//calcLineNo($ID.getTokenStartIndex());        int docCharNo = $varAssign.start.getCharPositionInLine();//($ID.getTokenStartIndex(), docLineNo);        predConstr.setTraceInfo(predConcept.getSymbol(), predConcept.getPredicateInstances().size()+1, docLineNo, docCharNo, $varAssign.text);
 
I've gotten different results running this code by generating just the walker code versus generating the parser/lexer and walker.  Sometimes all the above getLine(), getCharPositionInLine() return "1".  Which is not correct because the source has multiple lines.  Is there a problem using java String?  Then when I generated code for both the grammar and the tree walker I got different results and it was actually returning the correct line number and character position information.
 
Has anyone else experienced this inconsistency?  Should I be trying to access the line/char info from the top rule, imaginary tokens, or real tokens or does it not matter?  Would it be wise to always regenerate both the grammar and the tree walker?
 
Thanks,
Randall Barnhart
_________________________________________________________________
In a rush?  Get real-time answers with Windows Live Messenger.
http://www.windowslive.com/messenger/overview.html?ocid=TXT_TAGLM_WL_Refresh_realtime_042008
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20080321/2f973a34/attachment.html