[antlr-interest] Antlr 3 and the newline token problem

[Jim Idle]

________________________________

From: antlr-interest-bounces at antlr.org on behalf of Ric Klaren
Sent: Sat 11/26/2005 5:07 AM
To: ANTLR Interest
Subject: Re: [antlr-interest] Antlr 3 and the newline token problem



Prashant Deva wrote:
> Me and Terence were recently having a discussion about this.
> Its about how to handle newlines in antlr 3.

Can't you do it in a similar way as I intend to do it in C++? E.g. plug
in a user defined object that knows about the type of token tracking you
want to do? And one that knows how to put this location info inside a
newly created token?

In fact this is what I am doing for the C target. Though it is harder to do such things in C (neatly at least) as the language does not have built-in mechanisms for it, it is still the case that if we just get the very basics correct then a grammar author should be able to set up things to override anything and do as they will. By 'correct', I mean merely that we don't lose information that some object/class/struct/programmar might want to track.  

By default though, you will get the 'standard' stuff with lines and columns if only for the fact that I don't want to have the C runtime be inherently differnt to the other language targets. If that were to change then I would change the C target. But, just tracking offsets does not seem the way to me. 

In short then I agree with Ric's point of view. In fact the ability to override anything that makes sense to in a fairly trivial fashion (so long as you know the target language reasonably well) should be a general precept. If you don't like the default object/struct/typedef/blob then just supply your own definition and a bit of code to handle it.

Jim