[antlr-interest] Re: Enhance ANTLR support for comments?

[micheal_jor]

Hi David,

> I would like to support this suggestion strongly.

I suspect the comments-handling suggestion probably translates into a 
requirement for more docs and clarity on the HiddenTokenStream 
mechanism. It already exists in ANTLR and supports quite flexible 
handling of out-of-band data (such as comments and whitespace) 
present in a TokenStream.

> When I was saving comments I "overloaded?" the consume function 
> which enabled me to interleave the comments again using line 
> numbers (and that's another thing which should be in standard 
> ANTLR, Line and Column numbers!). It was horrendous and I don't 
> think it ever worked perfectly!

For ANTLR 3, I would actually like even better support than that for 
token locations. At the least it should be:
- source (e.g. filename, URL etc)
- start line & column
- end line & column

One complicating issue is that depending on the constraints of 
execution environment, I might want a compact or even more compact 
representation of these. For instance, I might want to shoehorn both 
the line and column numbers into a single 32-bit integer (16:16 or 
24:8 split) or leave them as two separate integers.

Not sure how ANTLR 3 can support such scenarios easily.

Cheers,

Micheal



 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/