[antlr-interest] detecting transitions in stanza-based files

[Chris Black]

Jon.Rand at thomson.com wrote:

>Chris Black wrote:
>  
>
>>>Excel likes to add enough delimiters at the end of every 
>>>line so that all lines have an equal number of columns, 
>>>this leads to lots of rules in my grammars that end 
>>>in "(DELIM)* NEWLINE" which I understand can be 
>>>inefficient and also lead to some nondeterminism 
>>>difficulties. I am going to have my TokenFilter remove 
>>>these. 
>>>      
>>>
>
>Could you use a regular expression against the input string to remove
>the extraneous delimiters before you pass the input to ANTLR?  Then you
>wouldn't have to clutter your grammar with stuff that essentially does
>exactly the same thing.  It's not a pure ANTLR solution, but it might
>make your grammar a lot easier to deal with.
>
>Jon
>
>  
>
I'm getting to the point of doing that, but I had hoped to be able to 
make a token filter to do this. I'm also kind of trying to justify the 
use of antlr over regexps, and resorting to regexp to solve a problem 
that I can't fix in ANTLR doesn't help :)

Chris