[antlr-interest] single-pass pattern matching "for free"?

[Terence Parr]

On Jan 14, 2006, at 8:18 PM, Nigel Sheridan-Smith wrote:
>> Would thing kind of thing be useful?  Best of both worlds?  For many
>> applications, this would be great!
>>
>
> Sounds good to me! But I think one of Andy's points was that not  
> everything
> in translation can be applied at the AST node level. You sometimes  
> need to
> look deeper (e.g. within strings), or broader (e.g. multiple lines  
> of code).

Correct.  Andy's experience in this would be gold!  perhaps he would  
like to help build this generic pattern engine...

Yes, we could do patterns on two levels: char and token I should think.

1. by char.  This is how I do fuzzy parsing (filter=true mode).  Try  
to match one of a set of rules against the input stream at position  
p.  If none matches, advance to p+1 and try again.  This would allow  
you to see inside strings.  The stuff in <...>  could only be token  
refs like <ID>

2. by token.  This would be the normal mode probably and much  
faster.  You could specify grammatical structures like <expr> and  
<stat>.  you could only see properly tokenized input.  The tool would  
tokenize via the grammar's lexer you provide separately and then  
generate a parser that would apply the rules and do replacement.

Make sense?

Ter