[antlr-interest] Doing translation in the lexer (the right tool for the job?)

[Randall R Schulz]

On Tuesday 19 June 2007 23:57, Wincent Colaiuta wrote:
> One of the many translation tasks I'd like to use ANTLR for is
> translating wikitext to HTML. Two things make this quite difficult:
>
> 1. The translator has to be extremely tolerant of malformed input
> (users may not understand the wikitext syntax or may deliberately
> subvert it) and make reasonable "cleanup" attempts for bad input
> (such as closing tags that were left open by the user).
>
> 2. Wikitext structure is not unlike HTML structure, which means it is
> highly recursive and is very sensitive to nesting and ordering
> issues.

Maybe you should investigate the techniques used by the browsers. They 
bend over backward to accommodate the kinds of ill-formed input you 
have to deal with and generally do pretty well.

Also, I take it the authoring of these Wiki pages is an interactive 
process? Perhaps you'd be better off guiding users towards correct 
usage rather than trying to come up with useful interpretations to bad 
input?


> ...
>
> Cheers,
> Wincent


Randall Schulz