[antlr-interest] Customizing token separators without recompiling

[Steve Cooper]

I presume that the separator changes other lexing rules, too? If one
customer chooses to use '$' as the separator, their other tokens can't
contain '$'. But if someone chooses '~', then a '$' *can* exist in the
other tokens. So the whole lexer will have to change.

If that's not so, then you can just pre-process the input before you
feed it to the lexer. So something like;

   constantSeparator = "~"
   cleanInput = originalInput.replace(userSeparator, constantSeparator);

and then write your lexer so it uses the constant separator '~'

   Steve