[antlr-interest] lexer : find every possibilities in a text (backtrack)

[Gavin Lambert]

At 02:13 29/07/2008, Laura Adam wrote:
 >I would like the lexer to give me every possibilities there are 
to
 >cut the text, following the rules I gave.
 >Rigth now I use backtrack and memoize. I found one solution 
while
 >reading the text.

ANTLR isn't really designed for that sort of thing (and 
backtracking and memoisation only apply to the parser, not the 
lexer).

You need to construct your lexer grammar in such a way that any 
given input leads to exactly one unambiguous token stream.  If 
your language structure is ambiguous then you'll need to either 
force longer lookahead with predicates or defer decisions until 
parse time (eg. accepting multiple types of tokens at certain 
points).