[antlr-interest] Tokenizing question

[Amal Khailtash]

In a language that whitespace is ignored, how can one tokenize and parse
constructs like this:

  word : number identifier ;

where 'word' could look like:

  10 abc  or  10abc

In this case number and identifier could have no whitespace between them or
have some.

-- Amal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20080210/a292a196/attachment.html