[antlr-interest] Tokenizing question

[Mark Volkmann]

On Feb 10, 2008 9:17 AM, Amal Khailtash <akhailtash at gmail.com> wrote:
> In a language that whitespace is ignored, how can one tokenize and parse
> constructs like this:
>
>   word : number identifier ;
>
> where 'word' could look like:
>
>   10 abc  or  10abc
>
> In this case number and identifier could have no whitespace between them or
> have some.

How can you tell where one "word" ends and the next begins?
Is each "word" on its own line?

-- 
R. Mark Volkmann
Object Computing, Inc.