[antlr-interest] Is there any way to tokenize using the delimiter in lexer.java?

ganesh_rgk2003 ganesh_rgk2003 at yahoo.com
Mon Sep 13 21:57:53 PDT 2004


In Lexer.java the input string is tokenized based on the tokens 
defined under this.

I want to tokenize giving my own delimiter.

eg:
I have two tokens, say name and age defined in lexer.
And also have one more token alphanumeric.

Now, if my input string is

 name=raj age123=21

I need to tokenize this by giving delimiter as blankspace and '=' and 
generate four tokens 'name', 'raj', 'age123' and '21'.

Instead, since name, age and alphanumeric are the tokens defined, my 
lexer is generating token as follows,

    1.    'name'
    2.    '=raj'
    3.    'age'
    4.    '123=21'
which i dont require.

IS THERE ANY WAY WHERE I CAN SPECIFY IN LEXER TO TOKENIZE ACCORDINGLY?

Thanks in advance.
Ganz








 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
    antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/
 



More information about the antlr-interest mailing list