[antlr-interest] beginer question

[Mike Quilleash]

The lexer is trying to match characters to produce tokens.  If you
define two lexer rules that have the same start, LETTER in this case,
then the lexer can't figure out which token it should be producing.

If I define these rules

LETTER
	:
	'a'..'z'|'A'..'Z'
	;

WORD: (LETTER)+

When the lexer sees a letter it can't tell whether to produce the LETTER
token or the WORD token, the WORD token is allowed to be a single
LETTER.  Usually you would make the LETTER rule protected which means it
is only used to build other lexer rules and will never be sent to the
parser as a token in it's own right.

A general rule is hard to define  as each grammar is different but
generally I use the lexer to break down the input into basic tokens,
word, integer, different sorts of punctuation, whitespace etc.  And then
have the parser define the syntax of when and how often these basic
blocks will appear.

-----Original Message-----
From: antlr-interest-bounces at antlr.org
[mailto:antlr-interest-bounces at antlr.org] On Behalf Of pepone pepone
Sent: 22 April 2006 16:13
To: antlr-interest at antlr.org
Subject: [antlr-interest] beginer question

Hello all

I'm begining work with antlr, and i a bit confused. I'm not sure when to
put a rule in lexer or in the parser. There is a general rule to decide
when a rule must be in lexer or in parser?

and i confused with protected rules, what is the diference in a lexer
between this tow rules

protected
LETTER
	:
	'a'..'z'|'A'..'Z'
	;

LETTER
	:
	'a'..'z'|'A'..'Z'
	;

if i don't use protected i get warning when use (LETTER)+ in other rule,
"warning:lexical nondeterminism upon"


Thanks in advance
--
play tetris http://pepone.on-rez.com/tetris run gentoo
http://gentoo-notes.blogspot.com/