[antlr-interest] lexer: display all rules matched

[Edwards, Waverly]

>>For the purposes of tests, just create an instance of the lexer and
ask it for all the tokens.
 
I'll have to more homework as I haven't seen an example of how this is
done.  The only way I've seen this done
is by chaining together parser or lexical rules and setting an action to
it.  I have two problems with this.
1.  The starting rule appears to be the type that gets returned when you
chain rules. 
2.  There isn't a way to determine which lexical rule is being used to
match the text.
 
I was especially troubled when I began to think that rule PERSON :
'person'; and rule ID could return the text
'person' and I wouldn't know which one was used or if some other
substring was matched such as
 
PER : 'per';
SON : 'son';
PERSON : 'person';
 
This brings me to another question.  Is the order the rules are created
significant to the matching.  Using the
example above in order to match 'person' would I need to order the rules
this way
 
PERSON : 'person';
PER : 'per';
SON : 'son';
 
These are hypothetical examples used to explain what I'm dealing with
and learn how to move forward.
 
 
Thanks,
 
 
W.

________________________________

From: antlr-interest-bounces at antlr.org
[mailto:antlr-interest-bounces at antlr.org] On Behalf Of Gavin Lambert
Sent: Tuesday, August 28, 2007 4:13 PM
To: Edwards, Waverly; antlr
Subject: Re: [antlr-interest] lexer: display all rules matched


At 01:58 29/08/2007, Edwards, Waverly wrote:


	Is there a way to inspect what lexer rules have been matched or
inspect the token buffer?
	What I'm trying to do is to setup test(s) to ensure that every
keyword is matched and that no rule conflicts with another.
	I have just short of 550 lexical rules just for keywords alone
(originally I thought it was just a couple hundred).


For the purposes of tests, just create an instance of the lexer and ask
it for all the tokens.  (Exactly how you do this depends on your target
language.)

There's no need to create a parser if all you want to do is to validate
the output of the lexer against a "known good" or "expected" list. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20070828/db93b2cc/attachment.html