[antlr-interest] Newbe lexer question

[HC]

Thanks Kay,

Yes, I guess for question #2 I think I mean the second option. However, my
question 1 and 2 are sort of inter-related since the word I would be passing
in would be very much like the list in question #1.

"If you do not have a whole lot of those cases it shouldn't be a
concern."....

well, since I don't have control over the number of words provided by the
user, there might be many cases where I have composite words as in question
#1. I will look at actions and see how that can help. I just can't write a
custom rule for each word patter.

Thanks
HC




On 11/12/06, Kay Roepke <kroepke at classdump.org> wrote:
>
> Hi!
>
> On 12. Nov 2006, at 22:50 , HC wrote:
>
> > 1. Can (an what would a pseudo lexer rule be) antlr handle the
> > following
> > scenario easily:
> >
> > Let's say my lexer needs to be able to tokenize the following strings:
> >
> > a) "car"
> > b) "bus"
> > c) "bus car"
> >
> > It is fairly easy to write a rule for antlr that is smart enough to
> > recognize the string "bus car" as token C and not as token B
> > followed by
> > token A?
>
> Yes, this is no problem, although it looks a bit ugly at times ;)
> If you do not have a whole lot of those cases it shouldn't be a concern.
> I.e. you could solve this with syntactic predicates.
>
> > 2. Can I pass strings to be tokenized into the lexer from outside the
> > grammar file?
> >
> > Let's say I have lexer rule called WORD which may be any character
> > sequence
> > which I specify externally. I am looking to maintain an external
> > dictionary
> > of words which are valid WORD tokens which can be expanded and
> > reduced by a
> > user without modifying the grammar file.
>
> Am I understanding you correctly, in that you want to have a dynamic
> lexer rule, or
> do you want this:
> I could imagine to do this is to let the lexer recognize the said
> strings
> as WORDs and then use a custom action to look up the token's text in
> your external
> dictionary. You could indicate a recognition error to ANTLR and try
> error recovery
> (like ANTLR does internally) or simply fail.
> If you want the latter, this is easy. Dynamically changing lexer
> rules isn't because
> that would affect lookahead.
> Another way could be to employ syntactic predicates, but it really
> depends on what you
> are trying to achieve.
>
> > Am I using Antlr in the right context here?
>
> If you are doing language recognition, yes ;)
>
> HTH,
>
> -k
> --
> Kay Röpke
> http://classdump.org/
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20061113/7c77bbc0/attachment-0001.html