[antlr-interest] Global scopes for lexers

[Thomas Brandon]

On 8/1/07, Kay Roepke <kroepke at classdump.org> wrote:
>
> On Jul 31, 2007, at 5:51 PM, Alex Shneyderman wrote:
>
> > I thought it would be neat to have the same mechanism for lexer as
> > exists in parser. But of course you probably have a better idea of how
> > to keep track of
> > these levels. I am all ears :-)
>
> I'd say that this is a pretty good case for scopes. :)
>
> It's just that having to have them in a lexer is pretty uncommon,
> thus the reluctance to support the
> feature (at least that's my understanding). The same reasoning
> applies to return values in lexer rules. It doesn't
> really fit into the concept of the lexer (which should tokenize
> preferably without any context dependencies).
I would like to see return values and scopes supported in lexers. I
have tried to use return values in the lexer, before switching to
scopes when they couldn't be used. My use case was pretty non-standard
but there are situations where more work than desirable needs to be
done in the lexer and here they can come in handy.
Certainly return values for non-fragment rules seem unnecessary. But
for fragment rules they seem more useful. Scopes or return types can
be used to centralise action logic as well as matching logic in
fragment rules.
Yes, global variables can be used and the limited recursion in lexers
means stacks will rarely be necessary but return values\scopes seem
cleaner to me, keeping everything together in the rule. And allow more
complex cases like this.
>
> OTOH, given that the implementation of lexers and parsers are so
> alike scopes just magically work, as
> long as the target developer thinks of it, maybe there's a case for it.
Certainly if significant work was required to implement them there
would be a case to be made for excluding them. But from a quick scan
of the templates it looked like almost all of the support could be
shared. And I can't see any performance implications for them. The
intelligent return value handling for parsers nicely chooses between
wrapper objects and single return values as needed so should
efficiently handle them.

> BTW, that's one messed up language you have there ;)
>
> cheers,
> -k
> --
> Kay Röpke
> http://classdump.org/

Tom.