[antlr-interest] Source positions for imaginary tokens

Jesse McGrew jmcgrew at gmail.com
Fri Sep 14 10:13:17 PDT 2012


For what it's worth, I tried to solve the "keywords as identifiers" problem
using what seems to be the recommended solution -- a parser rule that
accepts keywords as well as ID -- and could not get it to work: Antlr would
crash when I tried to generate code, which I assumed was because of the
number of alternatives I was adding to every place an identifier could
appear.

Instead, I ended up getting rid of the lexer rules for keywords, so every
keyword is lexed as ID, and overriding the "emit" function to look the text
up in a keyword hash table and set a field in the token. Keywords are
matched with parser rules that use gating semantic predicates to check that
field.

Jesse
On Sep 12, 2012 11:53 PM, "Mike Lischke" <mike at lischke-online.de> wrote:

>
> Hey Jim,
>
> > Sure – I can make it be either of those calls, but not both at once. I
> have
> > no context at code generation time that can tell me which one to
> generate.
>
> So you say, you don't know at this time what type $kw in the ID[$kw]
> expression is? Absolutely no way to determine if that is a string or a
> token reference? That's odd.
>
> > If I change it to this, then all the people that want it to be the other
> > way, will claim that they have found a bug too. It only works in Java
> > because the Java compiler can see what the argument types are, and can
> > therefore call the “correct” method.
>
> I understand your restrictions, but find this situation all but pleasant
> (and I'm not alone I'm afraid).
>
> > However, it is much simpler to just use code to operate on the token
> > directly. Even before that, you should consider whether you need to
> change
> > something about the token because a later stage MUST receive a different
> > token, or whether you just think that you WANT it to.
>
>
> In the keyword case it is just so that I need only one (very common) token
> type but want to retain the token text for later processing. It's
> unfortunate that there is no general solution for the frequently
> encountered keywords-as-identifier problem.
>
> Anyway, Jim, thanks for patience and time!
>
> Mike
> --
> www.soft-gems.net
>
>
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe:
> http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>


More information about the antlr-interest mailing list