[antlr-interest] Context Sensitive Keyword Support?

[Jim Idle]

I think that every time we talk about this, we realize that the problems
lie in lookahead where you cannot decide what to do in certain cases. I
think that the quantum token idea is a much better one in that a token can
simultaneously be ID and WHERE or any other token that it is flagged as
being possible to be. This removes context from the lexer and allows the
parser to decide.

Predicates would still work and any other ambiguity should work out? It
means that input.LA(n) would also need

input.LA(n, TYPE)
  and perhaps
input.LT(n).getType(TYPE)

which would return TYPE if LT(n)/LA(n) can be one, otherwise the main type
that it actually is. Could have Boolean versions instead/as well of
course.

WHERE :: ID, KEYWORD : 'WHERE' ;

Or something like that. I think that this is well worth looking in to.



Jim

> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> bounces at antlr.org] On Behalf Of Terence Parr
> Sent: Wednesday, February 16, 2011 2:09 PM
> To: antlr-interest Interest
> Subject: Re: [antlr-interest] Context Sensitive Keyword Support?
>
> hiya. yep, i had a similar proposal way back when for v3.0 but I
> abandoned; can't remember why at this point. i love idea but impl had
> some issues.
> ter
> On Feb 16, 2011, at 2:02 PM, Scott Stanchfield wrote:
>
> > Has anyone looked at the context-sensitive-scanning proposal I sent a
> > while back?
> >
> >    http://javadude.com/articles/antlr-context-sensitive-scanner.html
> >
> > -- Scott
> >
> > ----------------------------------------
> > Scott Stanchfield
> > http://javadude.com
> >
> >
> >
> > On Wed, Feb 16, 2011 at 4:38 PM, Jim Idle <jimi at temporal-wave.com>
> wrote:
> >> No you do this in the parser by allowing the keywords where ID is
> >> expected then by predicating the WHERE keyword in the LINQ rules:
> >>
> >>
> >> .... ((WHERE)=>where_clause)? ....
> >>
> >>
> >> id : ID | WHERE | ....
> >>
> >>
> >> I have fully working C# lexer, parser, tree walker if that is what
> >> you are trying to build. It is a commercial product though.
> >>
> >> Jim
> >>
> >>> -----Original Message-----
> >>> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> >>> bounces at antlr.org] On Behalf Of chris king
> >>> Sent: Wednesday, February 16, 2011 1:32 PM
> >>> To: antlr-interest at antlr.org
> >>> Subject: [antlr-interest] Context Sensitive Keyword Support?
> >>>
> >>> Does ANTLR support context sensitive keywords? For example, in C#
> >>> "where" is a keyword only in a LINQ expression. To support this
> >>> ANTLR would need to change the lexer behavior given feedback from
> >>> the parser letting it know what context it is in. From what I hear
> >>> ANTLR lexes the entire file before passing it onto the parser. If
> >>> that's true than I guess it doesn't support this language concept.
> >>> In that case, is there a way to work around?
> >>>
> >>> Thanks,
> >>> Chris
> >>>
> >>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> >>> Unsubscribe:
> >>> http://www.antlr.org/mailman/options/antlr-interest/your-
> >>> email-address
> >>
> >> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> >> Unsubscribe:
> >> http://www.antlr.org/mailman/options/antlr-interest/your-email-
> addres
> >> s
> >>
> >
> > List: http://www.antlr.org/mailman/listinfo/antlr-interest
> > Unsubscribe:
> > http://www.antlr.org/mailman/options/antlr-interest/your-email-
> address
>
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
> email-address