[antlr-interest] Selection between lexer rule and literal

[Alexey Demakov]

>From abstract point of view, lexer works ahead of parser and doesn't know about parser rules.
If in lexer you have tokens section, VALUE rule with testLiterals = true, lexer will match "COMPONENTS" as VALUE
then look in literals table (that contains info from tokens section) and always requalify token type from VALUE to COMPONENTS.
As far as I understand, you are trying to parse the same text as different tokens depending on context?
I'm afraid it is not right and easy way. But you can add parser rule that matches VALUE and all keywords
and use it in appopriate places.

Regards,
Alexey

-----
Alexey Demakov
TreeDL: Tree Description Language: http://treedl.sourceforge.net
RedVerst Group: http://www.unitesk.com

----- Original Message ----- 
From: "togol machillan" <togolmach2 at lycos.com>
To: <antlr-interest at antlr.org>
Sent: Tuesday, April 19, 2005 4:31 PM
Subject: Re: [antlr-interest] Selection between lexer rule and literal


> Thanks for the prompt reply. But defining tokens section in the lexer does not solve the problem. I would like the parser to match
the occurence of COMPONENTS, for example, in the input file with the VALUE rule if it does not find any reference to the
p_keyword_list rule in the rule currently being matched. The parser never matches COMPONENTS with the VALUE rule, although it is a
valid token of the VALUE type.
>
> Regards,
>
> Mach
>
> ----- Original Message -----
> From: "Alexey Demakov" <demakov at ispras.ru>
> To: antlr-interest at antlr.org
> Subject: Re: [antlr-interest] Selection between lexer rule and literal
> Date: Tue, 19 Apr 2005 15:06:04 +0400
>
> >
> > ----- Original Message -----
> > From: "togol machillan" <togolmach2 at lycos.com>
> > To: <antlr-interest at antlr.org>
> > Sent: Tuesday, April 19, 2005 2:54 PM
> > Subject: [antlr-interest] Selection between lexer rule and literal
> >
> >
> > > I have a rule in the lexer which looks like the following.
> > >
> > > VALUE  options {testLiterals = true; }
> > >        :CAPITAL_LETTER (CAPITAL_LETTER|DIGIT|','|':'|'+'|'-')*
> > >        ;
> > >
> > > In the parser, I have defined a list of literals like the
> > > following example rule
> > >
> > > p_keyword_list returns [ANTLR_USE_NAMESPACE(std)string k]
> > >   :
> > >    (
> > >    {k = LT(1)->getText(); col = LT(1)->getColumn();} "COMPONENTS"
> > >    |{k = LT(1)->getText(); col = LT(1)->getColumn();} "DATABANKS"
> > >    |{k = LT(1)->getText(); col = LT(1)->getColumn();} "PROP-SOURCES"
> > >    )
> > >                           ;
> > >
> > > Now when some parser rule is looking for a VALUE token and it
> > > sees, for example, COMPONENTS, it returns an unexpected token
> > > error.
> > It there some way of making the parser intelligent enough so that
> > it considers COMPONENTS (defined as a literal) as a token of type
> > VALUE.
> >
> > Try to define all tokens in lexer. For keywords there is "tokens" section
> > as described in
> > http://marlboro/docs/cc/antlr-2.7.4/metalang.html#TokensSection
> > So, your grammar should look like:
> >
> > In lexer:
> >
> > tokens
> > {
> >      COMPONENTS = "COMPONENTS";
> >      DATABANKS = "DATABANKS";
> >      PROP_SOURCES = "PROP-SOURCES";
> > }
> >
> > In parser:
> >
> > p_keyword_list returns [ANTLR_USE_NAMESPACE(std)string k]
> > :
> >    {k = LT(1)->getText(); col = LT(1)->getColumn();}
> >    (  COMPONENTS
> >     | DATABANKS
> >     | PROP_SOURCES
> >    )
> > ;
> >
> > Regards,
> > Alexey
> >
> > -----
> > Alexey Demakov
> > TreeDL: Tree Description Language: http://treedl.sourceforge.net
> > RedVerst Group: http://www.unitesk.com