[antlr-interest] Selection between lexer rule and literal

[Alexey Demakov]

----- Original Message ----- 
From: "togol machillan" <togolmach2 at lycos.com>
To: <antlr-interest at antlr.org>
Sent: Tuesday, April 19, 2005 2:54 PM
Subject: [antlr-interest] Selection between lexer rule and literal


> I have a rule in the lexer which looks like the following.
>
> VALUE  options {testLiterals = true; }
>        :CAPITAL_LETTER (CAPITAL_LETTER|DIGIT|','|':'|'+'|'-')*
>        ;
>
> In the parser, I have defined a list of literals like the following example rule
>
> p_keyword_list returns [ANTLR_USE_NAMESPACE(std)string k]
>   :
>    (
>    {k = LT(1)->getText(); col = LT(1)->getColumn();} "COMPONENTS"
>    |{k = LT(1)->getText(); col = LT(1)->getColumn();} "DATABANKS"
>    |{k = LT(1)->getText(); col = LT(1)->getColumn();} "PROP-SOURCES"
>    )
>                           ;
>
> Now when some parser rule is looking for a VALUE token and it sees, for example, COMPONENTS, it returns an unexpected token error.
It there some way of making the parser intelligent enough so that it considers COMPONENTS (defined as a literal) as a token of type
VALUE.

Try to define all tokens in lexer. For keywords there is "tokens" section
as described in http://marlboro/docs/cc/antlr-2.7.4/metalang.html#TokensSection
So, your grammar should look like:

In lexer:

tokens
{
    COMPONENTS = "COMPONENTS";
    DATABANKS = "DATABANKS";
    PROP_SOURCES = "PROP-SOURCES";
}

In parser:

p_keyword_list returns [ANTLR_USE_NAMESPACE(std)string k]
:
  {k = LT(1)->getText(); col = LT(1)->getColumn();}
  (  COMPONENTS
   | DATABANKS
   | PROP_SOURCES
  )
;

Regards,
Alexey

-----
Alexey Demakov
TreeDL: Tree Description Language: http://treedl.sourceforge.net
RedVerst Group: http://www.unitesk.com