[antlr-interest] Literals, Predicates and Actions

Julian Mensch jmensch at shaw.ca
Fri Mar 6 10:32:59 PST 2009


  Hi,

  I'm a newbie to ANTLR working on translating a long ACCENT
grammer to ANTLR under the C target, and have a few
questions about the use of literals to define tokens in combined
lexer-parser grammars. I understand that literals in parser rules
create implicit lexer rules, and I find this to be a very useful
feature for naturalistic languages that have a set of keywords of
notable size which can increase frequently, and include frequent
alternatives.

  What I'm wondering is if I can somehow apply global predicates
and actions to the implicit rules generated for literals. For example,
I know I can write:

LCURLY:
  '{' { theCompiler->BraceLevel++; }
  ;

block: LCURLY (statement)* RCURLY
  ;

  But I'd really like to be able to write:

@literals
  { if (strcmp(GETTEXT()->chars,"{"))
      theCompiler->BraceLevel++; }

block: '{' (statement)* '}'

  simply because including literals makes the code so
much more intuitive and readable.

  Predicates for literals would also be really useful, in
the case, for example, where you have a limited set
of keywords that are universal to the language, but
your ever-expanding larger set is only valid in some
lexical circumstances. For example:

@literals
  { isUniversalKeyword(GETTEXT()->chars) || inFullKeywordMode }?
  ;

  To add a syntactic predicate to all the literals in the
parser causing most of them not to be matched by the
lexer unless it is in a specific state.

  I know there's no such thing as the "@literals"
construct I'm showing here, but I'm wondering if there's
any way to duplicate the effect I'm going for with it.
Currently I'm matching all keywords as IDENT and
using string tables, setType() and tokens with 'fragment'
to handle keywords, but I find it really ugly because I
have to make changes in three places to add new
keywords, rather than just entering a literal into a
parser rule. If I want syntactic predicates or actions
on my keyword tokens, I have to hand-code it all.

-- Julian Mensch


More information about the antlr-interest mailing list