[antlr-interest] difference between tokens and string literals

[Gavin Lambert]

At 04:34 1/01/2008, Mark Volkmann wrote:
 >
 >I'm confused about how to detail with the difference between
 >tokens and string literals.

Tokens are tokens, and string literals are mere sequences of 
characters.  In certain cases, string literals can be promoted 
automatically to tokens, but I usually find that more confusing 
than helpful.

 >I don't know the correct way to write the "list" rule in the 
tree
 >grammar. When I give this the input "list variables", why is the
 >output "failed"?

It's because the actual tree from the rewrite is ^(LIST 
LIST_OPTION), which matches option 3.  In fact options 1 and 2 
really ought to give you a compiler error since it's impossible 
for those to ever match anything, even if you did keep the actual 
text of the option around -- but ANTLR 3's grammar error checking 
is a little flaky at the moment because it's still using ANTLR 2 
to do much of the work.

You could use a semantic predicate to compare the contents of the 
LIST_OPTION token (assuming you modified the parser to actually 
give it some contents), but personally I think it'd make things 
much simpler if you broke this up into multiple tokens anyway:

lexer grammar BasicLexer;

tokens {
   LIST = 'list';
   FUNCTIONS = 'functions';
   VARIABLES = 'variables';
}

NEWLINE: ('\r'? '\n')+;
WHITESPACE: ' '+ { $channel = HIDDEN; };

parser grammar BasicParser;
options {
   output = AST;
   tokenVocab = BasicLexer;
}

list: LIST list_option terminator -> ^(LIST[$LIST] $list_option);
list_option: FUNCTIONS | VARIABLES;
terminator: NEWLINE | EOF;

tree grammar BasicTree;
options {
   ASTLabelType = CommonTree;
   tokenVocab = BasicLexer;
   output = template;
}

list
   : ^(LIST FUNCTIONS) { System.out.println("list functions isn't 
supported yet"); }
   | ^(LIST VARIABLES) { System.out.println("list variables isn't 
supported yet"); }
   | ^(LIST .) { System.out.println("failed"); }
   ;