[antlr-interest] Capture content between tokens

consiliens at gmail.com consiliens at gmail.com
Sun Mar 7 23:38:38 PST 2010


Previously titled Multiple Lines with Implicit Endings in ANTLR.

Here is a simple example of the issue.
* 1
a
* b

I would like to define this wiki syntax as two tokens, "* 1\na" and "* 
b".  The token beginning is * and ends when another list token is found. 
  WikiToHTML is from tpdsl.
http://media.pragprog.com/titles/tpdsl/code/tpdsl-code.zip

This is similar to Ghrekin syntax.
Then 1
a
Then b

I'm trying to extend this concept in ANTLR to work correctly across more 
than one line.  In ragel you can do something like this:

FeatureHeading = space* I18N_Feature %begin_content ^FeatureHeadingEnd* 
:>>   FeatureHeadingEnd @store_feature_content;
FeatureHeadingEnd = EOL+ space* (I18N_Background | I18N_Scenario | 
I18N_ScenarioOutline | '@' | '#' | EOF)>next_keyword_start;
source: 
http://github.com/aslakhellesoy/gherkin/blob/master/ragel/lexer_common.rl.erb

That is what I meant by defining the current token as extending until 
the next valid token.  Is the reason I'm not finding examples of this in 
ANTLR because it's not possible?


More information about the antlr-interest mailing list