[antlr-interest] Reuse of tokens and rules

Emond Papegaaij e.papegaaij at student.utwente.nl
Mon Jul 3 08:26:45 PDT 2006


On Friday 30 June 2006 16:48, Thomas Brandon wrote:
> On 7/1/06, Emond Papegaaij <e.papegaaij at student.utwente.nl> wrote:
> <cut need token reuse>
> > Is it possible to write small specifications for these constructs and
> > import the one needed? These rules are only a small portion of all the
> > rules. I don't like the solution of having to maintain the same parser and
> > lexer for multiple target languages.
> >
> > PS. I'm using ANTLR v3.
>
> Check out the island-grammar example in the v3 examples pack. Should give
> you the basis for everything you need.

I'm running into a problem with the island-grammar solution. In that example 
the boundaries of the tokens are easily matched in the outer language. For 
example javadoc starts with /** and ends with */. The embedded Simple 
statements are surrounded by '{' and '}'. In my language I most of 
the 'islands' are not that clear. For example 'identifier' is taken from the 
java specification.

In the island-grammar example the nested grammar is started with the beginning 
of the token is matched. However in my case there is no way to know when to 
switch grammars. It seems I need some way to try to parse a token (and thus a 
rule in the nested grammar) and continue when it is not a match.

Another problem I'm facing is terminating the nested grammar. Most rules don't 
end with a specific token. This makes it impossible to end the lexer. Isn't 
it possible to have the lexer produce tokens on demand, rather than at the 
start? That way I can simply return from the parser rule and continue with 
the outer grammar.

Best regards,
Emond


More information about the antlr-interest mailing list