[antlr-interest] Lexer grammar design issue

[Ric Klaren]

Hi,

On Wed, Jan 21, 2004 at 08:32:36AM -0800, Jason wrote:
> of grammars to ANTLR.  I've spent a couple days
> playing with ANTLR and the ANTLR documentation. 

You might also have a look at some of the (bigger) examples.

> The ANTLR documentation recommends using Semantic
> Predicates to simulate DFA based lexical states.  I
> would like to do something like this:

We also have token stream multiplexing to mimick some lexer state stuffs.

> where MYTOKEN is a rule which matches the literal
> 'mytoken.'  Unfortunately, it seems that ANTLR doesn't
> generate rules corresponding to literals.

I presume you mean for entries in the tokens section? The tokens section
works in combination with the testLiterals option. Usualy you have some
rule that recognizes all id's and set testLiterals to true (and have it
turned of for the rest of the lexer):

IDENTIFIER options { testLiterals = true; }:
        ( 'a' .. 'z' | 'A' .. 'Z' | '_' )
        ( 'a' .. 'z' | 'A' .. 'Z' | '0'..'9' | '_' )*

Whenever an IDENTIFIER is matched the match is first checked against the
literals table and if it matches an entry that literal is returned in stead
of IDENTIFIER.

Have a look at the multiLexer example for tokenstream multiplexing. Basic
idea is that in the lexer upon encountering a specific start token one
switches to a lexer that is better suited to deal with that type of token,
and switch back to the calling lexer when the 'close' token is encountered.
The example uses this to deal with java doc comments. But you could also
use it for exotic string formats or whatever.

Cheers,

Ric
--
-----+++++*****************************************************+++++++++-------
    ---- Ric Klaren ----- j.klaren at utwente.nl ----- +31 53 4893722  ----
-----+++++*****************************************************+++++++++-------
     "Evil will always triumph, because Good is dumb." ---  Spaceballs


 

Yahoo! Groups Links

To visit your group on the web, go to:
 http://groups.yahoo.com/group/antlr-interest/

To unsubscribe from this group, send an email to:
 antlr-interest-unsubscribe at yahoogroups.com

Your use of Yahoo! Groups is subject to:
 http://docs.yahoo.com/info/terms/