[antlr-interest] Lexer token gating predicate problems

Red Rackham redrackem at yahoo.com
Fri Nov 12 15:44:05 PST 2010


I got it working... thanks.

For those who come later...

1. Add lexer header code functions for enabling, disabling, reading a global 
state variable like "pathIdEnabled", make sure in each function to declare 
global pathIdEnabled, like this;
@lexer::header {
    pathTokenEnabled = False
    def isEnabledPathToken():
        global pathTokenEnabled
        print 'isEnabledPathToken returning ' + str(pathTokenEnabled)
        return pathTokenEnabled
    def enablePathToken():
        global pathTokenEnabled
        print 'Enabling path Token Lexing'
        pathTokenEnabled = True
    def disablePathToken():
        global pathTokenEnabled
        print 'Disabling path Token Lexing'
        pathTokenEnabled = False
    }
2. Make tokens for 'include' & 'log', with calls to lexer header code to enable 
path id like this:
LOG: 'log' {enablePathToken()};
3. Make tokens for EOL or SEMI (or whatever) with calls to lexer header code to 
disable path id like this:
EOL: ('\r\n' | '\n') {disablePathToken()};
4. Rename PATH_ID token into fragment PATH_ID_PRE.
5. Make new token PATH_ID that only triggers if pathIdEnabled is True like this;
PATH_ID: {isEnabledPathToken()}?=> PATH_ID_PRE ;

Worked for me.  Glad I was able to figure it out myself finally.  Haha.

Mark


----- Original Message ----
From: Red Rackham <redrackem at yahoo.com>
To: antlr-interest at antlr.org
Sent: Fri, November 12, 2010 7:46:35 AM
Subject: [antlr-interest] Lexer token gating predicate problems

Hi I'm new so please go easy on me.  I have a situation where I have a PATH_ID 
that has to allow many characters, and a general ID that has to match many 
characters, but have their unique patterns.

So in my combined grammar in the lexer section, I have defined PATH_ID & 
IDENTIFIER.  I have also other tokens defined like INTEGER, STRING, etc.

So when I try to run input through the final implemenation (target=Python) I get 


PATH_ID catching IDENTIFIERS, and just about anything else.  What do I do to 
make PATH_ID only activate in context of "include PATH_ID" or "log PATH_ID", and 


never for other identifiers?

I've tried a lot of stuff with reordering tokens, predicates and gating symbols 
and no matter what I've tried it doesn't compile AND run right.  ("The following 


token definitions can never be matched because prior tokens match the same 
input", "rule PATH_ID has no defined parameters", "failed prediate", &c).  If I 
comment out PATH_ID then IDENTIFIER works fine, but I can't have a grammar 
without PATH_ID. 


Thanks in advance!
Mark



      

List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe: 
http://www.antlr.org/mailman/options/antlr-interest/your-email-address



      


More information about the antlr-interest mailing list