[antlr-interest] Lexer token gating predicate problems
Red Rackham
redrackem at yahoo.com
Fri Nov 12 15:44:05 PST 2010
I got it working... thanks.
For those who come later...
1. Add lexer header code functions for enabling, disabling, reading a global
state variable like "pathIdEnabled", make sure in each function to declare
global pathIdEnabled, like this;
@lexer::header {
pathTokenEnabled = False
def isEnabledPathToken():
global pathTokenEnabled
print 'isEnabledPathToken returning ' + str(pathTokenEnabled)
return pathTokenEnabled
def enablePathToken():
global pathTokenEnabled
print 'Enabling path Token Lexing'
pathTokenEnabled = True
def disablePathToken():
global pathTokenEnabled
print 'Disabling path Token Lexing'
pathTokenEnabled = False
}
2. Make tokens for 'include' & 'log', with calls to lexer header code to enable
path id like this:
LOG: 'log' {enablePathToken()};
3. Make tokens for EOL or SEMI (or whatever) with calls to lexer header code to
disable path id like this:
EOL: ('\r\n' | '\n') {disablePathToken()};
4. Rename PATH_ID token into fragment PATH_ID_PRE.
5. Make new token PATH_ID that only triggers if pathIdEnabled is True like this;
PATH_ID: {isEnabledPathToken()}?=> PATH_ID_PRE ;
Worked for me. Glad I was able to figure it out myself finally. Haha.
Mark
----- Original Message ----
From: Red Rackham <redrackem at yahoo.com>
To: antlr-interest at antlr.org
Sent: Fri, November 12, 2010 7:46:35 AM
Subject: [antlr-interest] Lexer token gating predicate problems
Hi I'm new so please go easy on me. I have a situation where I have a PATH_ID
that has to allow many characters, and a general ID that has to match many
characters, but have their unique patterns.
So in my combined grammar in the lexer section, I have defined PATH_ID &
IDENTIFIER. I have also other tokens defined like INTEGER, STRING, etc.
So when I try to run input through the final implemenation (target=Python) I get
PATH_ID catching IDENTIFIERS, and just about anything else. What do I do to
make PATH_ID only activate in context of "include PATH_ID" or "log PATH_ID", and
never for other identifiers?
I've tried a lot of stuff with reordering tokens, predicates and gating symbols
and no matter what I've tried it doesn't compile AND run right. ("The following
token definitions can never be matched because prior tokens match the same
input", "rule PATH_ID has no defined parameters", "failed prediate", &c). If I
comment out PATH_ID then IDENTIFIER works fine, but I can't have a grammar
without PATH_ID.
Thanks in advance!
Mark
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
More information about the antlr-interest
mailing list