[antlr-interest] How do I set token type in ANTLR 3.1b1?

Vaclav Barta vbar at comp.cz
Mon Jun 9 02:02:14 PDT 2008


Johannes Luber wrote:
 >> I've been trying out ANTLR 3.1b1 with C#, and I'm running into
 >> trouble... My grammar (available at
...
 >> That works fine with ANTLR 3.0.1, target CSharp. For ANTLR 3.1b1,
 >> CSharp is unusable (see
 >> http://www.antlr.org/pipermail/antlr-interest/2008-May/028461.html
 >> for details).
 > Actually, the reason is that a function is missing in
 > CSharpTarget.java.  How that happened is a total mystery to me, as I
 > copied the contents from CSharpTarget.java wholly...
Well, that's always a danger with cut&paste... I'm glad to hear the
fix is obvious.

 >> CSharp2 target doesn't have that problem, but has others:
 >> while $token.text in the grammar must be changed to $token.Text,
 > That isn't supposed to be necessary. I did explicitly test all cases
 > and found that .text is sufficient then (I even changed the wiki
Well, I have a counter-example. The files are a bit big for e-mail, so
I put them under http://www.mangrove.cz/antlr-3.1b1-trouble/ . There
are 3 subdirectories: whole is my original grammar (with just the
target changed to CSharp2). Note the use of text in (for example)
MacroScopeLexer.mAsciiStringLiteral() - action { text = ""; } is
translated into

if ( state.backtracking == 0 )
{
    text = "";
}

 > text afterwards). Does anyone else have this problem? Really - is
 > this an isolated problem? I once had to use .Text in 3.0.1, too,
 > which prompted my quest to undo this difference. But a search where
 > .Text would be translated failed to show up any result, a test
 > grammar created to replicate this issue worked fine (for whatever
...but I admit the problem is elusive - when I cut the grammar down to
include just the AsciiStringLiteral rule (available in
http://www.mangrove.cz/antlr-3.1b1-trouble/sample ),
mAsciiStringLiteral() starts using Text... The generated functions
differ more than I expected (since they have the same spec) - apparently 
the more complicated grammar triggers some unusual path...

 > reason), and as the runtimes otherwise don't touch the action
 > translation stuff, I doubt that the target choice does affect
 > things. But you could test if Java gives the same problems. I'll
Well, the Java target generates text - but isn't that correct for
Java? I think the problem is simply that org.antlr.runtime.Lexer has
protected String member text while Antlr.Runtime.Lexer has that data
in an instance of RecognizerSharedState...

 >> $token.type doesn't work at all: $type in the rule above generates
 >> compilable code which fails at runtime with NullReferenceException
 >> while $Type is just copied to the generated code, which therefore
 >> doesn't compile. What is the correct way to set an explicit token
 > The code should work, and because I don't have my notebook for four
 > weeks already, trunk is still the same as in b1. But you could try
Well, it certainly can wait as far as I'm concerned, but it probably
should be fixed before releasing 3.1...

 > it anyways. Still, you could tell more about your environment -
 > maybe somehow it affects ANTLR?  Johannes
Well, just a Vista (bloody unstable, but I don't think that could have 
any influence on ANTLR) with Java 1.6.0_03... My CLASSPATH is

.;C:\proj\macroscope\latest\antlr-3.1b1\lib\antlr-2.7.7.jar;C:\proj\macroscope\latest\antlr-3.1b1\lib\antlr-3.1b1.jar;C:\proj\macroscope\latest\antlr-3.1b1\lib\stringtemplate-3.1.jar;C:\soft\apache-log4j-1.2.15\log4j-1.2.15.jar;C:\soft\commons-logging-1.1.1\commons-logging-1.1.1.jar;C:\PROGRA~1\Java\jdk1.6.0_03\lib\tools.jar;C:\soft\javamail-1.4.1\lib\mailapi.jar;C:\soft\javamail-1.4.1\lib\smtp.jar;C:\soft\commons-codec-1.3\commons-codec-1.3.jar;C:\soft\commons-httpclient-3.1\commons-httpclient-3.1.jar;C:\soft\junit3.8.2\junit.jar;C:\soft\httpunit-1.6.2\lib\httpunit.jar;C:\soft\httpunit-1.6.2\jars\nekohtml.jar;C:\soft\httpunit-1.6.2\jars\Tidy.jar;C:\soft\httpunit-1.6.2\jars\xmlParserAPIs.jar;C:\soft\httpunit-1.6.2\jars\xercesImpl.jar;C:\soft\httpunit-1.6.2\jars\js.jar

- I suppose I could try pruning it, but I'm sceptical that would
change anything...

	Bye
		Vasek
-- 
http://www.mangrove.cz/
Open Source integration


More information about the antlr-interest mailing list