[antlr-interest] ANTLR Examples StdCParser.g for C++

[Ric Klaren]

Hi,

On Fri, Nov 23, 2001 at 03:36:04PM +0000, David Wigg wrote:
> I have, of course, had a look round for any documentation that might
> help me but all I can find is documentation about how to use the system,
> but not about how it works.

This is a known problem yes. The documentation of the support library is
lacking. I'm slowly improving on this in the C++ part. I don't know if you
already found some of the new documentation at:

http://wwwhome/~klaren/antlr/support-docs/

This is for the new release so it's not 100% compatible with the 2.7.1
release.

> This problem concerns the makeToken(int t) function in the lexer part of
> StdCParser.g.
> 
> I cannot find out how to create a RefToken which points to my CToken
> (which is a subclass of CommonToken which is a subclass of Token).
..
> I cannot work out the significance of "typedef RefToken
> (*factory_type)();" in CharScanner.hpp and various references to
> factory_type, tokenFactory, factory etc. etc.

This defines a function pointer to something that's supposed to make
RefTokens a socalled factory (look for instance at the book Design Patterns
from Gamma, Helm, Johnson and Vlissides for a description of this). Inside
makeToken the tokeFactory (of type factory_type) is called to generate the
tokens returned by the lexer. ANTLR uses internally only RefTokens (in java
mode Tokens this is a result of the Reference counting used in the C++
classes) Before starting a parse/lex run you can set the factory the lexer
uses for tokens with setTokenObjectFactory.

> For instance, is there a way to to use CharScanner::makeToken() create a
> RefToken pointing to a CToken?
> 
> Ric Klaren suggested I would need a factory in CToken but I regret I
> don't really know how to use it.

First step is to tell your lexer that it should generate CTokens this is
done by calling the setTokenObjectFactory method with the factory you made
for the CTokens: setTokenObjectFactory(Ctoken::factory) do this before you
actually start parsing.

Also define in CToken.hpp for convenience:

typedef RefCount<CToken> RefCToken;

Then to the definition of makeToken (which overrides the one in
CharScanner)

ANTLR_USE_NAMESPACE(antlr)RefToken makeToken(int t)
{
	if ( t != ANTLR_USE_NAMESPACE(antlr)Token.SKIP && countingTokens) {
		tokenNumber++;
	}
 
	// use superclasses makeToken to create a filled in CToken (returned as
	// RefToken)	
	ANTLR_USE_NAMESPACE(antlr)RefToken tok = CharScanner::makeToken(t);
	// this is dirty but works.
	ANTLR_USE_NAMESPACE(antlr)RefCToken ctok = 
		static_cast<ANTLR_USE_NAMESPACE(antlr)RefCToken>(tok);

	tok->setLine(lineObject.line);
	tok->setSource(lineObject.source);
	tok->setTokenNumber(tokenNumber);

	lineObject.line += deferredLineCount;
	deferredLineCount = 0;
	return tok;
}

Hope this helps,

Ric
--
-----+++++*****************************************************+++++++++-------
    ---- Ric Klaren ----- klaren at cs.utwente.nl ----- +31 53 4893722  ----
-----+++++*****************************************************+++++++++-------
 Why don't we just invite them to dinner and massacre them all when they're
  drunk? You heard the man. There's seven hundred thousand of them. Ah? ..
           So it'd have to be something simple with pasta, then.
                 From: Interesting Times by Terry Pratchet


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/