ANTLR3_COMMON_TOKEN_struct Struct Reference

#include <antlr3commontoken.h>

Collaboration diagram for ANTLR3_COMMON_TOKEN_struct:

Collaboration graph
[legend]

Detailed Description

The definition of an ANTLR3 common token structure, which all implementations of a token stream should provide, installing any further structures in the custom pointer element of this structure.

Remarks:
Token streams are in essence provided by lexers or other programs that serve as lexers.

Data Fields

ANTLR3_UINT32 channel
 The virtual channel that this token exists in.
ANTLR3_INT32 charPosition
 The character position in the line that this token was derived from.
void * custom
 Pointer to a custom element that the ANTLR3 programmer may define and install.
ANTLR3_BOOLEAN factoryMade
 Indicates that a token was produced from the token factory and therefore the the freeToken() method should not do anything itself because token factory is responsible for deleting it.
void(* freeCustom )(void *custom)
 Pointer to a function that knows how to free the custom structure when the token is destroyed.
ANTLR3_UINT32(* getChannel )(struct ANTLR3_COMMON_TOKEN_struct *token)
 Pointer to a function that gets the channel that this token was placed in (parsers can 'tune' to these channels.
ANTLR3_INT32(* getCharPositionInLine )(struct ANTLR3_COMMON_TOKEN_struct *token)
 Pointer to a function that gets the offset in the line where this token exists.
ANTLR3_UINT32(* getLine )(struct ANTLR3_COMMON_TOKEN_struct *token)
 Pointer to a function that gets the 'line' number where this token resides.
ANTLR3_MARKER(* getStartIndex )(struct ANTLR3_COMMON_TOKEN_struct *token)
 Pointer to a function that gets the start index in the input stream for this token.
ANTLR3_MARKER(* getStopIndex )(struct ANTLR3_COMMON_TOKEN_struct *token)
 Pointer to a function that gets the stop index in the input stream for this token.
pANTLR3_STRING(* getText )(struct ANTLR3_COMMON_TOKEN_struct *token)
 Pointer to function that returns the text pointer of a token, use toString() if you want a pANTLR3_STRING version of the token.
ANTLR3_MARKER(* getTokenIndex )(struct ANTLR3_COMMON_TOKEN_struct *token)
 Pointer to a function that returns an index 0.
ANTLR3_UINT32(* getType )(struct ANTLR3_COMMON_TOKEN_struct *token)
 Pointer to a function that returns the token type of this token.
ANTLR3_MARKER index
 What the index of this token is, 0, 1, .
pANTLR3_INPUT_STREAM input
 Pointer to the input stream that this token originated in.
ANTLR3_UINT32 line
 The line number in the input stream where this token was derived from.
void * lineStart
 The offset into the input stream that the line in which this token resides starts.
void(* setChannel )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 channel)
 Pointer to a function that sets the channel that this token should belong to.
void(* setCharPositionInLine )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_INT32 pos)
 Pointer to a function that sets the offset in the line where this token exists.
void(* setLine )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 line)
 Pointer to a function that sets the 'line' number where this token reside.
void(* setStartIndex )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER index)
 Pointer to a function that sets the start index in the input stream for this token.
void(* setStopIndex )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER index)
 Pointer to a function that sets the stop index in the input stream for this token.
void(* setText )(struct ANTLR3_COMMON_TOKEN_struct *token, pANTLR3_STRING text)
 Pointer to a function that 'might' be able to set the text associated with a token.
void(* setText8 )(struct ANTLR3_COMMON_TOKEN_struct *token, pANTLR3_UINT8 text)
 Pointer to a function that 'might' be able to set the text associated with a token.
void(* setTokenIndex )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER)
 Pointer to a function that can set the token index of this token in the token input stream.
void(* setType )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 ttype)
 Pointer to a function that sets the type of this token.
ANTLR3_MARKER start
 The character offset in the input stream where the text for this token starts.
ANTLR3_MARKER stop
 The character offset in the input stream where the text for this token stops.
pANTLR3_STRING_FACTORY strFactory
 A string factory that we can use if we ever need the text of a token and need to manufacture a pANTLR3_STRING.
ANTLR3_UINT32 textState
 Indicates whether this token is carrying:.
union {
   pANTLR3_UCHAR   chars
 Pointer that is used when the token just has a pointer to a char *, such as when a rewrite of an imaginary token supplies a string in the grammar.
   pANTLR3_STRING   text
 Some token types actually do carry around their associated text, hence (*getText)() will return this pointer if it is not NULL.
tokText
pANTLR3_STRING(* toString )(struct ANTLR3_COMMON_TOKEN_struct *token)
 Pointer to a function that returns this token as a text representation that can be printed with embedded control codes such as
replaced with the printable sequence "\\n" This also yields a string structure that can be used more easily than the pointer to the input stream in certain situations.
ANTLR3_UINT32 type
 The actual type of this token.
ANTLR3_UINT32 user1
 Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token.
ANTLR3_UINT32 user2
 Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token.
ANTLR3_UINT32 user3
 Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token.

Field Documentation

The virtual channel that this token exists in.

Referenced by emit(), getChannel(), and setChannel().

The character position in the line that this token was derived from.

Referenced by emit(), getCharPositionInLine(), and setCharPositionInLine().

Pointer that is used when the token just has a pointer to a char *, such as when a rewrite of an imaginary token supplies a string in the grammar.

No sense in constructing a pANTLR3_STRING just for that, as mostly the text will not be accessed - if it is, then we will build a pANTLR3_STRING for it a that point.

Referenced by antlr3CommonTreeNodeStreamNew(), createToken(), createTokenFromToken(), getText(), and setText8().

Pointer to a custom element that the ANTLR3 programmer may define and install.

Referenced by emit(), factoryClose(), and getMissingSymbol().

Indicates that a token was produced from the token factory and therefore the the freeToken() method should not do anything itself because token factory is responsible for deleting it.

Referenced by antlr3LexerNew(), antlr3TokenFactoryNew(), newPoolToken(), newToken(), and nextTokenStr().

Pointer to a function that knows how to free the custom structure when the token is destroyed.

Referenced by factoryClose().

Pointer to a function that gets the channel that this token was placed in (parsers can 'tune' to these channels.

Referenced by antlr3SetTokenAPI(), createTokenFromToken(), serializeToken(), and toString().

Pointer to a function that gets the offset in the line where this token exists.

Referenced by antlr3SetTokenAPI(), createTokenFromToken(), getCharPositionInLine(), getMissingSymbol(), serializeNode(), serializeToken(), and toString().

Pointer to a function that gets the 'line' number where this token resides.

Referenced by antlr3SetTokenAPI(), createTokenFromToken(), getLine(), getMissingSymbol(), serializeNode(), serializeToken(), and toString().

Pointer to a function that gets the start index in the input stream for this token.

Referenced by antlr3SetTokenAPI(), getText(), and toString().

Pointer to a function that gets the stop index in the input stream for this token.

Referenced by antlr3SetTokenAPI(), getText(), and toString().

Pointer to function that returns the text pointer of a token, use toString() if you want a pANTLR3_STRING version of the token.

Referenced by antlr3SetTokenAPI(), createTokenFromToken(), getMissingSymbol(), serializeToken(), toString(), and toStringSS().

Pointer to a function that returns an index 0.

..n-1 of the token in the token input stream.

Referenced by antlr3SetTokenAPI(), createTokenFromToken(), dbgSetTokenBoundaries(), serializeToken(), setTokenBoundaries(), toString(), and toStringTT().

Pointer to a function that returns the token type of this token.

Referenced by antlr3SetTokenAPI(), createTokenFromToken(), getMissingSymbol(), getTokensSet(), getType(), and serializeToken().

What the index of this token is, 0, 1, .

., n-2, n-1 tokens

Referenced by getTokenIndex(), and setTokenIndex().

Pointer to the input stream that this token originated in.

Referenced by antlr3RecognitionExceptionNew(), createToken(), getMissingSymbol(), getStartIndex(), getText(), newPoolToken(), and setInputStream().

The line number in the input stream where this token was derived from.

Referenced by emit(), getLine(), and setLine().

The offset into the input stream that the line in which this token resides starts.

Referenced by emit(), and getMissingSymbol().

Pointer to a function that sets the channel that this token should belong to.

Referenced by antlr3SetTokenAPI(), createTokenFromToken(), and getMissingSymbol().

Pointer to a function that sets the offset in the line where this token exists.

Referenced by antlr3SetTokenAPI(), createTokenFromToken(), and getMissingSymbol().

Pointer to a function that sets the 'line' number where this token reside.

Referenced by antlr3SetTokenAPI(), createTokenFromToken(), getMissingSymbol(), and nextTokenStr().

Pointer to a function that sets the start index in the input stream for this token.

Referenced by antlr3SetTokenAPI(), newPoolToken(), nextTokenStr(), and tokLT().

Pointer to a function that sets the stop index in the input stream for this token.

Referenced by antlr3SetTokenAPI(), nextTokenStr(), and tokLT().

Pointer to a function that 'might' be able to set the text associated with a token.

Imaginary tokens such as an ANTLR3_CLASSIC_TOKEN may actually do this, however many tokens such as ANTLR3_COMMON_TOKEN do not actaully have strings associated with them but just point into the current input stream. These tokens will implement this function with a function that errors out (probably drastically.

Referenced by antlr3SetTokenAPI().

Pointer to a function that 'might' be able to set the text associated with a token.

Imaginary tokens such as an ANTLR3_CLASSIC_TOKEN may actually do this, however many tokens such as ANTLR3_COMMON_TOKEN do not actully have strings associated with them but just point into the current input stream. These tokens will implement this function with a function that errors out (probably drastically.

Referenced by antlr3SetTokenAPI(), createTypeTokenText(), and getMissingSymbol().

Pointer to a function that can set the token index of this token in the token input stream.

Referenced by antlr3SetTokenAPI(), and createTokenFromToken().

The character offset in the input stream where the text for this token starts.

Referenced by emit(), getStartIndex(), and setStartIndex().

The character offset in the input stream where the text for this token stops.

Referenced by emit(), getStopIndex(), and setStopIndex().

A string factory that we can use if we ever need the text of a token and need to manufacture a pANTLR3_STRING.

Referenced by ANTLR3_TREE_ADAPTORNew(), antlr3CommonTreeNodeStreamNew(), antlr3CommonTreeNodeStreamNewStream(), antlr3LexerNew(), createToken(), getText(), newPoolToken(), setCharStream(), and setInputStream().

Some token types actually do carry around their associated text, hence (*getText)() will return this pointer if it is not NULL.

Referenced by createTokenFromToken(), emit(), getText(), setText(), and setText8().

Indicates whether this token is carrying:.

State | Meaning ------+-------------------------------------- 0 | Nothing (neither rewrite text, nor setText) 1 | char * to user supplied rewrite text 2 | pANTLR3_STRING because of setText or similar action

Affects the union structure tokText below (uses 32 bit so alignment is always good)

Referenced by antlr3CommonTreeNodeStreamNew(), createToken(), createTokenFromToken(), emit(), getText(), setText(), and setText8().

Pointer to a function that returns this token as a text representation that can be printed with embedded control codes such as
replaced with the printable sequence "\\n" This also yields a string structure that can be used more easily than the pointer to the input stream in certain situations.

Referenced by antlr3SetTokenAPI(), createTokenFromToken(), and displayRecognitionError().

The actual type of this token.

Referenced by displayRecognitionError(), emit(), getText(), getType(), nextToken(), setType(), and toString().

Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token.

This is one of them.

Referenced by emit(), and getMissingSymbol().

Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token.

This is one of them.

Referenced by emit(), and getMissingSymbol().

Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token.

This is one of them.

Referenced by emit(), and getMissingSymbol().


The documentation for this struct was generated from the following file:

Generated on Mon Nov 29 17:24:03 2010 for ANTLR3C by  doxygen 1.5.5