All token streams should provide their tokens using an instance of this common token. A custom pointer is provided, wher you may attach a further structure to enhance the common token if you feel the need to do so. The C runtime will assume that a token provides implementations of the interface functions, but all of them may be rplaced by your own implementation if you require it.
#include <antlr3defs.h>
Data Structures | |
struct | ANTLR3_COMMON_TOKEN_struct |
The definition of an ANTLR3 common token structure, which all implementations of a token stream should provide, installing any further structures in the custom pointer element of this structure. More... | |
struct | ANTLR3_TOKEN_FACTORY_struct |
ANTLR3 Token factory interface to create lots of tokens efficiently rather than creating and freeing lots of little bits of memory. More... | |
Defines | |
#define | ANTLR3_EOR_TOKEN_TYPE 1 |
#define | ANTLR3_FACTORY_POOL_SIZE 1024 |
How many tokens to allocate at once in the token factory. | |
#define | ANTLR3_MIN_TOKEN_TYPE ANTLR3_TOKEN_UP + 1 |
First token that can be used by users/generated code. | |
#define | ANTLR3_TEXT_CHARP 1 |
#define | ANTLR3_TEXT_NONE 0 |
#define | ANTLR3_TEXT_STRING 2 |
#define | ANTLR3_TOKEN_DEFAULT_CHANNEL 0 |
Default channel for a token. | |
#define | ANTLR3_TOKEN_DOWN 2 |
Imaginary token type to cause a traversal of child nodes in a tree parser. | |
#define | ANTLR3_TOKEN_EOF (ANTLR3_CHARSTREAM_EOF & 0xFFFFFFFF) |
End of file token. | |
#define | ANTLR3_TOKEN_INVALID 0 |
Indicator of an invalid token. | |
#define | ANTLR3_TOKEN_UP 3 |
Imaginary token type to signal the end of a stream of child nodes. | |
#define | HIDDEN 99 |
Reserved channel number for a HIDDEN token - a token that is hidden from the parser. | |
Typedefs | |
typedef struct ANTLR3_COMMON_TOKEN_struct | ANTLR3_COMMON_TOKEN |
typedef struct ANTLR3_TOKEN_FACTORY_struct | ANTLR3_TOKEN_FACTORY |
#define ANTLR3_EOR_TOKEN_TYPE 1 |
Referenced by combineFollows(), mismatchIsMissingToken(), and recoverFromMismatchedElement().
#define ANTLR3_FACTORY_POOL_SIZE 1024 |
How many tokens to allocate at once in the token factory.
Referenced by factoryClose(), newPool(), newPoolToken(), and newPoolTree().
#define ANTLR3_MIN_TOKEN_TYPE ANTLR3_TOKEN_UP + 1 |
First token that can be used by users/generated code.
#define ANTLR3_TEXT_CHARP 1 |
Referenced by antlr3CommonTreeNodeStreamNew(), createToken(), createTokenFromToken(), getText(), and setText8().
#define ANTLR3_TEXT_NONE 0 |
Referenced by emit(), and setText8().
#define ANTLR3_TEXT_STRING 2 |
Referenced by createTokenFromToken(), emit(), getText(), and setText().
#define ANTLR3_TOKEN_DEFAULT_CHANNEL 0 |
Default channel for a token.
Referenced by antlr3CommonTokenStreamSourceNew(), getMissingSymbol(), nextTokenStr(), reset(), and toString().
#define ANTLR3_TOKEN_DOWN 2 |
Imaginary token type to cause a traversal of child nodes in a tree parser.
Referenced by antlr3CommonTreeNodeStreamNew(), and fillBuffer().
#define ANTLR3_TOKEN_EOF (ANTLR3_CHARSTREAM_EOF & 0xFFFFFFFF) |
End of file token.
Referenced by antlr3CommonTreeNodeStreamNew(), antlr3dfapredict(), antlr3LexerNew(), antlr3RecognitionExceptionNew(), consumeUntil(), consumeUntilSet(), displayRecognitionError(), getMissingSymbol(), getText(), and nextToken().
#define ANTLR3_TOKEN_INVALID 0 |
Indicator of an invalid token.
Referenced by antlr3CommonTreeNodeStreamNew(), antlr3LexerNew(), errorNode(), recoverFromMismatchedSet(), and reset().
#define ANTLR3_TOKEN_UP 3 |
Imaginary token type to signal the end of a stream of child nodes.
Referenced by antlr3CommonTreeNodeStreamNew(), and fillBuffer().
#define HIDDEN 99 |
Reserved channel number for a HIDDEN token - a token that is hidden from the parser.
typedef struct ANTLR3_COMMON_TOKEN_struct ANTLR3_COMMON_TOKEN |
typedef struct ANTLR3_TOKEN_FACTORY_struct ANTLR3_TOKEN_FACTORY |