ANTLR3_TOKEN_SOURCE_struct Struct Reference
#include <antlr3tokenstream.h>
Detailed Description
Definition of a token source, which has a pointer to a function that returns the next token (using a token factory if it is going to be efficient) and a pointer to an ANTLR3_INPUT_STREAM.This is slightly different to the Java interface because we have no way to implement multiple interfaces without defining them in the interface structure or casting (void *), which is too convoluted.
Data Fields | |
| ANTLR3_COMMON_TOKEN | eofToken |
| A special pre-allocated token, which signifies End Of Tokens. | |
| pANTLR3_STRING | fileName |
| When the token source is constructed, it is populated with the file name from whence the tokens were produced by the lexer. | |
| pANTLR3_COMMON_TOKEN(* | nextToken )(struct ANTLR3_TOKEN_SOURCE_struct *tokenSource) |
| Pointer to a function that returns the next token in the stream. | |
| ANTLR3_COMMON_TOKEN | skipToken |
| A special pre-allocated token, which is returned by mTokens() if the lexer rule said to just skip the generated token altogether. | |
| pANTLR3_STRING_FACTORY | strFactory |
| Whoever is providing tokens, needs to provide a string factory too. | |
| void * | super |
| Whatever is supplying the token source interface, needs a pointer to itself so that this pointer can be passed to it when the nextToken function is called. | |
Field Documentation
A special pre-allocated token, which signifies End Of Tokens.
Because this must be set up with the current input index and so on, we embed the structure and return the address of it. It is marked as factoryMade, so that it is never attempted to be freed.
Referenced by antlr3LexerNew(), nextTokenStr(), setCharStream(), and tokLT().
When the token source is constructed, it is populated with the file name from whence the tokens were produced by the lexer.
This pointer is a copy of the one supplied by the CharStream (and may be NULL) so should not be manipulated other than to copy or print it.
Referenced by setCharStream().
| pANTLR3_COMMON_TOKEN(* ANTLR3_TOKEN_SOURCE_struct::nextToken)(struct ANTLR3_TOKEN_SOURCE_struct *tokenSource) |
A special pre-allocated token, which is returned by mTokens() if the lexer rule said to just skip the generated token altogether.
Having this single token stops us wasting memory by have the token factory actually create something that we are going to SKIP(); anyway.
Referenced by antlr3LexerNew(), and nextTokenStr().
Whoever is providing tokens, needs to provide a string factory too.
Referenced by antlr3LexerNew(), setCharStream(), and toStringSS().
Whatever is supplying the token source interface, needs a pointer to itself so that this pointer can be passed to it when the nextToken function is called.
Referenced by antlr3LexerNew(), nextToken(), and nextTokenStr().
The documentation for this struct was generated from the following file:
- include/antlr3tokenstream.h
