ANTLR3_TOKEN_STREAM_struct Struct Reference
#include <antlr3tokenstream.h>
Detailed Description
Definition of the ANTLR3 common token stream interface.
- Remarks:
- Much of the documentation for this interface is stolen from Ter's Java implementation.
Data Fields | |
| pANTLR3_COMMON_TOKEN(* | _LT )(struct ANTLR3_TOKEN_STREAM_struct *tokenStream, ANTLR3_INT32 k) |
| Get Token at current input pointer + i ahead where i=1 is next Token. | |
| pANTLR3_DEBUG_EVENT_LISTENER | debugger |
| Debugger interface, is this is a debugging token stream. | |
| void(* | free )(struct ANTLR3_TOKEN_STREAM_struct *tokenStream) |
| Function that knows how to free the memory for an ANTLR3_TOKEN_STREAM. | |
| pANTLR3_COMMON_TOKEN(* | get )(struct ANTLR3_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 i) |
| Get a token at an absolute index i; 0. | |
| pANTLR3_TOKEN_SOURCE(* | getTokenSource )(struct ANTLR3_TOKEN_STREAM_struct *tokenStream) |
| Where is this stream pulling tokens from? This is not the name, but a pointer into an interface that contains a ANTLR3_TOKEN_SOURCE interface. | |
| ANTLR3_BOOLEAN | initialStreamState |
| Indicates the initial stream state for dbgConsume(). | |
| pANTLR3_INT_STREAM | istream |
| All input streams implement the ANTLR3_INT_STREAM interface. | |
| void(* | setDebugListener )(struct ANTLR3_TOKEN_STREAM_struct *tokenStream, pANTLR3_DEBUG_EVENT_LISTENER debugger) |
| Function that sets the token stream into debugging mode. | |
| void(* | setTokenSource )(struct ANTLR3_TOKEN_STREAM_struct *tokenStream, pANTLR3_TOKEN_SOURCE tokenSource) |
| Function that installs a token source for teh stream. | |
| void * | super |
| Whatever is providing this interface needs a pointer to itself so that this can be passed back to it whenever the api functions are called. | |
| pANTLR3_TOKEN_SOURCE | tokenSource |
| Pointer to the token source for this stream. | |
| pANTLR3_STRING(* | toString )(struct ANTLR3_TOKEN_STREAM_struct *tokenStream) |
| Return the text of all the tokens in the stream, as the old tramp in Leeds market used to say; "Get the lot!". | |
| pANTLR3_STRING(* | toStringSS )(struct ANTLR3_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop) |
| Return the text of all tokens from start to stop, inclusive. | |
| pANTLR3_STRING(* | toStringTT )(struct ANTLR3_TOKEN_STREAM_struct *tokenStream, pANTLR3_COMMON_TOKEN start, pANTLR3_COMMON_TOKEN stop) |
| Because the user is not required to use a token with an index stored in it, we must provide a means for two token objects themselves to indicate the start/end location. | |
Field Documentation
| pANTLR3_COMMON_TOKEN(* ANTLR3_TOKEN_STREAM_struct::_LT)(struct ANTLR3_TOKEN_STREAM_struct *tokenStream, ANTLR3_INT32 k) |
Get Token at current input pointer + i ahead where i=1 is next Token.
i<0 indicates tokens in the past. So -1 is previous token and -2 is two tokens ago. LT(0) is undefined. For i>=n, return Token.EOFToken. Return null for LT(0) and any index that results in an absolute address that is negative.
Referenced by antlr3CommonTokenDebugStreamSourceNew(), antlr3CommonTokenStreamNew(), antlr3RecognitionExceptionNew(), dbgConsume(), getMissingSymbol(), and setDebugListener().
Debugger interface, is this is a debugging token stream.
Referenced by antlr3CommonTokenDebugStreamSourceNew(), consumeInitialHiddenTokens(), dbgConsume(), and setDebugListener().
| void(* ANTLR3_TOKEN_STREAM_struct::free)(struct ANTLR3_TOKEN_STREAM_struct *tokenStream) |
Function that knows how to free the memory for an ANTLR3_TOKEN_STREAM.
Referenced by antlr3CTSFree(), and antlr3TokenStreamNew().
| pANTLR3_COMMON_TOKEN(* ANTLR3_TOKEN_STREAM_struct::get)(struct ANTLR3_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 i) |
Get a token at an absolute index i; 0.
.n-1. This is really only needed for profiling and debugging and token stream rewriting. If you don't want to buffer up tokens, then this method makes no sense for you. Naturally you can't use the rewrite stream feature. I believe DebugTokenStream can easily be altered to not use this method, removing the dependency.
Referenced by antlr3CommonTokenStreamNew(), consumeInitialHiddenTokens(), dbgConsume(), getTokensSet(), and toStringSS().
| pANTLR3_TOKEN_SOURCE(* ANTLR3_TOKEN_STREAM_struct::getTokenSource)(struct ANTLR3_TOKEN_STREAM_struct *tokenStream) |
Where is this stream pulling tokens from? This is not the name, but a pointer into an interface that contains a ANTLR3_TOKEN_SOURCE interface.
The Token Source interface contains a pointer to the input stream and a pointer to a function that returns the next token.
Referenced by antlr3CommonTokenStreamNew(), and toStringSS().
Indicates the initial stream state for dbgConsume().
Referenced by antlr3CommonTokenDebugStreamSourceNew(), consumeInitialHiddenTokens(), dbgConsume(), dbgTokLT(), and setDebugListener().
All input streams implement the ANTLR3_INT_STREAM interface.
..
Referenced by alreadyParsedRule(), antlr3CommonTokenDebugStreamSourceNew(), antlr3CommonTokenStreamNew(), antlr3CTSFree(), antlr3RecognitionExceptionNew(), consumeUntil(), consumeUntilSet(), dbgTokLT(), displayRecognitionError(), getTokensSet(), match(), matchAny(), memoize(), mismatch(), recover(), recoverFromMismatchedElement(), recoverFromMismatchedSet(), recoverFromMismatchedToken(), setDebugListener(), synpred(), tokLT(), toString(), and toStringSS().
| void(* ANTLR3_TOKEN_STREAM_struct::setDebugListener)(struct ANTLR3_TOKEN_STREAM_struct *tokenStream, pANTLR3_DEBUG_EVENT_LISTENER debugger) |
Function that sets the token stream into debugging mode.
Referenced by antlr3CommonTokenStreamNew(), and setDebugListener().
| void(* ANTLR3_TOKEN_STREAM_struct::setTokenSource)(struct ANTLR3_TOKEN_STREAM_struct *tokenStream, pANTLR3_TOKEN_SOURCE tokenSource) |
Function that installs a token source for teh stream.
Referenced by antlr3CommonTokenStreamNew(), and antlr3CommonTokenStreamSourceNew().
Whatever is providing this interface needs a pointer to itself so that this can be passed back to it whenever the api functions are called.
Referenced by antlr3CommonTokenStreamNew(), antlr3CTSFree(), antlr3RecognitionExceptionNew(), consume(), get(), getMissingSymbol(), tokLT(), toString(), and toStringSS().
Pointer to the token source for this stream.
Referenced by getTokenSource(), setTokenSource(), and tokLT().
| pANTLR3_STRING(* ANTLR3_TOKEN_STREAM_struct::toString)(struct ANTLR3_TOKEN_STREAM_struct *tokenStream) |
Return the text of all the tokens in the stream, as the old tramp in Leeds market used to say; "Get the lot!".
Referenced by antlr3CommonTokenStreamNew().
| pANTLR3_STRING(* ANTLR3_TOKEN_STREAM_struct::toStringSS)(struct ANTLR3_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop) |
Return the text of all tokens from start to stop, inclusive.
If the stream does not buffer all the tokens then it can just return an empty ANTLR3_STRING or NULL; Grammars should not access $ruleLabel.text in an action in that case.
Referenced by antlr3CommonTokenStreamNew(), toString(), and toStringTT().
| pANTLR3_STRING(* ANTLR3_TOKEN_STREAM_struct::toStringTT)(struct ANTLR3_TOKEN_STREAM_struct *tokenStream, pANTLR3_COMMON_TOKEN start, pANTLR3_COMMON_TOKEN stop) |
Because the user is not required to use a token with an index stored in it, we must provide a means for two token objects themselves to indicate the start/end location.
Most often this will just delegate to the other toString(int,int). This is also parallel with the pTREENODE_STREAM->toString(Object,Object).
Referenced by antlr3CommonTokenStreamNew().
The documentation for this struct was generated from the following file:
- include/antlr3tokenstream.h
