The most common stream of tokens is one where every token is buffered up
and tokens are prefiltered for a certain channel (the parser will only
see these tokens and cannot change the filter channel number during the
parse).
TODO: how to access the full token stream? How to track all tokens matched per rule?
protected var channel:int = 0
Skip tokens on any channel but this one; this is how we skip whitespace...
protected var channelOverrideMap:Array
Map to override some Tokens' channel numbers
protected var _discardOffChannelTokens:Boolean = false
By default, track all incoming tokens
protected var discardSet:Array
Set; discard any tokens with this type
index:int
[read-only]Implementation
public function get index():int
protected var lastMarker:int
Track the last mark() call result value for use in rewind().
protected var p:int = -1
The index into the tokens list of the current token (next token
to consume). p==-1 indicates that the tokens list is empty
size:int
[read-only]Implementation
public function get size():int
sourceName:String
[read-only]Implementation
public function get sourceName():String
protected var tokens:Array
Record every single token pulled from the source so we can reproduce
chunks of it later.
protected var _tokenSource:TokenSource
tokenSource:TokenSource
[read-write]
Implementation
public function get tokenSource():TokenSource
public function set tokenSource(value:TokenSource):void
public function CommonTokenStream(tokenSource:TokenSource = null, channel:int)
Parameters
public function consume():void
Move the input pointer to the next incoming token. The stream
must become active with LT(1) available. consume() simply
moves the input pointer so that LT(1) points at the next
input symbol. Consume at least one token.
Walk past any token not on the channel the parser is listening to.
public function discardOffChannelTokens(discardOffChannelTokens:Boolean):void
Parameters
| discardOffChannelTokens:Boolean |
public function discardTokenType(ttype:int):void
Parameters
protected function fillBuffer():void
Load all tokens from the token source and put in tokens.
This is done upon first LT request because you might want to
set some token type / channel overrides before filling buffer.
public function getToken(i:int):Token
Return absolute token i; ignore which channel the tokens are on;
that is, count all tokens not just on-channel tokens.
Parameters
Returns
public function getTokens():Array
Returns
public function getTokensArray(start:int, stop:int, types:Array):Array
Parameters
| start:int |
|
| stop:int |
|
| types:Array |
Returns
public function getTokensBitSet(start:int, stop:int, types:BitSet):Array
Given a start and stop index, return a List of all tokens in
the token type BitSet. Return null if no tokens were found. This
method looks at both on and off channel tokens.
Renamed from getTokens
Parameters
| start:int |
|
| stop:int |
|
| types:BitSet |
Returns
public function getTokensInt(start:int, stop:int, ttype:int):Array
Parameters
| start:int |
|
| stop:int |
|
| ttype:int |
Returns
public function getTokensRange(start:int, stop:int):Array
Parameters
Returns
public function LA(i:int):int
Parameters
Returns
protected function LB(k:int):Token
Look backwards k tokens on-channel tokens
Parameters
Returns
public function LT(k:int):Token
Get the ith token from the current position 1..n where k=1 is the
first symbol of lookahead.
Parameters
Returns
public function mark():int
Returns
public function release(marker:int):void
Parameters
public function reset():void
public function rewind():void
public function rewindTo(marker:int):void
Parameters
public function seek(index:int):void
Parameters
public function setTokenTypeChannel(ttype:int, channel:int):void
A simple filter mechanism whereby you can tell this token stream
to force all tokens of type ttype to be on channel. For example,
when interpreting, we cannot exec actions so we need to tell
the stream to force all WS and NEWLINE to be a different, ignored
channel.
Parameters
protected function skipOffTokenChannels(i:int):int
Given a starting index, return the index of the first on-channel
token.
Parameters
Returns
protected function skipOffTokenChannelsReverse(i:int):int
Parameters
Returns
public function toString():String
Returns
public function toStringWithRange(start:int, stop:int):String
Parameters
Returns
public function toStringWithTokenRange(start:Token, stop:Token):String
Parameters
Returns