Packageorg.antlr.runtime
Classpublic class TokenRewriteStream
InheritanceTokenRewriteStream Inheritance CommonTokenStream

Useful for dumping out the input stream after doing some augmentation or other manipulations. You can insert stuff, replace, and delete chunks. Note that the operations are done lazily--only if you convert the buffer to a String. This is very efficient because you are not moving data around all the time. As the buffer of tokens is converted to strings, the toString() method(s) check to see if there is an operation at the current index. If so, the operation is done and then normal String rendering continues on the buffer. This is like having multiple Turing machine instruction streams (programs) operating on a single input tape. :) Since the operations are done lazily at toString-time, operations do not screw up the token index values. That is, an insert operation at token index i does not change the index values for tokens i+1..n-1. Because operations never actually alter the buffer, you may always get the original token stream back without undoing anything. Since the instructions are queued up, you can easily simulate transactions and roll back any changes if there is an error just by removing instructions. For example, var input:CharStream = new ANTLRFileStream("input"); var lex:TLexer = new TLexer(input); var tokens:TokenRewriteStream = new TokenRewriteStream(lex); var parser:T = new T(tokens); parser.startRule(); Then in the rules, you can execute var t:Token t, u:Token; ... input.insertAfter(t, "text to put after t");} input.insertAfter(u, "text after u");} trace(tokens.toString()); Actually, you have to cast the 'input' to a TokenRewriteStream. :( You can also have multiple "instruction streams" and get multiple rewrites from a single pass over the input. Just name the instruction streams and use that name again when printing the buffer. This could be useful for generating a C file and also its header file--all from the same buffer: tokens.insertAfter("pass1", t, "text to put after t");} tokens.insertAfter("pass2", u, "text after u");} trace(tokens.toString("pass1")); trace(tokens.toString("pass2")); If you don't use named rewrite streams, a "default" stream is used as the first example shows.


Public Properties
 PropertyDefined by
 Inheritedindex : int
CommonTokenStream
 Inheritedsize : int
CommonTokenStream
 InheritedsourceName : String
CommonTokenStream
 InheritedtokenSource : TokenSource
CommonTokenStream
Protected Properties
 PropertyDefined by
 Inheritedchannel : int = 0
Skip tokens on any channel but this one; this is how we skip whitespace...
CommonTokenStream
 InheritedchannelOverrideMap : Array
Map to override some Tokens' channel numbers
CommonTokenStream
 Inherited_discardOffChannelTokens : Boolean = false
By default, track all incoming tokens
CommonTokenStream
 InheriteddiscardSet : Array
Set; discard any tokens with this type
CommonTokenStream
 InheritedlastMarker : int
Track the last mark() call result value for use in rewind().
CommonTokenStream
  lastRewriteTokenIndexes : Object
Map String (program name) -> Integer index
TokenRewriteStream
 Inheritedp : int = -1
The index into the tokens list of the current token (next token to consume).
CommonTokenStream
  programs : Object
You may have multiple, named streams of rewrite operations.
TokenRewriteStream
 Inheritedtokens : Array
Record every single token pulled from the source so we can reproduce chunks of it later.
CommonTokenStream
 Inherited_tokenSource : TokenSource
CommonTokenStream
Public Methods
 MethodDefined by
  
TokenRewriteStream(tokenSource:TokenSource = null, channel:int)
TokenRewriteStream
 Inherited
consume():void
Move the input pointer to the next incoming token.
CommonTokenStream
  
deleteProgram(programName:String):void
Reset the program so that no instructions exist
TokenRewriteStream
 Inherited
discardOffChannelTokens(discardOffChannelTokens:Boolean):void
CommonTokenStream
 Inherited
discardTokenType(ttype:int):void
CommonTokenStream
  
getLastRewriteTokenIndex(programName:String):int
TokenRewriteStream
 Inherited
getToken(i:int):Token
Return absolute token i; ignore which channel the tokens are on; that is, count all tokens not just on-channel tokens.
CommonTokenStream
 Inherited
getTokens():Array
CommonTokenStream
 Inherited
getTokensArray(start:int, stop:int, types:Array):Array
CommonTokenStream
 Inherited
getTokensBitSet(start:int, stop:int, types:BitSet):Array
Given a start and stop index, return a List of all tokens in the token type BitSet.
CommonTokenStream
 Inherited
getTokensInt(start:int, stop:int, ttype:int):Array
CommonTokenStream
 Inherited
getTokensRange(start:int, stop:int):Array
CommonTokenStream
  
insertAfter(index:int, text:Object, programName:String):void
TokenRewriteStream
  
insertAfterToken(t:Token, text:Object, programName:String):void
TokenRewriteStream
  
insertBefore(index:int, text:Object, programName:String):void
TokenRewriteStream
  
insertBeforeToken(t:Token, text:Object, programName:String):void
TokenRewriteStream
 Inherited
LA(i:int):int
CommonTokenStream
 Inherited
LT(k:int):Token
Get the ith token from the current position 1..n where k=1 is the first symbol of lookahead.
CommonTokenStream
 Inherited
mark():int
CommonTokenStream
 Inherited
release(marker:int):void
CommonTokenStream
  
remove(index:int, programName:String):void
TokenRewriteStream
  
removeRange(fromIndex:int, toIndex:int, programName:String):void
TokenRewriteStream
  
removeToken(token:Token, programName:String):void
TokenRewriteStream
  
removeTokenRange(fromToken:Token, toToken:Token, programName:String):void
TokenRewriteStream
  
replace(index:int, text:Object, programName:String):void
TokenRewriteStream
  
replaceRange(fromIndex:int, toIndex:int, text:Object, programName:String):void
TokenRewriteStream
  
replaceToken(indexT:Token, text:Object, programName:String):void
TokenRewriteStream
  
replaceTokenRange(fromToken:Token, toToken:Token, text:Object, programName:String):void
TokenRewriteStream
 Inherited
reset():void
CommonTokenStream
 Inherited
rewind():void
CommonTokenStream
 Inherited
rewindTo(marker:int):void
CommonTokenStream
  
rollback(instructionIndex:int, programName:String):void
Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream.
TokenRewriteStream
 Inherited
seek(index:int):void
CommonTokenStream
 Inherited
setTokenTypeChannel(ttype:int, channel:int):void
A simple filter mechanism whereby you can tell this token stream to force all tokens of type ttype to be on channel.
CommonTokenStream
  
toDebugString():String
TokenRewriteStream
  
toDebugStringWithRange(start:int, end:int):String
TokenRewriteStream
  
toOriginalString():String
TokenRewriteStream
  
toOriginalStringWithRange(start:int, end:int):String
TokenRewriteStream
  
toString():String
TokenRewriteStream
  
toStringWithRange(start:int, end:int):String
TokenRewriteStream
  
toStringWithRangeAndProgram(start:int, end:int, programName:String):String
TokenRewriteStream
 Inherited
toStringWithTokenRange(start:Token, stop:Token):String
CommonTokenStream
Protected Methods
 MethodDefined by
  
catOpText(a:Object, b:Object):String
TokenRewriteStream
 Inherited
fillBuffer():void
Load all tokens from the token source and put in tokens.
CommonTokenStream
  
getKindOfOps(rewrites:Array, kind:Class, before:int = -1):Array
Get all operations before an index of a particular kind
TokenRewriteStream
  
getProgram(name:String):Array
TokenRewriteStream
 Inherited
LB(k:int):Token
Look backwards k tokens on-channel tokens
CommonTokenStream
  
reduceToSingleOperationPerIndex(rewrites:Array):Array
We need to combine operations and report invalid operations (like overlapping replaces that are not completed nested).
TokenRewriteStream
  
setLastRewriteTokenIndex(programName:String, i:int):void
TokenRewriteStream
 Inherited
skipOffTokenChannels(i:int):int
Given a starting index, return the index of the first on-channel token.
CommonTokenStream
 Inherited
skipOffTokenChannelsReverse(i:int):int
CommonTokenStream
Public Constants
 ConstantDefined by
  DEFAULT_PROGRAM_NAME : String = "default"
[static]
TokenRewriteStream
  MIN_TOKEN_INDEX : int = 0
[static]
TokenRewriteStream
Property detail
lastRewriteTokenIndexesproperty
protected var lastRewriteTokenIndexes:Object

Map String (program name) -> Integer index

programsproperty 
protected var programs:Object

You may have multiple, named streams of rewrite operations. I'm calling these things "programs." Maps String (name) -> rewrite (List)

Constructor detail
TokenRewriteStream()constructor
public function TokenRewriteStream(tokenSource:TokenSource = null, channel:int)Parameters
tokenSource:TokenSource (default = null)
 
channel:int
Method detail
catOpText()method
protected function catOpText(a:Object, b:Object):StringParameters
a:Object
 
b:Object

Returns
String
deleteProgram()method 
public function deleteProgram(programName:String):void

Reset the program so that no instructions exist

Parameters
programName:String
getKindOfOps()method 
protected function getKindOfOps(rewrites:Array, kind:Class, before:int = -1):Array

Get all operations before an index of a particular kind

Parameters
rewrites:Array
 
kind:Class
 
before:int (default = -1)

Returns
Array
getLastRewriteTokenIndex()method 
public function getLastRewriteTokenIndex(programName:String):intParameters
programName:String

Returns
int
getProgram()method 
protected function getProgram(name:String):ArrayParameters
name:String

Returns
Array
insertAfter()method 
public function insertAfter(index:int, text:Object, programName:String):voidParameters
index:int
 
text:Object
 
programName:String
insertAfterToken()method 
public function insertAfterToken(t:Token, text:Object, programName:String):voidParameters
t:Token
 
text:Object
 
programName:String
insertBefore()method 
public function insertBefore(index:int, text:Object, programName:String):voidParameters
index:int
 
text:Object
 
programName:String
insertBeforeToken()method 
public function insertBeforeToken(t:Token, text:Object, programName:String):voidParameters
t:Token
 
text:Object
 
programName:String
reduceToSingleOperationPerIndex()method 
protected function reduceToSingleOperationPerIndex(rewrites:Array):Array

We need to combine operations and report invalid operations (like overlapping replaces that are not completed nested). Inserts to same index need to be combined etc... Here are the cases: I.i.u I.j.v leave alone, nonoverlapping I.i.u I.i.v combine: Iivu R.i-j.u R.x-y.v | i-j in x-y delete first R R.i-j.u R.i-j.v delete first R R.i-j.u R.x-y.v | x-y in i-j ERROR R.i-j.u R.x-y.v | boundaries overlap ERROR I.i.u R.x-y.v | i in x-y delete I I.i.u R.x-y.v | i not in x-y leave alone, nonoverlapping R.x-y.v I.i.u | i in x-y ERROR R.x-y.v I.x.u R.x-y.uv (combine, delete I) R.x-y.v I.i.u | i not in x-y leave alone, nonoverlapping I.i.u = insert u before op

Parameters
rewrites:Array

Returns
Array
remove()method 
public function remove(index:int, programName:String):voidParameters
index:int
 
programName:String
removeRange()method 
public function removeRange(fromIndex:int, toIndex:int, programName:String):voidParameters
fromIndex:int
 
toIndex:int
 
programName:String
removeToken()method 
public function removeToken(token:Token, programName:String):voidParameters
token:Token
 
programName:String
removeTokenRange()method 
public function removeTokenRange(fromToken:Token, toToken:Token, programName:String):voidParameters
fromToken:Token
 
toToken:Token
 
programName:String
replace()method 
public function replace(index:int, text:Object, programName:String):voidParameters
index:int
 
text:Object
 
programName:String
replaceRange()method 
public function replaceRange(fromIndex:int, toIndex:int, text:Object, programName:String):voidParameters
fromIndex:int
 
toIndex:int
 
text:Object
 
programName:String
replaceToken()method 
public function replaceToken(indexT:Token, text:Object, programName:String):voidParameters
indexT:Token
 
text:Object
 
programName:String
replaceTokenRange()method 
public function replaceTokenRange(fromToken:Token, toToken:Token, text:Object, programName:String):voidParameters
fromToken:Token
 
toToken:Token
 
text:Object
 
programName:String
rollback()method 
public function rollback(instructionIndex:int, programName:String):void

Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream. UNTESTED!

Parameters
instructionIndex:int
 
programName:String
setLastRewriteTokenIndex()method 
protected function setLastRewriteTokenIndex(programName:String, i:int):voidParameters
programName:String
 
i:int
toDebugString()method 
public function toDebugString():String

Returns
String
toDebugStringWithRange()method 
public function toDebugStringWithRange(start:int, end:int):StringParameters
start:int
 
end:int

Returns
String
toOriginalString()method 
public function toOriginalString():String

Returns
String
toOriginalStringWithRange()method 
public function toOriginalStringWithRange(start:int, end:int):StringParameters
start:int
 
end:int

Returns
String
toString()method 
public override function toString():String

Returns
String
toStringWithRange()method 
public override function toStringWithRange(start:int, end:int):StringParameters
start:int
 
end:int

Returns
String
toStringWithRangeAndProgram()method 
public function toStringWithRangeAndProgram(start:int, end:int, programName:String):StringParameters
start:int
 
end:int
 
programName:String

Returns
String
Constant detail
DEFAULT_PROGRAM_NAMEconstant
public static const DEFAULT_PROGRAM_NAME:String = "default"
MIN_TOKEN_INDEXconstant 
public static const MIN_TOKEN_INDEX:int = 0