[antlr-interest] Using the TokenRewriteStream

Terence Parr parrt at cs.usfca.edu
Wed Aug 1 10:40:21 PDT 2007


On Jul 30, 2007, at 1:53 PM, Susan Jolly wrote:

> However, I'm probably going to need modify  
> TokenRewriteStream.toString() to
> use and return a buffer that is a List (of imaginary and real  
> Tokens) rather
> than a StringBuffer.

Yeah, I have some thoughts on token stream rewrite on the blog.

> One of the things I need to be able to do is to interline the input  
> (which
> is print text) with its braille translation.  ("Interline" means  
> alternating
> a line of print with a line of braille.) This requires adding line  
> breaks to
> the input so as to keep it in sync with the braille since each line  
> of the
> braille translation has a maximum length and must be terminated by  
> a hard
> return.  (The print lines typically have more characters than the  
> braille
> lines.) This alone I could do by estimating the length of a braille  
> line by
> totaling the String length of the real Tokens and then invoking  
> toString
> again with a smaller "end" index if the imaginary Tokens added by the
> Rewrite make the line too long. (I can't put the line breaks in at  
> a later
> point because I'd lose the synchronization with the input.)
>
> However, there are multiple ways to represent braille and a sighted  
> user
> might want more than one of them. In other words, the general case  
> is that
> the output consists of sets of lines where each set has one or more  
> of the
> following: a line of print, a line of standard braille (that  
> controls the
> line length), and possibly additional lines with other alternatives.
>
> This is (I think) easier to do if I associate each Token with an  
> object that
> has all these multiple alternatives which is why I want to get a  
> List of the
> Tokens rather than the concatenated text.
>
> The pictures at the tops of these two pages show what the output  
> looks like
> except that the first one doesn't include the original print text.
> http://www.dotlessbraille.org/screencap.htm
> http://www.dotlessbraille.org/gemini.htm

Interesting.  Well, a first pass to rewrite the token stream is a  
decent idea.  Then process the altered stream with another parser.
Ter


More information about the antlr-interest mailing list