[antlr-interest] Re: Stream Positions

Trey Spiva Trey.Spiva at embarcadero.com
Mon Jan 7 14:46:10 PST 2002


>> I was not able to just override the consume() method because the
>> lexer can backtrack adn teh same character position may be visted
>> many times.
>
>Yeah, I was wondering about that...did you have to override rewind in
>the buffers?  Is that it?
 No, when InputBuffer::fill is called I retrieved the character as well as
the file position.  I then
stored both the character and the file position on to the character queue.
Of course I had to 
make the queue take a structure not a character

class CharacterData
   {
   public:
      CharacterData(int c, long fp)
      {
         m_character = c;
         m_filepos = fp;
      }

      int getCharacter() {return m_character;}
      int getFilePosition() {return m_filepos;}
   private:
      int m_character;
      long m_filepos;
   };

CircularQueue<CharacterData> queue;

Now InputBuffer::LA(int I) looks like this.
inline int InputBuffer::LA(int i)
{
	fill(i);

   CharacterData data = queue.elementAt(markerOffset + i - 1);
   return data.getCharacter();

	//return queue.elementAt(markerOffset + i - 1);
}

I also added a getPosition method to InputBuffer.

/** Get the file position of a lookahead character */
inline long InputBuffer::getPosition(int i)
{
   fill(i);

   CharacterData data = queue.elementAt(markerOffset + i - 1);
   return data.getFilePosition();
}

I then added the position data to the input state inside the
CharScanner::comsume method.
>>> I had to alter CharBuffer, CharScanner, CommonToken, InputBuffer,
>>> LexerSharedInputState, Token
>>>


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 



More information about the antlr-interest mailing list