[antlr-interest] antlr python unit testing

Laurie Harper laurie at holoweb.net
Thu May 3 22:51:43 PDT 2007


Hmm, I tried this but it only works if I re-assign stdin before creating 
the lexer, which makes sense of course, but isn't really what I want.

I'm building an expression parser as part of a page templating system. 
It will be used to parse an arbitrary number of expressions within the 
template, each representing a dynamic reference to external data. So, I 
need to be able to invoke the parser repeatedly on varying inputs.

That's the same behaviour I was looking for in order to unit test the 
parser/lexer, hence this thread. The solution I came up with looks 
something like:

class MockReader(object):
   def setInput(self, text):
     self.text = text
     self.index = 0

   def read(count):
     start = self.index
     end = self.index + count
     self.index = end

     return self.text[start, end]

class Test(unittest.TestCase):
   def setUp(self):
     self.L = Lexer()
     self.P = Parser(self.L)

   def test_first(self):
     src = MockReader()

     src.setInput('test expr 1')
     self.L.setInput(src)
     self.P.someRule()

     src.setInput('test expr 2')
     self.L.setInput(src)
     self.P.someRule()

     ...

This seemed to be working, until I tried calling different rules in the 
same test case. It looks like calling setInput() on the lexer leaves the 
lexer and/or parser in an inconsistent state :-(

So, what is the general solution to re-using a parser instance with 
different string inputs? Any suggestions would be greatly appreciated.

L.

Kaleb Pederson wrote:
> Hi Laurie,
> 
> In python, stdin is just a file descriptor that you can change.  So, if you 
> wanted to manipulate it to read from a file, or some other object that had 
> the same interface as a file descriptor, you could do so.
> 
> For example:
> 
> import sys
> sys.stdin = open('myfile','r');
> 
> Then, if you needed to get the regular stdin back, sys.stdin = sys.__stdin__
> 
> So, take advantage of the StringIO class:
> 
> import StringIO
> import sys
> 
> sys.stdin = StringIO.StringIO("My text to be parsed")
> 
> Now you can run your unit tests using whatever text you want.
> 
> Hope that helps.
> 
> --Kaleb
> 
> On Thursday 03 May 2007, Laurie Harper wrote:
>> I'm using ANTLR for Python and have my lexer and parser partially
>> working. I would like to start writing unit tests to identify exactly
>> what is and isn't right in what I have so far. (Call it test-first
>> debugging ;-)
>>
>> This is a total newbie question but, how do I pass input into my lexer
>> or parser from a Python script, rather than having it come from stdin?
>> Ideally, I'd like to create a lexer instance and a parser instance and
>> then write unit tests that exercise specific rules with varying input,
>> al la:
>>
>>      import unittest
>>      class LexerTests(unittest.TestCase):
>>          def setUp(self):
>>              self.L = lexer()
>>              self.P = parser(self.L)
>>
>>          def test_lexer(self):
>>              input = "specimen input"
>>
>>              # *****
>>              self.L.INJECT_INPUT_CHARACTERS(input)
>>              # *****
>>
>>              self.failUnlessEqual(some_result, self.L.nextToken())
>>
>>          def test_parser(self):
>>              input = "specimen input"
>>
>>              # *****
>>              self.L.INJECT_INPUT_CHARACTERS(input)
>>              # *****
>>
>>              self.failUnlessEqual(some_result, self.P.someRule()
>>
>> How do I achieve the INJECT_INPUT_CHARACTERS(input) part?
>>
>> L.
> 
> 
> 



More information about the antlr-interest mailing list