[antlr-interest] Generating Fake Lexical Tokens

[shyamgopale]

Hi,

  I am writing a parser for Python and due to
the nature of Python's way of specifying statement
blocks I have to generate fake tokens.
  In Python statements are grouped together by their
indentation level. So, to let the parser detect a
change in the grouping the lexer needs to generate
fake tokens for INDENT and DEDENT.
   Consider the Python program
   if test:
       print "something"
       do_something()
   # Outside if
   do_somethingmore()
Now for the above program - The lexer needs to generate
an INDENT token before the print to let the parser
know that the following statements are part of an
if block. And similarly it needs to generate a DEDENT
token after do_something() to indicate end of the if
block. 
  I have the logic to generate the INDENT and DEDENT
tokens but I have no idea how to make the lexer report
them before or after the real tokens. Can anyone help
me out with this. I am looking for a way to insert
additional tokens in the token stream.

Thanks,
Shyam

   



 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/