[antlr-interest] Generating Fake Lexical Tokens

shyamgopale shyam at persistent.co.in
Wed Sep 18 08:14:12 PDT 2002


  I am writing a parser for Python and due to
the nature of Python's way of specifying statement
blocks I have to generate fake tokens.
  In Python statements are grouped together by their
indentation level. So, to let the parser detect a
change in the grouping the lexer needs to generate
fake tokens for INDENT and DEDENT.
   Consider the Python program
   if test:
       print "something"
   # Outside if
Now for the above program - The lexer needs to generate
an INDENT token before the print to let the parser
know that the following statements are part of an
if block. And similarly it needs to generate a DEDENT
token after do_something() to indicate end of the if
  I have the logic to generate the INDENT and DEDENT
tokens but I have no idea how to make the lexer report
them before or after the real tokens. Can anyone help
me out with this. I am looking for a way to insert
additional tokens in the token stream.




Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 

More information about the antlr-interest mailing list