[antlr-interest] EOF in Lexer- how to?

Tomasz Jastrzebski tdjastrzebski at yahoo.com
Mon Jan 16 01:03:26 PST 2006


Hi Everybody,
   
  Is it possible to recognize EOF in the lexer?
   
  Ok, why would someone wanted to do it in the first place?
  Lets suppose I want my lexer to recognize a SingleLineComment, let's say Java "// comment" style. My lexer rules should look more or less like this:
  NewLine :(options{greedy=true;}:"\r\n" | '\r' | '\n' ) ;
  SingleLineComment :"//" ( ~('\r' | '\n') )* NewLine ;
   
  But there is a problem here. What if my input stream consists of only single comment and no NewLine? E.g.
  // comment text <EOF>
  This lexer will not recognize such an input correctly.
  That is why I want my lexer to be able to treat EOF as NewLine.
   
  However it seems like I can not use or define EOF token within Lexer. An attempt to use '\uFFFF' within the NewLine rule seems to block the lexer and lead to unpredictable results.
   
  I would appreciate any help.

			
---------------------------------
Yahoo! Photos
 Got holiday prints? See all the ways to get quality prints in your hands ASAP.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20060116/0fb26e84/attachment.html


More information about the antlr-interest mailing list