[antlr-interest] Re: Checking for expression end in Javascript parser

Xue Yong Zhi zhixueyong at hotmail.com
Tue Oct 25 11:49:09 PDT 2005


I have the same problem since I am working on a ruby parser. Checking line 
number is an interesting idea, I will think it a little more tonight.

I am using another approach right now:
1. Do not filter line break in the lexer.
2. Subclass antlr generated lexer, overide nextToken(), add a flag to 
control the visibility of the linebreak.
For example:
Token nextToken()
{
   if (!filter_line_break)
  {
    //act like a normal lexer
    return super.nextToken();
  }
  else
 {
    //filter line break
    while (true)
    {
       Token t = super.nextToken();
       if (t.Type() != Token.LINE_BREAK)
      {
          return t;
      }
    }
 }
}

Most of the time linebreaks is ignored in the parser (by setting 
lexer.filter_line_break = true), but you can "see" them when you need.
I have not finished my parser yet so I am not sure if it works in all cases, 
but so far so good.

-- 
Xue Yong Zhi
http://seclib.blogspot.com

> expression. However, I don't really want to include newline characters in 
> the parser, as you'd have to add them in everywhere. I was thinking of 
> trying to use a semantic predicate that checked the line number of the 
> current token against the line number of the next token just for 
> expression statements. How do you programatically reference the current 
> token being checked? Is it always effectively LT(1)?
> If there is a better approach, I would be interested to know it. This is 
> only really out of academic interest, currently my solution is to assume 
> the input file is well written javascript ;-)





More information about the antlr-interest mailing list