[antlr-interest] Re: Checking for expression end in Javascript parser

[Tech]

Hi Xue,

I had been thinking along similar lines, but was going to use the 
TokenStreamHiddenTokenFilter (see Preserving Whitespace During 
Translation <http://antlr.org/article/whitespace/index.html>) to store 
the newline characters. However, this would still need some kind of 
semantic predicate to access the values as part of a parser check (I 
think, I've not tried using semantic predicates yet). So both options 
are quite similar. Your approach means you can use newline tokens in the 
recognition rule, which might be simpler. Looking forward to hearing 
what you think.

Mark

Xue Yong Zhi wrote:

>I have the same problem since I am working on a ruby parser. Checking line 
>number is an interesting idea, I will think it a little more tonight.
>
>I am using another approach right now:
>1. Do not filter line break in the lexer.
>2. Subclass antlr generated lexer, overide nextToken(), add a flag to 
>control the visibility of the linebreak.
>For example:
>Token nextToken()
>{
>   if (!filter_line_break)
>  {
>    //act like a normal lexer
>    return super.nextToken();
>  }
>  else
> {
>    //filter line break
>    while (true)
>    {
>       Token t = super.nextToken();
>       if (t.Type() != Token.LINE_BREAK)
>      {
>          return t;
>      }
>    }
> }
>}
>
>Most of the time linebreaks is ignored in the parser (by setting 
>lexer.filter_line_break = true), but you can "see" them when you need.
>I have not finished my parser yet so I am not sure if it works in all cases, 
>but so far so good.
>
>  
>


-- 
No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.1.361 / Virus Database: 267.12.5/148 - Release Date: 25/10/2005