[antlr-interest] how to let parser control lexer state.

Jim Idle jimi at temporal-wave.com
Sat Apr 28 14:54:51 PDT 2007


Take a look at the island grammar example if you want the lexer to jump
out and call a different parse stream until some termination point is
reached, then jump back in to the main stream. Otherwise you can just
hide the text in the lexer by setting a token { $type = HIDDEN }

Jim

-----Original Message-----
From: antlr-interest-bounces at antlr.org
[mailto:antlr-interest-bounces at antlr.org] On Behalf Of femto gary
Sent: Friday, April 27, 2007 8:47 PM
To: antlr-interest at antlr.org
Subject: Re: [antlr-interest] how to let parser control lexer state.

yes, so that's the problem.
syntax predicate or semantic predicate all happpens at parser level
and can't be used. because lex tokenStream are already constanted..

On 4/28/07, David Holroyd <dave at badgers-in-foil.co.uk> wrote:
> On Fri, Apr 27, 2007 at 10:42:24AM -0700, Loring Craymer wrote:
> > That's what semantic predicates (in the parser) are
> > for--there are no lexical issues involved in this
> > case.  You want to do the symbol table lookup and test
> > in a predicate.
>
> Here-docs are a lexing problem, I think -- content on subsequent lines
> must not be interpreted with normal lexer rules until the document
> terminator is seen..?
>
> > --- femtowin1 <femtowin1 at gmail.com> wrote:
> >
> > > Hi all, in antlr3, can parser control lexer state
> > > and decide how lexer lexing? some grammar has
> > > ambiguity
> > > decided upon by parser knowledge.
> > >   for ruby grammar <<
> > > x << 1
> > > test
> > > 1
> > > if x is a variable, then << is shift operator,
> > > otherwise it is a heredoc. so lexing must know
> > > from the symbol table whether x has been defined
> > > beforehand. But current antlrv3 implementation,
> > > lexer lexing to a constant token stream, and feed
> > > it into parser, so can't achieve this effect.
>
> --
> http://david.holroyd.me.uk/
>


-- 
Best Regards
XRuby http://xruby.com
femto http://hi.baidu.com/femto


More information about the antlr-interest mailing list