[antlr-interest] Switching lexer rules on/off from parser

[Bryan Ewbank]

Yes, but be careful.  Any communications upstream from the parser to
the lexer have a funny smell to them.  If you can change your design
(or is this a requirement?), it's probably better to do that.

...

Since you mark a token as being skipped by setting it's type, you can
do this with some nifty interface - or a global variable :-)

The "be careful" has to do with lookahead.  If the lexer gets a few
tokens ahead of the parser, it's possible to "see" a newline in
declaration mode that should, in fact, terminate a statement.

I'm making this up, as I don't know your grammar, but consider:

   decl ....
   a=b

If you scanner reads the newline after "b" before the parser knows to
switch from decl to statement mode, it will ignore that newline.

If you allow declarations to follow statements, you see the same
problem in reverse:
   // I'm assuming "i: integer" is a declaration here
   a = b
   i
   : integer
Now, the newline after "i" might cause problems because it's not clear
if this is a declaration or a statement.

As a general rule, be aware of k in LL(k), and also - where I've been
burned - syntactic predicates which require more information that is
available

    ( decl ) => decl
    | statement

It's likely that (decl) will not match because the newlines are not
being correctly processed - and therefore your grammar will attempt to
process something that looks like a declaration as a statement.

Hope this helps,
B-)

On Mon, 21 Mar 2005 10:36:00 +0100, Marcus Tangermann
<marcus.tangermann at web.de> wrote:
> I have a grammar with a declaration part that can ignore newline
> and a part with statements. A statement in this part is terminated
> by a newline.