[antlr-interest] Handling explicit continuation characters

[Gavin Lambert]

At 06:53 13/01/2009, Brisard, Fred D wrote:
>I am parsing a grammar that uses minus or plus at the end of line 
>to indicate a continuation.
[...]
>I would like to just absorb them into the hidden stream so that 
>the input appears to be on a single line.
>
>If I wanted to make a first pass on the input and just absorb 
>'-\n' and '+\n' then all would be well.  That seems pretty 
>wasteful to make a pass to just do that.  It seems like I should 
>be able to do it during the lexical pass.

You can; you just need to give the lexer enough hinting so that it 
can cope with both cases properly:

MINUS: '-' ('\r'? '\n' { skip(); })?;
PLUS: '+' ('\r'? '\n' { skip(); })?;

(Or instead of skipping, you could change the $type to 
CONTINUATION and the $channel to HIDDEN.)

You will also need to ensure that '-' and '+' won't be consumed by 
other lexer rules (or at least that if they are, it's in contexts 
where a continuation isn't permitted).  And your parser will have 
to be able to cope with getting MINUS and PLUS tokens between 
other tokens.