[antlr-interest] Error recovery in lexer (~ unclosed string)

[Jim Idle]

T: '<|'  ' ( options {greedy=false;} : . )*
    ( '|>'
      | // error message
     )
;

Jim

On Jun 30, 2009, at 5:53 PM, Thibaut Colar <tcolar at colar.> wrote:

> Hello there.
>
> In my grammar file, i have a lexer rule to match DSL string like this:
> DSL        :'<|' ( options {greedy=false;} : . )* '|>' ;
>
> Now my Lexer is hooked up int the NetBeans IDE to parse(well lexer  
> only
> for now) files and that works well.
> The problem is that if you type an oening <| (unclosed at this time),
> the lexer errors out.
> What happens is that it goes all the way to the end of the file trying
> to find the closing |>, hits the EOF and throws an exception (null
> nextToken / unmatched tokens left).
>
> I'm not surprised it doesn't like this, but i want to add some error
> recovery so it deos not fail.
> I've tried many thing in last 24h but can't seem to find something  
> that
> works.
>
> Latest try is basically this (using states):
> Grammar:
> DSL        :'<|' {state=INCOMPLETE_DSL} ( options  
> {greedy=false;} : . )*
> '|>' {state=NORMAL};
>
> Lexer: ------------------------
>    public Token<FanTokenID> nextToken()
>    {
>        curToken = (CommonToken) lexer.nextToken();
>    if(curToken.getType()==-1) //prob. EOF
>    {
>        int state = lexer.getState();
>        switch (state)
>        {
>        case FanStates.INCOMPLETE_DSL:
>            curToken.setType(FanLexer.INCOMPLETE_DSL); // set as
> incomplete token type
>            break;
>        }
>        lexer.clearState();
>    }
>    }
> ------------------------
>
> However that still does not work quite right, it does replace the  
> token
> correctly, but i guess after that it does not continue at the right
> place (do i need a rewind() or consume() or something ?)
>
> Or maybe i'm doing this the wrong way - should i "emit" a fake closing
> token(|>) instead ?
>
> I'm sure this is a pretty common issue but i couldn't find any
> documented way to do this in antlr3 in the book or online.
>
> Any help would be greatly appreciated, Thanks.
>
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address