[antlr-interest] [C#] Lexer Not Throwing Errors

[Sam Harwell]

Hi Justin,

The exception does get thrown, but is handled in Lexer.NextToken() by
tossing out a character and continuing. There are many ways to deal with
this situation ranging from overriding NextToken in your lexer to creating
an ANYCHAR rule at the very end that allows the incorrect input characters
to pass on to the parser for handling there.

ANYCHAR : . ;

Sam

-----Original Message-----
From: antlr-interest-bounces at antlr.org
[mailto:antlr-interest-bounces at antlr.org] On Behalf Of Justin Holewinski
Sent: Tuesday, April 19, 2011 11:49 AM
To: antlr-interest at antlr.org
Subject: [antlr-interest] [C#] Lexer Not Throwing Errors

I'm using the C# 3 port of Antlr (target and code generator) and I'm
experiencing a lack of lexer errors being generated.  As a concrete example,
consider the following grammar:

grammar CSTest;

public
compilation_unit
  : IDENTIFIER* EOF
  ;

IDENTIFIER
  : (ID_PART) (ID_PART_OR_NUMBER)*
  ;

fragment
ID_PART
  : ('a'..'z' | 'A'..'Z' | '_')
  ;

fragment
ID_PART_OR_NUMBER
  : ID_PART
  | '0'..'9'
  ;

WS
  :  (' '|'\r'|'\t'|'\n') { /*Skip();*/ $channel=Hidden; }
  ;

The only production in this grammar should allow for an arbitrary sequence
of identifiers, and it does.  However, it also accepts ALL input, including
binary files and anything else I pipe into it.  Is the default in the C#
target to silently ignore any lexing errors?  In my driver program, I am not
seeing any throw exceptions when I call the compilation_unit rule.

using System;
using System.IO;
using Antlr.Runtime;


namespace CSTest {
  public class Program {
    public static void Main(string[] args) {
      string filename;
      filename = args[0];
      if(!Path.IsPathRooted(filename)) {
        filename = Path.Combine(Environment.CurrentDirectory, filename);
      }
      ICharStream input = new ANTLRFileStream(filename);
      CSTestLexer lexer = new CSTestLexer(input);
      CommonTokenStream tokens = new CommonTokenStream(lexer);
      CSTestParser parser = new CSTestParser(tokens); try {
parser.compilation_unit(); } catch(RecognitionException re) {
  Console.Out.WriteLine("EXCEPTION");
Console.Out.WriteLine(re.StackTrace);
}

    }

  }

}

-- 

Thanks,

Justin Holewinski

List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address