[antlr-interest] [C#] Lexer Not Throwing Errors

[Sam Harwell]

I'm not sure why it differs from the Java target. The nextToken()
implementation in the Java target appears to use the same exception handling
that the CSharp3 target uses.

 

Sam

 

From: Justin Holewinski [mailto:justin.holewinski at gmail.com] 
Sent: Thursday, April 21, 2011 12:49 PM
To: Sam Harwell
Cc: antlr-interest at antlr.org
Subject: Re: [antlr-interest] [C#] Lexer Not Throwing Errors

 

On Thu, Apr 21, 2011 at 1:06 PM, Sam Harwell <sharwell at pixelminegames.com>
wrote:

Hi Justin,

The exception does get thrown, but is handled in Lexer.NextToken() by
tossing out a character and continuing. There are many ways to deal with
this situation ranging from overriding NextToken in your lexer to creating
an ANYCHAR rule at the very end that allows the incorrect input characters
to pass on to the parser for handling there.

ANYCHAR : . ;

 

I see.  Out of curiosity, any reason why this is different from the bahavior
of the Java target?

 


Sam

-----Original Message-----
From: antlr-interest-bounces at antlr.org
[mailto:antlr-interest-bounces at antlr.org] On Behalf Of Justin Holewinski
Sent: Tuesday, April 19, 2011 11:49 AM
To: antlr-interest at antlr.org
Subject: [antlr-interest] [C#] Lexer Not Throwing Errors


I'm using the C# 3 port of Antlr (target and code generator) and I'm
experiencing a lack of lexer errors being generated.  As a concrete example,
consider the following grammar:

grammar CSTest;

public
compilation_unit
 : IDENTIFIER* EOF
 ;

IDENTIFIER
 : (ID_PART) (ID_PART_OR_NUMBER)*
 ;

fragment
ID_PART
 : ('a'..'z' | 'A'..'Z' | '_')
 ;

fragment
ID_PART_OR_NUMBER
 : ID_PART
 | '0'..'9'
 ;

WS
 :  (' '|'\r'|'\t'|'\n') { /*Skip();*/ $channel=Hidden; }
 ;

The only production in this grammar should allow for an arbitrary sequence
of identifiers, and it does.  However, it also accepts ALL input, including
binary files and anything else I pipe into it.  Is the default in the C#
target to silently ignore any lexing errors?  In my driver program, I am not
seeing any throw exceptions when I call the compilation_unit rule.

using System;
using System.IO;
using Antlr.Runtime;


namespace CSTest {
 public class Program {
   public static void Main(string[] args) {
     string filename;
     filename = args[0];
     if(!Path.IsPathRooted(filename)) {
       filename = Path.Combine(Environment.CurrentDirectory, filename);
     }
     ICharStream input = new ANTLRFileStream(filename);
     CSTestLexer lexer = new CSTestLexer(input);
     CommonTokenStream tokens = new CommonTokenStream(lexer);
     CSTestParser parser = new CSTestParser(tokens); try {
parser.compilation_unit(); } catch(RecognitionException re) {
 Console.Out.WriteLine("EXCEPTION");
Console.Out.WriteLine(re.StackTrace);
}

   }

 }

}

--

Thanks,

Justin Holewinski

List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address




-- 

Thanks,

 

Justin Holewinski