[antlr-interest] [C#] Lexer Not Throwing Errors

[Justin Holewinski]

On Thu, Apr 21, 2011 at 1:06 PM, Sam Harwell <sharwell at pixelminegames.com>wrote:

> Hi Justin,
>
> The exception does get thrown, but is handled in Lexer.NextToken() by
> tossing out a character and continuing. There are many ways to deal with
> this situation ranging from overriding NextToken in your lexer to creating
> an ANYCHAR rule at the very end that allows the incorrect input characters
> to pass on to the parser for handling there.
>
> ANYCHAR : . ;
>

I see.  Out of curiosity, any reason why this is different from the bahavior
of the Java target?


>
> Sam
>
> -----Original Message-----
> From: antlr-interest-bounces at antlr.org
> [mailto:antlr-interest-bounces at antlr.org] On Behalf Of Justin Holewinski
> Sent: Tuesday, April 19, 2011 11:49 AM
> To: antlr-interest at antlr.org
> Subject: [antlr-interest] [C#] Lexer Not Throwing Errors
>
> I'm using the C# 3 port of Antlr (target and code generator) and I'm
> experiencing a lack of lexer errors being generated.  As a concrete
> example,
> consider the following grammar:
>
> grammar CSTest;
>
> public
> compilation_unit
>  : IDENTIFIER* EOF
>  ;
>
> IDENTIFIER
>  : (ID_PART) (ID_PART_OR_NUMBER)*
>  ;
>
> fragment
> ID_PART
>  : ('a'..'z' | 'A'..'Z' | '_')
>  ;
>
> fragment
> ID_PART_OR_NUMBER
>  : ID_PART
>  | '0'..'9'
>  ;
>
> WS
>  :  (' '|'\r'|'\t'|'\n') { /*Skip();*/ $channel=Hidden; }
>  ;
>
> The only production in this grammar should allow for an arbitrary sequence
> of identifiers, and it does.  However, it also accepts ALL input, including
> binary files and anything else I pipe into it.  Is the default in the C#
> target to silently ignore any lexing errors?  In my driver program, I am
> not
> seeing any throw exceptions when I call the compilation_unit rule.
>
> using System;
> using System.IO;
> using Antlr.Runtime;
>
>
> namespace CSTest {
>  public class Program {
>    public static void Main(string[] args) {
>      string filename;
>      filename = args[0];
>      if(!Path.IsPathRooted(filename)) {
>        filename = Path.Combine(Environment.CurrentDirectory, filename);
>      }
>      ICharStream input = new ANTLRFileStream(filename);
>      CSTestLexer lexer = new CSTestLexer(input);
>      CommonTokenStream tokens = new CommonTokenStream(lexer);
>      CSTestParser parser = new CSTestParser(tokens); try {
> parser.compilation_unit(); } catch(RecognitionException re) {
>  Console.Out.WriteLine("EXCEPTION");
> Console.Out.WriteLine(re.StackTrace);
> }
>
>    }
>
>  }
>
> }
>
> --
>
> Thanks,
>
> Justin Holewinski
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe:
> http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>
>


-- 

Thanks,

Justin Holewinski