[antlr-interest] [C#] Lexer Not Throwing Errors

Justin Holewinski justin.holewinski at gmail.com
Thu Apr 21 09:50:51 PDT 2011


Just a quick follow-up:  an easy re-production is to use the Antlr-provided
examples-v3 archive.  In the csharp/LL-star directory, change the SimpleC.g
file to use the CSharp3 target instead of CSharp2, add "System." to the
WriteLine methods, and use either the Antlr jar archive or the C# tool to
generate the grammar files.  Upon testing, the provided "input" file works
as expected, but again, any other file is also parsed *without errors*.

Test run:

Good:
jholewinski at rapture [LL-star]$ mono Test.exe input
Processing file: /Volumes/Data/personal/examples-v3/csharp/LL-star/input
bar is a declaration
foo is a definition
Finished processing file:
/Volumes/Data/personal/examples-v3/csharp/LL-star/input

Bad:
jholewinski at rapture [LL-star]$ mono Test.exe output
Processing file: /Volumes/Data/personal/examples-v3/csharp/LL-star/output
Finished processing file:
/Volumes/Data/personal/examples-v3/csharp/LL-star/output
jholewinski at rapture [LL-star]$ mono Test.exe files
Processing file: /Volumes/Data/personal/examples-v3/csharp/LL-star/files
Finished processing file:
/Volumes/Data/personal/examples-v3/csharp/LL-star/files
jholewinski at rapture [LL-star]$ mono Test.exe Main.cs
Processing file: /Volumes/Data/personal/examples-v3/csharp/LL-star/Main.cs
Main is a definition
Finished processing file:
/Volumes/Data/personal/examples-v3/csharp/LL-star/Main.cs
jholewinski at rapture [LL-star]$

On Tue, Apr 19, 2011 at 12:49 PM, Justin Holewinski <
justin.holewinski at gmail.com> wrote:

> I forgot to add that if I instead use the Java target, I get lexer
> exceptions thrown on invalid input.
>
>
> On Tue, Apr 19, 2011 at 12:48 PM, Justin Holewinski <
> justin.holewinski at gmail.com> wrote:
>
>> I'm using the C# 3 port of Antlr (target and code generator) and I'm
>> experiencing a lack of lexer errors being generated.  As a concrete example,
>> consider the following grammar:
>>
>> grammar CSTest;
>>
>> public
>> compilation_unit
>>   : IDENTIFIER* EOF
>>   ;
>>
>> IDENTIFIER
>>   : (ID_PART) (ID_PART_OR_NUMBER)*
>>   ;
>>
>> fragment
>> ID_PART
>>   : ('a'..'z' | 'A'..'Z' | '_')
>>   ;
>>
>> fragment
>> ID_PART_OR_NUMBER
>>   : ID_PART
>>   | '0'..'9'
>>   ;
>>
>> WS
>>   :  (' '|'\r'|'\t'|'\n') { /*Skip();*/ $channel=Hidden; }
>>   ;
>>
>> The only production in this grammar should allow for an arbitrary sequence
>> of identifiers, and it does.  However, it also accepts ALL input, including
>> binary files and anything else I pipe into it.  Is the default in the C#
>> target to silently ignore any lexing errors?  In my driver program, I am not
>> seeing any throw exceptions when I call the compilation_unit rule.
>>
>> using System;
>> using System.IO;
>> using Antlr.Runtime;
>>
>>
>> namespace CSTest {
>>   public class Program {
>>     public static void Main(string[] args) {
>>       string filename;
>>       filename = args[0];
>>       if(!Path.IsPathRooted(filename)) {
>>         filename = Path.Combine(Environment.CurrentDirectory, filename);
>>       }
>>       ICharStream input = new ANTLRFileStream(filename);
>>       CSTestLexer lexer = new CSTestLexer(input);
>>       CommonTokenStream tokens = new CommonTokenStream(lexer);
>>       CSTestParser parser = new CSTestParser(tokens);
>> try {
>> parser.compilation_unit();
>> }
>> catch(RecognitionException re) {
>>   Console.Out.WriteLine("EXCEPTION");
>> Console.Out.WriteLine(re.StackTrace);
>>  }
>>
>>     }
>>
>>   }
>>
>> }
>>
>> --
>>
>> Thanks,
>>
>> Justin Holewinski
>>
>>
>
>
> --
>
> Thanks,
>
> Justin Holewinski
>
>


-- 

Thanks,

Justin Holewinski


More information about the antlr-interest mailing list