[antlr-interest] Customizing token separators without recompiling

[Dukie Banderjee]

Thanks, Steve, that looks very promising!

Rob

----------------------------------------
> Date: Mon, 8 Jun 2009 01:14:19 +0100
> Subject: Re: [antlr-interest] Customizing token separators without recompiling
> From: steve at stevecooper.org
> To: dukie_banderjee at hotmail.com
> CC: jsrs701 at yahoo.com; antlr-interest at antlr.org
>
> I don't know if this is any closer, but I had this idea.
>
> Your problem seems to be in getting a lexer which will give you the
> right stream of tokens, and not in writing the parser that feeds off
> them. You could write your own lexer to do the splitting of the
> strings, and use ANTLR to write the parser. ANTLR parsers don't feed
> directly off a string, but off an ITokenSource object;
>
> public interface ITokenSource
> {
> string SourceName { get; }
> IToken NextToken();
> }
>
> You could create your own token source which would do the separation
> by hand, and return a stream of tokens. Something like
>
> public class UnEdifactLexer: ITokenSource
> {
> // token types
> public const int EOF = -1;
> public const int ID = 0;
> public const int NUMBER = 1;
> public const int COLON = 2;
> ...
>
> // all the tokens in the input
> private Queue tokens;
>
> public UnEditfactLexer(string input, char userSeparator)
> {
> this.tokens = new Queue();
> foreach(var line in input.Split('\n'))
> {
> foreach(var piece in CustomSplit(userSeparator))
> {
> // custom code to convert a line
> // into a set of tokens
> tokens.Enqueue(new Token(...));
> }
> }
> }
>
> public IToken NextToken()
> {
> if (tokens.Count> 0)
> return tokens.Dequeue();
> else
> return new Token(EOF,...);
> }
> }
>
> Then you write a parser grammar in ANTLR which does the parsing and
> tree-building.
>
> Anyway, the benefit of this approach is that you have full power over
> splitting up the strings and converting them into tokens. After that,
> the parser takes up the strain.
>
> Steve

_________________________________________________________________
Windows Live helps you keep up with all your friends, in one place.
http://go.microsoft.com/?linkid=9660826