[antlr-interest] Re: antlr-interest@antlr.org

[Joseph Goettgens]

Karl,

> succeded with the grammar before, ...

> Basically I'm trying to use Antlr as a natrual language parser.
you seem to put a lot of (read: too much) stuff  into the lexer. The grammar 
is probaly the center for organizing things since it is more  "capable" than 
a lexer, though antlr blurs this difference to some degree. The grammar 
eventually will tell you which token types you need to 
distinguish---shouldn't be too many as natural languages don't have things 
like reserved words or other constructs derived from regular expressions. 
Have a look at http://en.wikipedia.org/wiki/Formal_language . I guess you 
are still dealing with a formal languange and not a natural one (e.g. see 
http://en.wikipedia.org/wiki/Noam_Chomsky). If you were really dealing with 
a natural language you'd be dealing with ungrammatical things sooner or 
later. Kalle, für den Fall, dass Du zuviel Bier trinkts, the lexer will 
become a real challenge, Prost and indeed.

Even if you were able to specify a description of your language with a 
single meta-language you are giving up one level of abstraction, since 
grammars give rules for types of words and lexers give rules on how to form 
words from elements of a vocabulary. In algebraic terms, you would avoid 
factoring out common terms and are punished by dealing with unnecessarily 
complex expression, subsequently.

jg