[antlr-interest] lexer versus parser

[Silvain Piree]

Hi,

the nice thing about ANTLR is that you can build parsers
using the lexer. I've recently built some preprocessors using
ANTLR lexers, and I really liked the result.

Using a lexer-based parser is a lot more powerful than using
a separate lexer which passes tokens to a parser. Many issues
that are difficult to solve in lexer+parser solutions are easily 
solved using lexer-based parsers.

For example:
- keywords as identifier is no problem
- inline data (supported by certain job control languages) can
   be easily skipped (or collected)
- different grammar rules are easy to support (e.g. embedded 
   SQL that may have different rules for matching identifiers)
- it's easy to build partial-parsers, i.e. parsers that skip parts 
   that are not of interest (or not yet supported). 
- etc. etc. etc.

However, despite these many advantages everybody is
building parsers with separate lexers (like I've done myself). 
There must be a very good reason for that! What are the
problems one would encounter when building lexer-based
parsers? Is this concept the same as "scanner-less parsers"?

Silvain




 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/