[antlr-interest] Lexer

[Jim Idle]

> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> bounces at antlr.org] On Behalf Of Steven Kibler
> Sent: Thursday, October 21, 2010 7:58 PM
> To: antlr-interest at antlr.org
> Subject: [antlr-interest] Lexer
> 
> Windows generation, C++, Microsoft compiler 2008.
> 
> 
> 
> 1.       Are there some configuration options for the lexer?  If so,
> where
> are they?

What kind of options are you looking for?

> 2.       How do I reduce the size of the tables to something less than
> 30,000  program lines?  Input is standard ASCII.

Usually this means that your lexer is trying to be very specific on the
characters it matches. It is generally best to be as vague as possible (but
no vaguer) on the patterns and counts. Then you apply a checking function
that can give a nice error about the problems with a token such as too many
characters or invalid characters. 

Also in cases like:

T: 'X'? 'X'; use

T: 'X' 'X'? ;


However, you will need to post your grammar if you want more specific help.

Also, what version of ANTLR are you using? Most of those lines are data
table static definitions by the way.

> 
> 3.       I have a runtime problem and can't determine if it is a parser
> issue or lexer issue.  How can I track the tokens returned from the
> lexer?

Look at the example projects in the downloads. Start with the Java parser
example. You will see that it calls LT(1) directly on the lexer. You can
also do that and call LT(n), get the token type and print the token text.

> 
> 4.       How do I insert my own lexer?


Look at the generated lexer, you will see that it creates a struct, installs
the pointers to its functions, but also creates standard library stuff. You
need to create the same thing, but provide your own implementations of
nextToken and the other library lexer stuff.

> 
> 5.       It appears that none of the functions in the generated lexer
> code,
> other than the initialization functions, are called.  Is the lexing
> done in the libraries?  How do I make a lexer that uses that code?

The functions are called from the match and nextToken methods. Trace the
code using the debugger.


Jim

> 
> 
> 
> Thanks
> 
> 
> 
> 
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
> email-address