[antlr-interest] Re: 2 flex lexers 2 antlr parsers -> 1 simple solution?

[elias_biris]

--- In antlr-interest at y..., "elias_biris" <user_sofia at b...> wrote:
> 
> Flex reads the entire input file, storing it in its internal buffers,
> in one go at the beginning of scanning. So, even if the 2 

--
I should have said here that, 'Flex reads as much of the input file as
its internal buffers permit ...' 
--

Apparently there are no easy ways to keep track of the input file
position in flex (apart from tracking the line number)... Things can
get quite hairy when there is preprocessing of the input files.

So, I think that there are 2 alternatives:

1- create a single lexer for the 2/or more languages and separate the
patterns using lexer states ---> Error prone and hard to maintain if
the languages are large/complicated. Also there is no easy way to
subclass a lexer specification in flex and override pattern
definitions (afaik)

2- use the antlr feature for lexer generation. This would provide a
nice clean separation and flexibility for future
extension/maintainance. However, I am working for a lexer that should
efficiently handle EXTREMELY LARGE input files (up to GBs in size), so
I am naturally concerned about the efficiency of the implementation of
the antlr structures/algorithms for scanning such big files. Another
issue is that the languages can have quite intricate patterns (to give
you a clue I am working on SPICE-dialects). I think a pilot prototype
is in order here .... if someone has done work on SPICE definitions
before please let me know.

Many thanks to all that answered to my query, and put up with my
ignorance ;-)

Elias



 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/