[antlr-interest] ANTLR 2.7.7 - prepend to InputBuffer

[Jim Idle]

The better solution is to create a separate pre-processor stage. You run the
pre-processor and it feed its output in to the 'real' lexer. This is a lot
easier to maintain and by the time you had messed around trying to create
secondary lexers, include stacks and so on for other solutions, you end up
with this being faster. Also, the ability to have a much simpler lexer that
is just handling #if, #define and macro expansion makes things much easier
(other than macros in macros).

Now, if your environment is known,then you could just implement this in m4,
which is available just about everywhere.

http://en.wikipedia.org/wiki/Preprocessor
http://en.wikipedia.org/wiki/M4_(computer_language) 
http://www.gnu.org/software/m4/ 

You could also just run the C pre-processor and define your language as
using the C macro engine.

And further, if you can do without macros altogether, you can stop the
language users creating unreadable, undebuggable code because they thought
macros were cool and clever and used them nested to 23 levels ;-)

Jim



> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> bounces at antlr.org] On Behalf Of Alex Marin
> Sent: Wednesday, August 18, 2010 8:00 AM
> To: antlr-interest at antlr.org
> Cc: etools at amiq.ro
> Subject: [antlr-interest] ANTLR 2.7.7 - prepend to InputBuffer
> 
> Hello everyone,
> 
> Is it possible in ANTLR 2.7.7 to add characters on the fly to a lexer's
> InputBuffer? Do you have any suggestion/guideline/experience on how to
> do this? (without re-implementing the InputBuffer/CharQueue classes)
> 
> I will explain the reason behind the need to do this. The language we are
> lexing allows pre-processing, by using C-like defines (or macros).
> 
> // Define foo macro
> `define foo this is the replacement of foo
> 
> // Use foo macro
> `foo // this expands to 'this is the replacement of foo'
> 
> // Define moo macro with a parameter
> `define moo(x) x+2
> 
> // Use moo macro
> `moo(1) // this expands to '1+2'
> 
> At the moment, when our lexer encounters a macro, it starts another lexer
> which lexes the replacement. E.g:
> 
> ...
> `define foo this is the replacement of foo ...
> `foo // Here another lexer is started and provided the input stream 'this
is
> the replacement of foo'
> ...
> 
> This approach works very well at the time being, except for the situations
> when the replacement text is not a fully "lexable" piece of code, like in
the
> example below:
> 
> ...
> `define a(x,y) x+y // Define macro a with parameters x and y `define b
`a(1  //
> Define macro b without parameters ...
> `b,2) // Use macro b; this should expand to `a(1,2) and therefore to 1+2
...
> 
> Now, the problem is that the lexer we start for `b will lex '`a(`' which
is not
> lexically correct and will fail with an error.
> 
> The best solution would be to be able to insert in the main lexer's buffer
the
> replacement of `b and continue lexing normally. Of course, no new lexer
> would be needed then. So, after matching `b, the buffer would look like
> `a(1,2)
> 
> Thanks in advance,
> Alex
> 
> 
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
> email-address