[antlr-interest] ANTLR 2.7.7 - prepend to InputBuffer

Alex Marin alex.marin at amiq.ro
Wed Aug 18 07:59:59 PDT 2010


Hello everyone,

Is it possible in ANTLR 2.7.7 to add characters on the fly to a lexer's 
InputBuffer? Do you have any suggestion/guideline/experience on how to 
do this? (without re-implementing the InputBuffer/CharQueue classes)

I will explain the reason behind the need to do this. The language we 
are lexing allows pre-processing, by using C-like defines (or macros).

// Define foo macro
`define foo this is the replacement of foo

// Use foo macro
`foo // this expands to 'this is the replacement of foo'

// Define moo macro with a parameter
`define moo(x) x+2

// Use moo macro
`moo(1) // this expands to '1+2'

At the moment, when our lexer encounters a macro, it starts another 
lexer which lexes the replacement. E.g:

...
`define foo this is the replacement of foo
...
`foo // Here another lexer is started and provided the input stream 
'this is the replacement of foo'
...

This approach works very well at the time being, except for the 
situations when the replacement text is not a fully "lexable" piece of 
code, like in the example below:

...
`define a(x,y) x+y // Define macro a with parameters x and y
`define b `a(1  // Define macro b without parameters
...
`b,2) // Use macro b; this should expand to `a(1,2) and therefore to 1+2
...

Now, the problem is that the lexer we start for `b will lex '`a(`' which 
is not lexically correct and will fail with an error.

The best solution would be to be able to insert in the main lexer's 
buffer the replacement of `b and continue lexing normally. Of course, no 
new lexer would be needed then. So, after matching `b, the buffer would 
look like `a(1,2)

Thanks in advance,
Alex



More information about the antlr-interest mailing list