[antlr-interest] Understanding priorities in lexing (newbie)

[Wincent Colaiuta]

El 13/7/2007, a las 2:49, Daniel Brosseau escribió:

> Where I have been coming from is that one grammar file (what  
> follows 'grammar' in the file) would define one language grammar  
> (not two). How the work gets divided up between lexer and parser is  
> a matter of convenience and efficiency but should not change the  
> meaning of the overall grammar and the character streams that gets  
> accepted or rejected by the resulting overall state machine. It  
> seems that in ANTLR's case, the one grammar file defines two  
> grammars (one for the lexer and one for the parser) and how the  
> work gets divided up between the two can have a considerable impact  
> on the character streams accepted or rejected. I get further  
> tripped up because the rules of these two grammars can be  
> interspersed in the file and look as if they mesh seamlessly. Ooof!

I think it's just something you have to accept: you must think of  
your lexer as a completely independent unit from the parser, despite  
the fact a single grammar file can define both a lexer and parser  
(this ability for ANTLR to extract lexer and parser rules from a  
single file is just a convenience provided to you). You could run the  
lexer on on a machine in another room, copy the list of resulting  
tokens using pencil and paper, walk back into the first room and  
manually feed the tokens into the parser and things would work  
exactly as they do now. In other words, the separation of lexer and  
parser is just something that you have to accept and is unlikely to  
ever change.

Cheers,
Wincent