[antlr-interest] Emit two Tokens in one Lexer Rule
Alexandre Porcelli
porcelli at uol.com.br
Tue Feb 12 05:26:44 PST 2008
Alexander,
You should override the emit and nextToken methods to do it.
A sample here:
// buffer (queue) to hold the emit()'d tokens
private LinkedList<Token> myToken = new LinkedList<Token>();
public void emit(Token t) {
token = t;
myToken.add(t);
}
private void advanceInput(){
tokenStartCharIndex = input.index();
tokenStartCharPositionInLine = input.getCharPositionInLine();
tokenStartLine = input.getLine();
}
public Token nextToken() {
while (true) {
if (myToken.size() == 0) {
token = null;
channel = Token.DEFAULT_CHANNEL;
advanceInput();
text = null;
if (input.LA(1) == CharStream.EOF) {
return Token.EOF_TOKEN;
}
try {
mTokens();
if (token == null) {
emit();
} else if (token == Token.SKIP_TOKEN) {
continue;
}
} catch (RecognitionException re) {
reportError(re);
recover(re);
}
} else {
Token result = myToken.poll();
if (result != Token.SKIP_TOKEN || result != null) { // discard
// SKIP
// tokens
return result;
}
}
}
}
Regards,
Alexandre Porcelli
On Feb 12, 2008 11:16 AM, Alexander Gängel <alexander at gaengel.de> wrote:
> All I know is that I can't emit 2 Tokens in one Lexer Rule.
>
> My problem it that I have to decide if I have an Float like 1.1 or an
> Int DOT Identifiert lik 5.max
> My workaround would be to emit just the first an restart lexing from the
> last correct charterer. So I would Just emit 5 as Int and reparse from
> the Dot.
>
> Thanks for your Help
> Alexander
>
>
>
More information about the antlr-interest
mailing list