[antlr-interest] Emit two Tokens in one Lexer Rule

Alexandre Porcelli porcelli at uol.com.br
Tue Feb 12 05:26:44 PST 2008


Alexander,

 You should override the emit and nextToken methods to do it.
 A sample here:

    // buffer (queue) to hold the emit()'d tokens
    private LinkedList<Token> myToken = new LinkedList<Token>();

    public void emit(Token t) {
        token = t;
        myToken.add(t);
    }

	private void advanceInput(){
		tokenStartCharIndex = input.index();
		tokenStartCharPositionInLine = input.getCharPositionInLine();
		tokenStartLine = input.getLine();
	}

	public Token nextToken() {
		while (true) {
			if (myToken.size() == 0) {
				token = null;
				channel = Token.DEFAULT_CHANNEL;
				advanceInput();
				text = null;
                                if (input.LA(1) == CharStream.EOF) {
					return Token.EOF_TOKEN;
				}
				try {
					mTokens();
					if (token == null) {
						emit();
					} else if (token == Token.SKIP_TOKEN) {
						continue;
					}
				} catch (RecognitionException re) {
					reportError(re);
					recover(re);
				}
			} else {
				Token result = myToken.poll();
				if (result != Token.SKIP_TOKEN || result != null) { // discard
																	// SKIP
																	// tokens
					return result;
				}
			}
		}
	}

Regards,
Alexandre Porcelli

On Feb 12, 2008 11:16 AM, Alexander Gängel <alexander at gaengel.de> wrote:
> All I know is that I can't emit 2 Tokens in one Lexer Rule.
>
> My problem it that I have to decide if I have an Float like  1.1 or an
> Int DOT Identifiert lik 5.max
> My workaround would be to emit just the first an restart lexing from the
> last correct charterer. So I would Just emit 5 as Int and reparse from
> the Dot.
>
> Thanks for your Help
> Alexander
>
>
>


More information about the antlr-interest mailing list