[antlr-interest] Token position 0, -1
Ale Strooisma
alestrooisma at gmx.net
Fri Jul 6 04:39:05 PDT 2012
I was trying to find out how to fix my problem, but is seems that your
solution works perfectly, thanks!
On 07/06/2012 01:23 PM, Stefan Mätje (d) wrote:
> Am 06.07.2012 11:47:27 schrieb(en) Ale Strooisma:
>
>> Hello,
>>
>> if I try to get the position of a token in the file with getLine and
>> getCharPositionInLine, many of my tokens give the coördinates 0, -1,
>> which obviously is wrong. Why is this the case, and how can I fix it?
>>
> Hi,
>
> please be aware of the fact that imaginary tokens that you create by using
> rewrite rules don't have any location information by default (and no text
> either).
>
> So I think you should first look if all the tokens generated by the lexer
> contains the expected location information. Also you then can see which tokens
> were generated by the lexer and which were generated by a rewrite rule of the
> parser.
>
> In rewrite rules you can attach the location information of a real lexer token
> to a generated imaginary token by a syntax like that (see ANTLR book paragraph
> "Deriving Imaginary Nodes from Real Tokens"):
>
> compoundStatement
> : lc='{' statement* '}'
> -> ^(SLIST[$lc] statement*)
> ;
>
> All over all my answer is a little bit guesswork because your question is a
> little bit unspecific (lexer / parser or combined grammar ...? What are you
> doing exactly?).
>
> Hope that helps,
> Stefan
>
> Some example code to dump the lexer tokens:
>
>
> /** Dump all tokens the<lexer> provides in a readable form to the
> location where<outFileName> points to.
> */
> private static void generateTokenOutput(Lexer lexer, String outFileName) {
> PrintWriter tknOut;
> boolean useOutFile;
> Token token;
>
> System.err.print("==== TKN to ");
> if (null == outFileName) {
> useOutFile = false;
> System.err.println("stdout");
> tknOut = new PrintWriter(System.out);
> }
> else {
> useOutFile = true;
> /* SNIP-SNAP .... */
> }
> // Proceed with the real output
> while (null != (token = lexer.nextToken())) {
> int tokenType = token.getType();
> if (tokenType == -1) break;
> tknOut.printf("%4d,%3d, tkn:%4d name:%-8s\t '%s'%n",
> token.getLine(), token.getCharPositionInLine(),
> tokenType, MyOwnParser.tokenNames[tokenType],
> token.getText());
> }
> // Need to close or flush the Writer to get the output
> if (useOutFile) tknOut.close();
> else tknOut.flush();
> }
>
> Call like this ...
> {
> MyOwnLexer lexer = new MyOwnLexer(input);
>
> // Dump all tokens?
> generateTokenOutput(lexer, null);
> }
>
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>
More information about the antlr-interest
mailing list