[antlr-interest] Token Stream Question

Terence Parr parrt at cs.usfca.edu
Tue Jul 22 09:56:21 PDT 2003


Howdy.  That unexpected char in nextToken implies that it received a 
char from the input stream that does not predict any of the tokens.  
Could be a weird EOF thing 0xFF.  It's on a PC, which is not something 
I tested the notes on sorry.  Perhaps drop the charVocab to '\u00FE' 
and see what happens.

Ter

On Tuesday, July 22, 2003, at 09:51 AM, colettekirwan wrote:

> Hi
>
> I was wondering if some could please explain what I am doing wrong.
>
> Using the "Lexical Analysis With Antlr" Lecture Notes, I copied the
> example that explains how to ignore but not throw away whitespaces.
>
> But everytime I run the example I get the following error.
> D:\antlrTest>java -classpath D:\antlrTest\antlr.jar;. TestMain <
> test.in
> a = 2;
> {
>
>       a=3;
>
> }
>
> b=aException in thread "main" line 9:8: unexpected char: 0x?F
>         at TestLexer.nextToken(TestLexer.java:148)
>         at antlr.TokenStreamHiddenTokenFilter.consume
> (TokenStreamHiddenTokenFilt
> er.java:38)
>         at antlr.TokenStreamHiddenTokenFilter.nextToken
> (TokenStreamHiddenTokenFi
> lter.java:148)
>         at antlr.TokenBuffer.fill(TokenBuffer.java:69)
>         at antlr.TokenBuffer.LT(TokenBuffer.java:86)
>         at antlr.LLkParser.LT(LLkParser.java:56)
>         at TestParser.stat(TestParser.java:121)
>         at TestParser.slist(TestParser.java:61)
>         at TestMain.main(TestMain.java:25)
>
>
>
> I am using the latest version of antlr, and believe I have copied the
> grammar code from the article correctly.
>
> Below is a copy of the grammar code I am using.
> class TestParser extends Parser;
>
> {
> public void hidden(Token tok)
> {
> antlr.CommonHiddenStreamToken t = (antlr.CommonHiddenStreamToken)tok;
> for ( ; t!=null ; t=TestMain.filter.getHiddenAfter(t) )
>   {
>   System.out.print(t.getText());
>   }
> }
> }
>
> slist
> : ( stat )+
> ;
>
> stat: l:LBRACE {hidden(l);} (stat)+ r:RBRACE {hidden(r);}
> | id:ID {hidden(id);} a:ASSIGN {hidden(a);} expr s:SEMI {hidden(s);}
> ;
>
> expr: i1:INT {hidden(i1);}
> | i2:ID {hidden(i2);}
> ;
>
> class TestLexer extends Lexer;
> options {
> charVocabulary='\u0000'..'\uFFFE';
> }
> WS : (' '
> | '\t'
> | ('\n'|'\r'('\n')?) {newline();}
> )+
> ;
>
>
>
> SL_COMMENT
> : "//"
> (~('\n'|'\r'))* ('\n'|'\r'('\n')?)
> {newline();}
> ;
>
> LBRACE: '{'
> ;
>
> RBRACE: '}'
> ;
>
> LPAREN: '('
> ;
> RPAREN: ')'
> ;
>
> STAR: '*'
> ;
>
> PLUS: '+'
> ;
> SEMI: ';'
> ;
>
> ASSIGN: '='
> ;
>
> protected
> DIGIT
> : '0'..'9'
> ;
>
> INT : (DIGIT)+
> ;
>
> ID : ('a'..'z')+
> ;
>
>
> Any help with the above quesion, one be greatly appreciated.
> Also if any one knows the new location to the Description of the
> preserve Whitespace example, that  would be great
>
>
> Thanks
>
>
>
>
>
> Your use of Yahoo! Groups is subject to 
> http://docs.yahoo.com/info/terms/
>
>
>
--
Professor Comp. Sci., University of San Francisco
Creator, ANTLR Parser Generator, http://www.antlr.org
Co-founder, http://www.jguru.com
Co-founder, http://www.knowspam.net enjoy email again!
Co-founder, http://www.peerscope.com pure link sharing




 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 




More information about the antlr-interest mailing list