[antlr-interest] Re: Token Stream Question

[colettekirwan]

Hi,

Still having difficulty,

I reduced the char vocab range as suggested and also checked both 
my .g and test.in file with a hex editor.

Any more suggestions would be greatly appreciated

Thanks

Colette


--- In antlr-interest at yahoogroups.com, Terence Parr <parrt at c...> 
wrote:
> Howdy.  That unexpected char in nextToken implies that it received 
a 
> char from the input stream that does not predict any of the 
tokens.  
> Could be a weird EOF thing 0xFF.  It's on a PC, which is not 
something 
> I tested the notes on sorry.  Perhaps drop the charVocab 
to '\u00FE' 
> and see what happens.
> 
> Ter
> 
> On Tuesday, July 22, 2003, at 09:51 AM, colettekirwan wrote:
> 
> > Hi
> >
> > I was wondering if some could please explain what I am doing 
wrong.
> >
> > Using the "Lexical Analysis With Antlr" Lecture Notes, I copied 
the
> > example that explains how to ignore but not throw away 
whitespaces.
> >
> > But everytime I run the example I get the following error.
> > D:\antlrTest>java -classpath D:\antlrTest\antlr.jar;. TestMain <
> > test.in
> > a = 2;
> > {
> >
> >       a=3;
> >
> > }
> >
> > b=aException in thread "main" line 9:8: unexpected char: 0x?F
> >         at TestLexer.nextToken(TestLexer.java:148)
> >         at antlr.TokenStreamHiddenTokenFilter.consume
> > (TokenStreamHiddenTokenFilt
> > er.java:38)
> >         at antlr.TokenStreamHiddenTokenFilter.nextToken
> > (TokenStreamHiddenTokenFi
> > lter.java:148)
> >         at antlr.TokenBuffer.fill(TokenBuffer.java:69)
> >         at antlr.TokenBuffer.LT(TokenBuffer.java:86)
> >         at antlr.LLkParser.LT(LLkParser.java:56)
> >         at TestParser.stat(TestParser.java:121)
> >         at TestParser.slist(TestParser.java:61)
> >         at TestMain.main(TestMain.java:25)
> >
> >
> >
> > I am using the latest version of antlr, and believe I have copied 
the
> > grammar code from the article correctly.
> >
> > Below is a copy of the grammar code I am using.
> > class TestParser extends Parser;
> >
> > {
> > public void hidden(Token tok)
> > {
> > antlr.CommonHiddenStreamToken t = (antlr.CommonHiddenStreamToken)
tok;
> > for ( ; t!=null ; t=TestMain.filter.getHiddenAfter(t) )
> >   {
> >   System.out.print(t.getText());
> >   }
> > }
> > }
> >
> > slist
> > : ( stat )+
> > ;
> >
> > stat: l:LBRACE {hidden(l);} (stat)+ r:RBRACE {hidden(r);}
> > | id:ID {hidden(id);} a:ASSIGN {hidden(a);} expr s:SEMI {hidden
(s);}
> > ;
> >
> > expr: i1:INT {hidden(i1);}
> > | i2:ID {hidden(i2);}
> > ;
> >
> > class TestLexer extends Lexer;
> > options {
> > charVocabulary='\u0000'..'\uFFFE';
> > }
> > WS : (' '
> > | '\t'
> > | ('\n'|'\r'('\n')?) {newline();}
> > )+
> > ;
> >
> >
> >
> > SL_COMMENT
> > : "//"
> > (~('\n'|'\r'))* ('\n'|'\r'('\n')?)
> > {newline();}
> > ;
> >
> > LBRACE: '{'
> > ;
> >
> > RBRACE: '}'
> > ;
> >
> > LPAREN: '('
> > ;
> > RPAREN: ')'
> > ;
> >
> > STAR: '*'
> > ;
> >
> > PLUS: '+'
> > ;
> > SEMI: ';'
> > ;
> >
> > ASSIGN: '='
> > ;
> >
> > protected
> > DIGIT
> > : '0'..'9'
> > ;
> >
> > INT : (DIGIT)+
> > ;
> >
> > ID : ('a'..'z')+
> > ;
> >
> >
> > Any help with the above quesion, one be greatly appreciated.
> > Also if any one knows the new location to the Description of the
> > preserve Whitespace example, that  would be great
> >
> >
> > Thanks
> >
> >
> >
> >
> >
> > Your use of Yahoo! Groups is subject to 
> > http://docs.yahoo.com/info/terms/
> >
> >
> >
> --
> Professor Comp. Sci., University of San Francisco
> Creator, ANTLR Parser Generator, http://www.antlr.org
> Co-founder, http://www.jguru.com
> Co-founder, http://www.knowspam.net enjoy email again!
> Co-founder, http://www.peerscope.com pure link sharing


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/