[antlr-interest] Re: Token Stream Question

Terence Parr parrt at cs.usfca.edu
Tue Jul 22 10:24:58 PDT 2003


Is there a weird char on the end?  Is it that weird PC ctrl-z char?

Perhaps you should modify the output lexer file to put in a print 
statement and check what that char/int is exactly.  Or, run thru 
debugger :)

Ter

On Tuesday, July 22, 2003, at 10:13 AM, colettekirwan wrote:

> Hi,
>
> Still having difficulty,
>
> I reduced the char vocab range as suggested and also checked both
> my .g and test.in file with a hex editor.
>
> Any more suggestions would be greatly appreciated
>
> Thanks
>
> Colette
>
>
> --- In antlr-interest at yahoogroups.com, Terence Parr <parrt at c...>
> wrote:
>> Howdy.  That unexpected char in nextToken implies that it received
> a
>> char from the input stream that does not predict any of the
> tokens.
>> Could be a weird EOF thing 0xFF.  It's on a PC, which is not
> something
>> I tested the notes on sorry.  Perhaps drop the charVocab
> to '\u00FE'
>> and see what happens.
>>
>> Ter
>>
>> On Tuesday, July 22, 2003, at 09:51 AM, colettekirwan wrote:
>>
>>> Hi
>>>
>>> I was wondering if some could please explain what I am doing
> wrong.
>>>
>>> Using the "Lexical Analysis With Antlr" Lecture Notes, I copied
> the
>>> example that explains how to ignore but not throw away
> whitespaces.
>>>
>>> But everytime I run the example I get the following error.
>>> D:\antlrTest>java -classpath D:\antlrTest\antlr.jar;. TestMain <
>>> test.in
>>> a = 2;
>>> {
>>>
>>>       a=3;
>>>
>>> }
>>>
>>> b=aException in thread "main" line 9:8: unexpected char: 0x?F
>>>         at TestLexer.nextToken(TestLexer.java:148)
>>>         at antlr.TokenStreamHiddenTokenFilter.consume
>>> (TokenStreamHiddenTokenFilt
>>> er.java:38)
>>>         at antlr.TokenStreamHiddenTokenFilter.nextToken
>>> (TokenStreamHiddenTokenFi
>>> lter.java:148)
>>>         at antlr.TokenBuffer.fill(TokenBuffer.java:69)
>>>         at antlr.TokenBuffer.LT(TokenBuffer.java:86)
>>>         at antlr.LLkParser.LT(LLkParser.java:56)
>>>         at TestParser.stat(TestParser.java:121)
>>>         at TestParser.slist(TestParser.java:61)
>>>         at TestMain.main(TestMain.java:25)
>>>
>>>
>>>
>>> I am using the latest version of antlr, and believe I have copied
> the
>>> grammar code from the article correctly.
>>>
>>> Below is a copy of the grammar code I am using.
>>> class TestParser extends Parser;
>>>
>>> {
>>> public void hidden(Token tok)
>>> {
>>> antlr.CommonHiddenStreamToken t = (antlr.CommonHiddenStreamToken)
> tok;
>>> for ( ; t!=null ; t=TestMain.filter.getHiddenAfter(t) )
>>>   {
>>>   System.out.print(t.getText());
>>>   }
>>> }
>>> }
>>>
>>> slist
>>> : ( stat )+
>>> ;
>>>
>>> stat: l:LBRACE {hidden(l);} (stat)+ r:RBRACE {hidden(r);}
>>> | id:ID {hidden(id);} a:ASSIGN {hidden(a);} expr s:SEMI {hidden
> (s);}
>>> ;
>>>
>>> expr: i1:INT {hidden(i1);}
>>> | i2:ID {hidden(i2);}
>>> ;
>>>
>>> class TestLexer extends Lexer;
>>> options {
>>> charVocabulary='\u0000'..'\uFFFE';
>>> }
>>> WS : (' '
>>> | '\t'
>>> | ('\n'|'\r'('\n')?) {newline();}
>>> )+
>>> ;
>>>
>>>
>>>
>>> SL_COMMENT
>>> : "//"
>>> (~('\n'|'\r'))* ('\n'|'\r'('\n')?)
>>> {newline();}
>>> ;
>>>
>>> LBRACE: '{'
>>> ;
>>>
>>> RBRACE: '}'
>>> ;
>>>
>>> LPAREN: '('
>>> ;
>>> RPAREN: ')'
>>> ;
>>>
>>> STAR: '*'
>>> ;
>>>
>>> PLUS: '+'
>>> ;
>>> SEMI: ';'
>>> ;
>>>
>>> ASSIGN: '='
>>> ;
>>>
>>> protected
>>> DIGIT
>>> : '0'..'9'
>>> ;
>>>
>>> INT : (DIGIT)+
>>> ;
>>>
>>> ID : ('a'..'z')+
>>> ;
>>>
>>>
>>> Any help with the above quesion, one be greatly appreciated.
>>> Also if any one knows the new location to the Description of the
>>> preserve Whitespace example, that  would be great
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>>
>>>
>>> Your use of Yahoo! Groups is subject to
>>> http://docs.yahoo.com/info/terms/
>>>
>>>
>>>
>> --
>> Professor Comp. Sci., University of San Francisco
>> Creator, ANTLR Parser Generator, http://www.antlr.org
>> Co-founder, http://www.jguru.com
>> Co-founder, http://www.knowspam.net enjoy email again!
>> Co-founder, http://www.peerscope.com pure link sharing
>
>
>
>
> Your use of Yahoo! Groups is subject to 
> http://docs.yahoo.com/info/terms/
>
>
>
--
Professor Comp. Sci., University of San Francisco
Creator, ANTLR Parser Generator, http://www.antlr.org
Co-founder, http://www.jguru.com
Co-founder, http://www.knowspam.net enjoy email again!
Co-founder, http://www.peerscope.com pure link sharing




 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 




More information about the antlr-interest mailing list