To Sinan --> RE: [antlr-interest] Simple questions

mzukowski at bco.com mzukowski at bco.com
Mon Mar 4 06:26:03 PST 2002


You don't have a '!' in your ID rule, so you would never match connect!
because you never match the !.  The way literals work is they are a table of
words and token numbers.  At the end of any rule with testLiterals=true on
the table is consulted and  the token type is changed if there was one in
the table.  You don't need k=8, you need your ID rule to have all the
characters that are possible in the literals table.

 ID 
optons { testLiterals = true; }
	:	(LETTER|UNDERSCORE)
 (LETTER|UNDERSCORE|MINUS|DOT|DIGIT|'!')*
 	;

Monty

> -----Original Message-----
> From: Eli Mesika [mailto:elim at tti-telecom.com]
> Sent: Sunday, March 03, 2002 8:11 AM
> To: 'antlr-interest at yahoogroups.com'
> Subject: To Sinan --> RE: [antlr-interest] Simple questions
> 
> 
> Thanks for your quick respond.
> 
> Since I have another lexer rule :
> 
> ID
> 	:	(LETTER|UNDERSCORE)
> (LETTER|UNDERSCORE|MINUS|DOT|DIGIT)*
> 	;
> 
> and I have added the following :
> 
> CATCH_LITERAL_TOKENS: "connect!"
> 
> I have to use k=8 in the lexer options.
> 
> Isn't that expensive ???
> I used till now k=2 for my grammar, is there any other 
> solution that doesn't
> force me to increase k ???
> 
> Thanks
> 
> 
> -----Original Message-----
> From: Sinan [mailto:sinan.karasu at boeing.com]
> Sent: Thursday, February 28, 2002 7:53 PM
> To: antlr-interest at yahoogroups.com
> Subject: Re: [antlr-interest] Simple questions
> 
> 
> Eli Mesika wrote:
> > 
> > HI folks
> > 
> > I have a real problem in using '!' in the Antlr Tokens section.
> > I have tried Matthew's solution, it doesn't work.
> > 
> > Attached please find connect.g which describes the 
> "connect" language.
> > The language can recognize the following statement:
> > (connect a b)
> > However, the exact command is
> > (connect! a b)
> > if you are generating java files from it compiling and 
> running Main it
> will
> > accept (connect a b)
> > However if you change the CONNECT token in the .g file to 
> "connect!" and
> > test (connect! a b) you
> > will get an error.
> > 
> > Am I missing something ???
> > 
> > Please HELP.
> > Thanks
> 
> 
> The way nextToken works in lexer, you have to have a rule that accepts
> the token before a hash
> table is consulted for keywords.
> 
> so do in the lexer
> 
> CATCH_LITERAL_TOKENS:  ( "connect!" | "whatever_else_i_want!@#") ;
> 
> so on...
> 
> you HAVE TO HAVE testLiterals=true; for this to work.....
> 
> 
> I think nextToken should check the keyword  table after consuming 
> until a special token ( such as WS ) is seen. But both Ter and Ric are
> pretty busy these days.....
> 
>  
> 
> Your use of Yahoo! Groups is subject to 
> http://docs.yahoo.com/info/terms/ 
> 
> 
>  
> 
> Your use of Yahoo! Groups is subject to 
> http://docs.yahoo.com/info/terms/ 
> 
> 
> 

 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 



More information about the antlr-interest mailing list