[antlr-interest] Antlr 3 Lexer problem

[Geoffrey Zhu]

I can't do this. I am constructing complicated lexer rules because in my
input there are code blocks within which the grammar is totally
different from the rest of the file. My file is a custom mini-language.
Embeded in it there are some SQL statements with surrounding
parentheses. My program does not need to deal with full SQL syntax. I
only need to correctly recognize the beginning and ending of the SQL
statements. 

What I am thinking is to match one code block with one single token, as
ANTLR did with the "action" token (
http://fisheye2.cenqua.com/browse/antlr/src/org/antlr/tool/antlr.g?r=324
4#l1186 ). Below is just a simplified example to show that the lexer
seems to choke... 





I just need to figure out how to make it not to choke.  


-----Original Message-----
From: antlr-interest-bounces at antlr.org
[mailto:antlr-interest-bounces at antlr.org] On Behalf Of Jim Idle
Sent: Tuesday, June 26, 2007 1:32 PM
To: antlr-interest at antlr.org
Subject: Re: [antlr-interest] Antlr 3 Lexer problem

Don't try to construct syntactical context in the lexer basically:

grammar T;
 
options {
 	language=CSharp;
 	backtracking=true;
}
 
ID : ('a'..'z')+;
 
LPAREN : '(';
 
SELECT : 'select';
 
prog: (ID | LPAREN SELECT?)+ ;


> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest- 
> bounces at antlr.org] On Behalf Of Geoffrey Zhu
> Sent: Tuesday, June 26, 2007 11:28 AM
> To: antlr-interest at antlr.org
> Subject: [antlr-interest] Antlr 3 Lexer problem
> 
> Hi,
> 
> I have a simple lexer problem. The below grammar will fail on "( 
> security" in the lexer, because when the lexer sees '(' and then sees 
> 's' , it will try to match the LP_SELECT rule, which cannot be
matched.
> In this case, I want the lexer to return LPAREN and then ID. How can I

> do that?
> 
> Thanks a lot,
> Geoffrey
> 
> 
> grammar T;
> 
> options {
> 	language=CSharp;
> 	backtracking=true;
> }
> 
> ID : ('a'..'z')+;
> 
> LPAREN : '(';
> 
> LP_SELECT : LPAREN 'select';
> 
> prog: (ID | LPAREN | LP_SELECT)+ ;
> 
> _______________________________________________________
> 
> The  information in this email or in any file attached hereto is 
> intended only for the personal and confiden- tial  use  of  the 
> individual or entity to which it is addressed and may contain 
> information that is  propri- etary  and  confidential.  If you are not

> the intended recipient of this message you are hereby notified that 
> any  review, dissemination, distribution or copying of this message is

> strictly prohibited.  This  communica- tion  is  for information 
> purposes only and should not be regarded as an offer to sell or as  a

> solicitation of an offer to buy any financial product. Email trans- 
> mission cannot be guaranteed to be  secure  or  error- free. P6070214