[antlr-interest] Line continuation character problem

[Craig Barker]

Hi Anthony,
 
Thanks for your input to this, it's been doing my head in!
 
Indeed, I've got the following worked out so far:

options {
 exportVocab=PVocab;
 k=3;
filter=AMP;
}
 
protected
AMP
//an & and line break.
 :  '&' '\r' '\n' {System.out.println("& HIT!");newline();}
 ; 
 
STRING_LITERAL
// string literals - the only token that can be split with an '&'.
options {paraphrase = "a string";}
 : '"'! (~('"'))* '"'!
 ;
 
This solves the first problem very well but the second point is that I still want the STRING_LITERAL token to be passed as one token, even when it's been split over 2 or more lines.
 
ie:
 
getrec(tag,"select * from acct,x__ir &
where acct001 = x__ir001 and &
x__ir006 = 'V'")
 
Effectively I need the 3 lines of strings to be treated as 1 single STRING_LITERAL token, eg:
 
getrec(tag,"select * from acct,x__ir where acct001 = x__ir001 and x_ir006 = 'V'")
 
and unfortuately shaving the last 3 characters wouldn't always work.
 
Again many thanks for your help,
 
Cheers
 
Craig

	-----Original Message----- 
	From: Anthony Youngman [mailto:Anthony.Youngman at ECA-International.com] 
	Sent: Thu 10/06/2004 10:16 
	To: antlr-interest at yahoogroups.com 
	Cc: 
	Subject: RE: [antlr-interest] Line continuation character problem
	
	
	Okay.
	 
	Define a lexer rule as '&' 'linefeed', and mark it with token type "skip". That should get round your original continuation line problem (unless you can split in the middle of a token). I do note you say you've solved this, though ...
	 
	And look at the example in the manual of lexing ESCape characters inside of a character string. You'll probably need a different lexer rule (almost identical to the one above, though). But if this rule is triggered inside a string, you want to finish it with something like "token.setlength( token.getlength() - 3)" which will lose the &\r\n that's just been added ...
	 
	Cheers,
	Wol

  _____  

	From: Craig Barker [mailto:cb at phoebussoftware.com] 
	Sent: 10 June 2004 10:10
	To: antlr-interest at yahoogroups.com
	Subject: RE: [antlr-interest] Line continuation character problem
	
	
	Hi Folks,
	 
	I have a small problem that, although trawling all the manuals, examples etc, i can't find an example of (although i'm sure i've read about it somewhere!).
	 
	I'm trying to analyse a language where each statement appears on a single line, apart from when the '&' character is explicitly used as a line continuation character. 
	 
	A statement like:
	 
	and(a>1,b<2) can also be written
	 
	and (a>1,&
	b<2)
	 
	The only other issue is that a string token can actually be validly split by '& \r \n' so we may have:
	 
	and(a>"abc",b<"def")
	 
	written as:
	 
	and(a>"abc",b<"d&
	ef")
	 
	 
	I've tried using a lexer filter and that works fine for the first example but either i'm incorrectly using it in the second example or as the lexer will already be partway through lexing the string (done as per the default java grammer in the distro.) it cannot invoke the filter rule.
	 
	Is the best way of dealing with this a change in the lexing class to automatically filter out any '&\r\n''s from the character stream before it hits the lexer?
	 
	If there's another way or if anyone can point me to an example I would be most grateful.
	 
	Many thanks
	 
	Craig
	
	
	****************************************************************************
	
	This transmission is intended for the named recipient only. It may contain private and confidential information. If this has come to you in error you must not act on anything disclosed in it, nor must you copy it, modify it, disseminate it in any way, or show it to anyone. Please e-mail the sender to inform us of the transmission error or telephone ECA International immediately and delete the e-mail from your information system.
	
	Telephone numbers for ECA International offices are: Sydney +61 (0)2 8272 5300, Hong Kong + 852 2121 2388, London +44 (0)20 7351 5000 and New York +1 212 582 2333.
	
	****************************************************************************
	
	
  _____  

	Yahoo! Groups Links
	

	*	To visit your group on the web, go to:
		http://groups.yahoo.com/group/antlr-interest/
		  
	*	To unsubscribe from this group, send an email to:
		antlr-interest-unsubscribe at yahoogroups.com <mailto:antlr-interest-unsubscribe at yahoogroups.com?subject=Unsubscribe> 
		  
	*	Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service <http://docs.yahoo.com/info/terms/> . 




 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/
 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/ms-tnef
Size: 10918 bytes
Desc: not available
Url : http://www.antlr.org/pipermail/antlr-interest/attachments/20040610/1ce4608d/attachment.bin