[antlr-interest] Re: Skipping irrelevant tokens in a source file

[trocusius]

Here it is how I did it using the TokenStreamSelector. 

Everything outside the "<'" and "'>" markers is ignored.

===IgnoreLexer.g
header {
  import antlr.*;
}
class IgnoreLexer extends Lexer; 
options {
  k = 2;
}
{
    TokenStreamSelector selector;
}

VOCAB
  :
    '\3'..'\377' { _ttype = Token.SKIP; }
  ;
FEATURE
  :
    "<'" {selector.select("fLexer");}
  ;
===FeatureLexer.g
header {
  import antlr.*;
}
class FeatureLexer extends Lexer;
options {
    exportVocab=FeatureLexer;
}
{
    TokenStreamSelector selector;
}
// Whitespace -- ignored
WS	:	(	' '
		|	'\t'
		|	'\f'
			// handle newlines
		|	(	options {generateAmbigWarnings=false;}
			:	"\r\n"  // Evil DOS
			|	'\r'    // Macintosh
			|	'\n'    // Unix (the right way)
			)
			{ newline(); }
		)+
		{ _ttype = Token.SKIP; }
	;
// Single-line comments
SL_COMMENT
	:	"//"
		(~('\n'|'\r'))* ('\n'|'\r'('\n')?)
		{$setType(Token.SKIP); newline();}
	;
ID
	:	
        ('a'..'z'|'A'..'Z') ('a'..'z'|'A'..'Z'|'0'..'9'|'_'|'.')*
    ;

ENDFEATURE
    :
        "'>" {selector.select("iLexer");}
    ;
SEMI        : ";";
EQ          : "=";
===FeatureMain.java
import antlr.*;
import java.io.*;

class FeatureMain {

public static void main(String[] args) {
try {
	TokenStreamSelector selector =
		 new TokenStreamSelector();
	//
	// Lexer
	//
	IgnoreLexer iLexer = 
		 new IgnoreLexer(new DataInputStream(System.in));
	iLexer.selector = selector;
	FeatureLexer fLexer =
		 new FeatureLexer(iLexer.getInputState());
	fLexer.selector = selector;
		selector.addInputStream(iLexer, "iLexer");
	selector.addInputStream(fLexer, "fLexer");
	selector.select("iLexer");
		Token t = selector.nextToken();
	while(t.getType() != Token.EOF_TYPE) {
		 System.out.println(t);
		 t = selector.nextToken();
	};
 } catch (Exception e) {
	System.err.println("exception: " + e);
	e.printStackTrace();
 }
}	 
}
===test.feature
ana are mere
<'
  ana = are;
  nicu = cere;
'>
nicu i le cere

--- In antlr-interest at yahoogroups.com, Matt Benson <gudnabrsam at y...>
wrote:
> Thanks, Monty.
> 
> -Matt
> 
> --- mzukowski at y... wrote:
> > The main thing is to not have feedback from the
> > parser to the lexer.
> > 
> > -----Original Message-----
> > From: Matt Benson [mailto:gudnabrsam at y...] 
> > Sent: Tuesday, August 19, 2003 9:23 AM
> > To: antlr-interest at yahoogroups.com
> > Subject: Re: [antlr-interest] Skipping irrelevant
> > tokens in a source file
> > 
> > 
> > Wouldn't a solution based on this approach imply
> > putting state in the Lexer?  Is that good?  I'm
> > asking
> > because I honestly don't know.
> > 
> > Thanks,
> > Matt
> > 
> > --- "Rodrigo B. de Oliveira" <rbo at a...> wrote:
> > >
> >
> http://www.antlr.org/doc/lexer.html#Filtering%20Input%20Streams
> > > 
> > > ----- Original Message -----
> > > From: <dm at c...>
> > > To: <antlr-interest at yahoogroups.com>
> > > Sent: Tuesday, August 19, 2003 12:45 PM
> > > Subject: [antlr-interest] Skipping irrelevant
> > tokens
> > > in a source file
> > > 
> > > 
> > > > Hi:
> > > >
> > > > What is the easiest way to ignore sections of a
> > > source file?
> > > >
> > > > Say I have a language and all I am interested
> > in,
> > > is the text between the
> > > token
> > > > "feature" and the token "end".
> > > >
> > > > How do I allow anything before feature and after
> > > end?
> > > >
> > > > Thanks in advance,
> > > >
> > > > Dave Makalsky
> > > >
> > > >
> > -------------------------------------------------
> > > > This mail sent through IMP:
> > http://horde.org/imp/
> > > >
> > > >
> > > >
> > > > Your use of Yahoo! Groups is subject to
> > > http://docs.yahoo.com/info/terms/
> > > >
> > > >
> > > >
> > > 
> > > 
> > >  
> > > 
> > > Your use of Yahoo! Groups is subject to 
> > > http://docs.yahoo.com/info/terms/
> > > 
> > > 
> > 
> > 
> > __________________________________
> > Do you Yahoo!?
> > Yahoo! SiteBuilder - Free, easy-to-use web site
> > design software
> > http://sitebuilder.yahoo.com
> > 
> >  
> > 
> > Your use of Yahoo! Groups is subject to
> > http://docs.yahoo.com/info/terms/ 
> > 
> > 
> >  
> > 
> > Your use of Yahoo! Groups is subject to
> > http://docs.yahoo.com/info/terms/ 
> > 
> > 
> 
> 
> __________________________________
> Do you Yahoo!?
> Yahoo! SiteBuilder - Free, easy-to-use web site design software
> http://sitebuilder.yahoo.com


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/