[antlr-interest] A Simple Question on Channels

[Indhu Bharathi]

As expected, there were some bugs :-) Here is a implementation that 
works fine:

grammar Test;

@members{

                private boolean newLineBeforeNextToken( int index ) {

                Token tkn;
               
                input = (DebugTokenStream) input;

                while( input.size()>(index+1) && 
(tkn=input.get(++index)).getChannel() != Token.DEFAULT_CHANNEL ) {

                                if( tkn.getType() == NL ) return true;

                                index++;

                }
                return false;

                }

}

r              :               t=ID { newLineBeforeNextToken( 
$t.getTokenIndex() ) }?=>/*nothing*/
                ;

NL           :               '\n' {$channel=HIDDEN;}

                ;

ID            :               'a'..'z'+

                ;


"foo<NL>" will pass whereas "foo" will fail  :-)

I casted input to 'DebugTokenStream' since I was running inside 
ANTLRWorks. You can cast it to CommonTokenStream.

Cheers, Indhu
 

Indhu Bharathi wrote:
>
> I don't have ANTLR right now to test this. But guess it would work
>
>  
>
> NEWLINE    :     '\r'?'\n' {$channel = HIDDEN;};
>
>  
>
> Then write expression_statement rule like this:
>
>  
>
> expression_statement
>
>     :    expression { NewLineBeforeNextToken( input.LT(1) ) }?=> 
> /*nothing*/
>
>     ;
>
>  
>
> And in the @members section you can write something like
>
>  
>
> private boolean NewLineBeforeNextToken( Token tkn ) {
>
>                 int index = tkn.getTokenIndex();
>
>                 while( (tkn=input.get(index).getChannel()) != DEFAULT) {
>
>                                 if( tkn.getType == NEWLINE ) return true;
>
>                                 index++;
>
>                 }
>
>                 return false;
>
> }
>
>  
>
> Hope that helps.
>
>  
>
> - Indhu
>
>  
>
> *From:* antlr-interest-bounces at antlr.org 
> [mailto:antlr-interest-bounces at antlr.org] *On Behalf Of *Andreas Stefik
> *Sent:* Friday, April 03, 2009 8:02 AM
> *To:* antlr-interest at antlr.org
> *Subject:* [antlr-interest] A Simple Question on Channels
>
>  
>
> Hello list,
>
> I am an academic working on a new compiler architecture. I've written 
> a number of compilers in the past in other languages, including JavaCC 
> and Lex/Yacc, and am pretty well versed in compiler theory, and 
> thought that it might be fun and interesting to try out ANTLR. As 
> such, I've written some practice grammars to get my feet wet, am 
> nearly finished reading Terence's fantastic book, and have started on 
> a first draft of a grammar for my little language in ANTLR. So far, 
> I've run into very few problems, and am positively in love with the 
> visualization system built into ANTLRWorks, but if the list will 
> indulge a new user, I do have one question, related to how "channels" 
> are used, and how I can manipulate them.
>
> Here's the basic problem I'm trying to figure out:
>
> In my programming language, I would like to use newlines as a 
> delimiter in some cases, but also to have it thrown away in "most" 
> cases. My initial guess as to how to do this would be something like this:
>
> NEWLINE    :     '\r'?'\n' {$channel = HIDDEN;};
>
> Where a newline is tracked, and not thrown away, but put away for 
> usage if I decide I need it. Now, ideally, what I would LIKE to do, is 
> something like the following:
>
> expression_statement
>     :    expression NEWLINE
>     ;
>
> Now, I know I can't do this, as the parser reads only from the default 
> channel, not the hidden channel, but even if it did read from the 
> hidden channel, there would be all sorts of garbage in the way anyway 
> (other, unrelated, white space). But, as the above statement implies, 
> I would like to determine if there is a newline token on the hidden 
> channel that is between whatever tokens end "expression" and begin 
> whatever rule is after my newline.
>
> After looking through the documentation, my best "guess" is that I 
> need to write an action, that will get the actual token stream from 
> the hidden channel (which I'm not completely sure how to do), then 
> check all the hidden tokens between that and the beginning of the next 
> rule (Which I'm also not completely sure how to do). If a NEWLINE is 
> found, I would issue an ACCEPT; as that's the end of the rule, 
> otherwise it isn't. However, all else being equal, I would really 
> prefer if there was a way to temporarily tell my grammar to search for 
> _only_ newlines in the hidden channel, but otherwise to process normally.
>
> If anyone has any thoughts or suggestions on the right way to go about 
> this, I would appreciate it.
>
> Sincerely,
>
> Andreas
>
> ------------------------------------------------------------------------
>
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20090403/a04f5b3d/attachment.html