[antlr-interest] Re: Folding Text

lgcraymer lgc at mail1.jpl.nasa.gov
Sun Feb 29 22:11:55 PST 2004


Yes, it is possible--it will require a syntactic predicate to skip
over the line continuation if you do the processing directly.  It
would be easier to do it as a token filter.

However, this looks like you are trying to do too much in the lexer (a
pretty common early mistake--part of the learning process is where to
divide processing passes).  Ask yourself why you want "value ... line
2" as a single token.  If you really want it as one string for post
processing, then you can re-consolidate it from multiple tokens in the
parser; that way, you can guarantee fixed spacing under your control.
  If you are going to do further processing on the contents, there is
a good chance you want to keep these as separate tokens in a syntax tree.

--Loring 


--- In antlr-interest at yahoogroups.com, "offline314159" <offline at s...>
wrote:
> The only snag with this is, i don't want to get the token split there.
> 
> Basically, if i have the text
> 
> NAME:value part 1 is
>      extended onto line 2
> 
> i want to see two tokens:
> one with type NAME text "NAME"
> and another with type VALUE text "value part 1 is extended onto line 2"
> 
> Is this possible?
> 
> --- In antlr-interest at yahoogroups.com, "lgcraymer" <lgc at m...> wrote:
> > This one's easy--note the use of "SKIP" for whitespace in the
> > examples.  Basically you want something like
> > 
> > ENDL :
> >       '\n'
> >       ( (' ' | '\t')+ { _ttype = SKIP; } | )
> >      ;
> > 
> > which looks for whitespace after a newline; if present, mark the token
> > as one to be skipped (you might need Token.SKIP instead of SKIP) so
> > that the parser does not see it.



 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/
 



More information about the antlr-interest mailing list