[antlr-interest] Re: Folding Text
lgcraymer
lgc at mail1.jpl.nasa.gov
Sun Feb 29 22:11:55 PST 2004
Yes, it is possible--it will require a syntactic predicate to skip
over the line continuation if you do the processing directly. It
would be easier to do it as a token filter.
However, this looks like you are trying to do too much in the lexer (a
pretty common early mistake--part of the learning process is where to
divide processing passes). Ask yourself why you want "value ... line
2" as a single token. If you really want it as one string for post
processing, then you can re-consolidate it from multiple tokens in the
parser; that way, you can guarantee fixed spacing under your control.
If you are going to do further processing on the contents, there is
a good chance you want to keep these as separate tokens in a syntax tree.
--Loring
--- In antlr-interest at yahoogroups.com, "offline314159" <offline at s...>
wrote:
> The only snag with this is, i don't want to get the token split there.
>
> Basically, if i have the text
>
> NAME:value part 1 is
> extended onto line 2
>
> i want to see two tokens:
> one with type NAME text "NAME"
> and another with type VALUE text "value part 1 is extended onto line 2"
>
> Is this possible?
>
> --- In antlr-interest at yahoogroups.com, "lgcraymer" <lgc at m...> wrote:
> > This one's easy--note the use of "SKIP" for whitespace in the
> > examples. Basically you want something like
> >
> > ENDL :
> > '\n'
> > ( (' ' | '\t')+ { _ttype = SKIP; } | )
> > ;
> >
> > which looks for whitespace after a newline; if present, mark the token
> > as one to be skipped (you might need Token.SKIP instead of SKIP) so
> > that the parser does not see it.
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/antlr-interest/
<*> To unsubscribe from this group, send an email to:
antlr-interest-unsubscribe at yahoogroups.com
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
More information about the antlr-interest
mailing list