[antlr-interest] Combining multiple tokens

Jeffrey Cameron jeffreycameron at gmail.com
Mon Oct 19 15:18:21 PDT 2009


>
> Thanks for the help to both of you.  I took Jim's recommendation and
> tweaked it a bit.  I used a subrule to accumulate the text I wanted
> and then in the rule that calls the subrule I gather the text for the
> subrule and transmit it with an imaginary token
>

So my code from before now looks like:

line_to_eol
: line_text EOL -> TEXT[$line_text.text]
;
line_text
: ( options {greedy=false;} : . )*
;

There may be some problems with this approach though it seems to be working
well in the tests I have

> ------------------------------
>
> Message: 5
> Date: Sun, 18 Oct 2009 14:24:42 -0700
> From: Graham Wideman <gwlist at grahamwideman.com>
> Subject: Re: [antlr-interest] Combining multiple tokens
> To: antlr-interest at antlr.org
> Message-ID: <4.1.20091018142036.016d4248 at grahamwideman.com>
> Content-Type: text/plain; charset="us-ascii"
>
> I'm not entirely sure I understand the particulars of what you're trying to
> do
> but...
>
> At 10/18/2009 05:01 PM, Jeffrey Cameron wrote:
> >
> > I have two parser rules in my grammar where I am trying to wrap the
> complete,
> > concatenated text of a series of tokens into a single token for a tree
> > grammar.  The rules look like this:
> >
> > #1:
> >
> > cell : (~(VBAR|EOL))* ;
> >
> > in this case I would like to write something like this:
> >
> > cell : (~(VBAR|EOL))* -> TABLECELL[$cell.text] ;
>
>
> I think you probably want syntax that's more like:
>
> -> ^(TABLECELL, $cell)
>
> See:
> http://www.antlr.org/wiki/display/ANTLR3/Tree+construction
>
> If that doesn't get you going, it might be good to post clarifying:
>
> > I am trying to wrap the complete, concatenated text
> > of a series of tokens into a single token for a tree grammar.
>
> ... are you are writing this *in* a tree grammar, or in a combined grammar
> trying to produce a tree?  Also describe what actual errors you are seeing.
>
> -- Graham
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://www.antlr.org/pipermail/antlr-interest/attachments/20091018/2cfbab19/attachment-0001.html
>
>
>
> ------------------------------
>
> Message: 10
> Date: Tue, 20 Oct 2009 13:25:12 +0530
> From: "Jim Idle" <jimi at temporal-wave.com>
> Subject: Re: [antlr-interest] Combining multiple tokens
> To: "antlr-interest at antlr.org" <antlr-interest at antlr.org>
> Message-ID: <24bb54a693d4784382c354e9de3baf30 at temporal-wave.com>
> Content-Type: text/plain; charset="us-ascii"
>
> All you need do is accumulate the text in a stringbuffer:
>
>
>
> ( s=TOK { sbuf.append($s.text); }) +
>
>
>
> Then use the string you have accumulated to set the text of an imaginary or
> even the last TOK (which will be in s).
>
>
>
> ->IMAGINARY[sbuf.toString()]
>
>
>
> Should work fine.
>
>
>
> Jim
>
>
>
> From: antlr-interest-bounces at antlr.org [mailto:
> antlr-interest-bounces at antlr.org] On Behalf Of Jeffrey Cameron
> Sent: Monday, October 19, 2009 2:32 AM
> To: antlr-interest at antlr.org
> Subject: [antlr-interest] Combining multiple tokens
>
>
>
> I have two parser rules in my grammar where I am trying to wrap the
> complete, concatenated text of a series of tokens into a single token for a
> tree grammar.  The rules look like this:
>
>
>
> #1:
>
>
>
> cell : (~(VBAR|EOL))* ;
>
>
>
> in this case I would like to write something like this:
>
>
>
> cell : (~(VBAR|EOL))* -> TABLECELL[$cell.text] ;
>
>
>
> but when I put that syntax in I get an exception.
>
>
>
> #2:
>
>
>
> line_to_eol : ( options {greedy=false;} : . )* EOL ;
>
>
>
> I would like to do something like the same as above but ignore the EOL
> token, so something like this:
>
>
>
> line_to_eol : stuff+=( options {greedy=false;} : . )* EOL ->
> TEXT[$stuff.text] ;
>
>
>
> Can someone tell me when I am doing wrong in here and how to correct it?
> I'm a bit stuck here and every other part of my grammar is working as
> expected.
>
>
>
> Thanks in advance
>
>
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://www.antlr.org/pipermail/antlr-interest/attachments/20091020/24c97f3c/attachment-0001.html
>
> ------------------------------
>
> Message: 11
> Date: Tue, 20 Oct 2009 13:29:18 +0530
> From: "Jim Idle" <jimi at temporal-wave.com>
> Subject: Re: [antlr-interest] [c target] memory leaks during error
>        recovery
> To: "antlr-interest at antlr.org" <antlr-interest at antlr.org>
> Message-ID: <43b4e74754894440b20f0721a1dbbef6 at temporal-wave.com>
> Content-Type: text/plain;       charset="iso-8859-1"
>
> It is because you are trying to do things while you parse - another reason
> to build a tree and THEN operate on the tree.
>
> Catch does not need a type in the C target, you can just use:
>
> r:
>  Ddddddd
> ;
> catch() { }
>
> (assuming 3.2 of ANTLR).
>
> The other thing you might do is break up your rule list so that exceptions
> in them do not drop out the whole rule, which is what happens in all targets
> unless you structure the rules a little. Break things down in to smaller
> units. The function call overhead (which may not even occur because of
> inlining) is very small in C.
>
> Jim
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20091019/15c5a578/attachment.html 


More information about the antlr-interest mailing list