[antlr-interest] code translation

[Benoit Fouletier]

On Thu, Jul 2, 2009 at 6:51 PM, Terence Parr <parrt at cs.usfca.edu> wrote:

>
> On Jul 2, 2009, at 9:43 AM, Benoit Fouletier wrote:
>
> Anyway... the translation itself is relatively straightforward, ANTLR is
> amazing for that!, but preserving comments & formatting is killing me. The
> best resources I found were all in v2, and much seems to have changed (the token
> stream filtering/splitting <http://www.antlr2.org/doc/streams.html>, and
> the "hidden channel" helpers are gone).
>
>
> Really?  $channel = HIDDEN doesn't work in lexer?
>

Oh yes it works, but I haven't found a convenient way to dump it back out. I
was talking about the whole CommonHiddenToken, ASTWithHiddenTokenStream,
etc, and getHiddenBefore/After, those are gone (right?).

I ended up doing it manually like so:

private void dumpHidden(Tree p)
{
 Token t = ((CommonTree)p).token;
dumpHidden(t);
}

private void dumpHidden(int start)
{
CommonTokenStream tokens = InstrMain.tokens;
for (int i = start; i < tokens.size(); i++)
 {
Token t = tokens.get(i);
if (t.getChannel() == HIDDEN)
 System.out.print(t.getText());
else
break;
 }
}


Works, but tedious... feels like a regression!

Also the "token stream specification" hints at ways to automatically get all
hidden tokens from a specific tree, but I haven't figured out how (of even
if it's actually been implemented in v3).

I'm nearly through converting the preserveWithSpace example to v3 (my only
problem is the manual callDumpInstrumentation, in v2 it took the original
tree as argument, in v3 no argument but it doesn't rewind the tree so the
original call cannot continue).
I'm not sure I'm making any sense (pretty late here ;), I'll try and post
the complete code tomorrow (btw does this list accept attachments?).

      Ben



>
> Ter
>
>  I understand v3 is supposed to be more generic & flexible (multiple
> channels instead of just "hidden or not"), but I just don't understand what
> the intended way is for a translation scenario.
>
>         Ben
>
>
> On Thu, Jul 2, 2009 at 5:38 PM, James Abley <james.abley at gmail.com> wrote:
>
>> 2009/6/24 Benoit Fouletier <benblo+ANTLR at gmail.com<benblo%2BANTLR at gmail.com>
>> >:
>> > Hi!
>> > Newbie to ANTLR here, hello all =).
>> > I'm trying to make a translator from Unity's JavaScript (sometimes
>> called
>> > UnityScript) to C#. This particular JS implementation is made to compile
>> to
>> > IL and run on Mono. It's actually based on Boo, the grammar was written
>> > by Rodrigo B. de Oliveira with ANTLR (don't know which version).
>> > So the 2 languages are very similar, they don't differ in principle,
>> only in
>> > syntax. I've already achieved a lot with regular expressions but that
>> just
>> > doesn't seem like the right tool for the job.
>> > 90% of the differences are in variable & function declaration, the core
>> is
>> > the same.
>> > Basically, I need to translate this:
>> >
>> > function MyFunction(i : int, b : boolean) : String
>> > {
>> >     return i + " " + b;
>> > }
>> >
>> > to this:
>> >
>> > private string MyFunction(int i, boolean b)
>> > {
>> >     return i + " " + b;
>> > }
>> >
>> > I have access to the compiled lexer & parser (but not the original
>> grammar).
>> > I've tried to go through the token stream and manually rearrange the
>> > declarations (which works but again, doesn't feel like the best way to
>> do
>> > it).
>> > The thing is, I want to retain comments & formatting during translation,
>> but
>> > the lexer doesn't give me the hidden tokens* (and anyway my little
>> fiddlings
>> > would probably explode!).
>> > Also, I don't need a complete compiler: all the code I want to translate
>> > already compiles so I know it's valid.
>> >
>> > It seems to me like my needs are fairly simple (!), but I don't know
>> what
>> > approach to take. If I were to rewrite the grammar (or just modify the
>> > ECMAScript grammar found here), are there ways to define translate
>> rules, in
>> > the fashion of regex replacement? Or should I be able to get away with
>> using
>> > the compiled lexer I have?
>> >
>> > Sorry my post is so long, I hope I've made the context clear. Cheers =),
>> >
>> >         Ben
>> >
>> > * I've read the token streams article in the v2 docs (it's not in the v3
>> > docs, is it still valid?), which is very informative... unfortunately
>> when I
>> > setTokenCreator to a hidden stream, I don't get anything more.
>> > Here's my loop:
>> > UnityScriptLexer lexer = new UnityScriptLexer(file);
>> > lexer.setTokenCreator(new
>> > antlr.CommonHiddenStreamToken.CommonHiddenStreamTokenCreator());
>> >
>> > antlr.IHiddenStreamToken token;
>> > while ((token = (antlr.IHiddenStreamToken)lexer.nextToken()).Type !=
>> EOF)
>> > {
>> >     if (token.getHiddenBefore() != null)
>> >         LogToken(token.getHiddenBefore());   // this never happens
>> >
>> >     LogToken(token);
>> >
>> >     if (token.getHiddenAfter() != null)
>> >         LogToken(token.getHiddenAfter());    // this never happens
>> > }
>> > LogToken() just outputs to the console.
>> >
>> > List: http://www.antlr.org/mailman/listinfo/antlr-interest
>> > Unsubscribe:
>> > http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>> >
>>
>>
>> http://www.antlr.org/wiki/display/ST/Language+Translation+Using+ANTLR+and+StringTemplate
>>
>> That seems to be ANTLR 2. but relevant. You would need a grammar
>> though, rather than just the compiled lexer and parser.
>>
>> The new book is supposed to contain information around this sort of
>> request. Ter might be able to tell more about that.
>>
>> http://www.pragprog.com/titles/tpdsl/language-design-patterns
>>
>> Cheers,
>>
>> James
>>
>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
>> Unsubscribe:
>> http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>>
>
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe:
> http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20090702/0d8c0210/attachment.html