[antlr-interest] code translation

[Terence Parr]

token stream .toString(x,y) print all tokens even hidden ones.  must  
be something in the API.
Ter
On Jul 2, 2009, at 1:50 PM, Benoit Fouletier wrote:

> On Thu, Jul 2, 2009 at 6:51 PM, Terence Parr <parrt at cs.usfca.edu>  
> wrote:
>
> On Jul 2, 2009, at 9:43 AM, Benoit Fouletier wrote:
>> Anyway... the translation itself is relatively straightforward,  
>> ANTLR is amazing for that!, but preserving comments & formatting is  
>> killing me. The best resources I found were all in v2, and much  
>> seems to have changed (the token stream filtering/splitting, and  
>> the "hidden channel" helpers are gone).
>
> Really?  $channel = HIDDEN doesn't work in lexer?
>
> Oh yes it works, but I haven't found a convenient way to dump it  
> back out. I was talking about the whole CommonHiddenToken,  
> ASTWithHiddenTokenStream, etc, and getHiddenBefore/After, those are  
> gone (right?).
>
> I ended up doing it manually like so:
>
> private void dumpHidden(Tree p)
> {
> 	Token t = ((CommonTree)p).token;
> 	dumpHidden(t);
> }
>
> private void dumpHidden(int start)
> {
> 	CommonTokenStream tokens = InstrMain.tokens;
> 	for (int i = start; i < tokens.size(); i++)
> 	{
> 		Token t = tokens.get(i);
> 		if (t.getChannel() == HIDDEN)
> 			System.out.print(t.getText());
> 		else
> 			break;
> 	}
> }
>
> Works, but tedious... feels like a regression!
>
> Also the "token stream specification" hints at ways to automatically  
> get all hidden tokens from a specific tree, but I haven't figured  
> out how (of even if it's actually been implemented in v3).
>
> I'm nearly through converting the preserveWithSpace example to v3  
> (my only problem is the manual callDumpInstrumentation, in v2 it  
> took the original tree as argument, in v3 no argument but it doesn't  
> rewind the tree so the original call cannot continue).
> I'm not sure I'm making any sense (pretty late here ;), I'll try and  
> post the complete code tomorrow (btw does this list accept  
> attachments?).
>
>       Ben
>
>
>
> Ter
>>  I understand v3 is supposed to be more generic & flexible  
>> (multiple channels instead of just "hidden or not"), but I just  
>> don't understand what the intended way is for a translation scenario.
>>
>>         Ben
>>
>>
>> On Thu, Jul 2, 2009 at 5:38 PM, James Abley <james.abley at gmail.com>  
>> wrote:
>> 2009/6/24 Benoit Fouletier <benblo+ANTLR at gmail.com>:
>> > Hi!
>> > Newbie to ANTLR here, hello all =).
>> > I'm trying to make a translator from Unity's JavaScript  
>> (sometimes called
>> > UnityScript) to C#. This particular JS implementation is made to  
>> compile to
>> > IL and run on Mono. It's actually based on Boo, the grammar was  
>> written
>> > by Rodrigo B. de Oliveira with ANTLR (don't know which version).
>> > So the 2 languages are very similar, they don't differ in  
>> principle, only in
>> > syntax. I've already achieved a lot with regular expressions but  
>> that just
>> > doesn't seem like the right tool for the job.
>> > 90% of the differences are in variable & function declaration,  
>> the core is
>> > the same.
>> > Basically, I need to translate this:
>> >
>> > function MyFunction(i : int, b : boolean) : String
>> > {
>> >     return i + " " + b;
>> > }
>> >
>> > to this:
>> >
>> > private string MyFunction(int i, boolean b)
>> > {
>> >     return i + " " + b;
>> > }
>> >
>> > I have access to the compiled lexer & parser (but not the  
>> original grammar).
>> > I've tried to go through the token stream and manually rearrange  
>> the
>> > declarations (which works but again, doesn't feel like the best  
>> way to do
>> > it).
>> > The thing is, I want to retain comments & formatting during  
>> translation, but
>> > the lexer doesn't give me the hidden tokens* (and anyway my  
>> little fiddlings
>> > would probably explode!).
>> > Also, I don't need a complete compiler: all the code I want to  
>> translate
>> > already compiles so I know it's valid.
>> >
>> > It seems to me like my needs are fairly simple (!), but I don't  
>> know what
>> > approach to take. If I were to rewrite the grammar (or just  
>> modify the
>> > ECMAScript grammar found here), are there ways to define  
>> translate rules, in
>> > the fashion of regex replacement? Or should I be able to get away  
>> with using
>> > the compiled lexer I have?
>> >
>> > Sorry my post is so long, I hope I've made the context clear.  
>> Cheers =),
>> >
>> >         Ben
>> >
>> > * I've read the token streams article in the v2 docs (it's not in  
>> the v3
>> > docs, is it still valid?), which is very informative...  
>> unfortunately when I
>> > setTokenCreator to a hidden stream, I don't get anything more.
>> > Here's my loop:
>> > UnityScriptLexer lexer = new UnityScriptLexer(file);
>> > lexer.setTokenCreator(new
>> > antlr.CommonHiddenStreamToken.CommonHiddenStreamTokenCreator());
>> >
>> > antlr.IHiddenStreamToken token;
>> > while ((token =  
>> (antlr.IHiddenStreamToken)lexer.nextToken()).Type != EOF)
>> > {
>> >     if (token.getHiddenBefore() != null)
>> >         LogToken(token.getHiddenBefore());   // this never happens
>> >
>> >     LogToken(token);
>> >
>> >     if (token.getHiddenAfter() != null)
>> >         LogToken(token.getHiddenAfter());    // this never happens
>> > }
>> > LogToken() just outputs to the console.
>> >
>> > List: http://www.antlr.org/mailman/listinfo/antlr-interest
>> > Unsubscribe:
>> > http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>> >
>>
>> http://www.antlr.org/wiki/display/ST/Language+Translation+Using+ANTLR+and+StringTemplate
>>
>> That seems to be ANTLR 2. but relevant. You would need a grammar
>> though, rather than just the compiled lexer and parser.
>>
>> The new book is supposed to contain information around this sort of
>> request. Ter might be able to tell more about that.
>>
>> http://www.pragprog.com/titles/tpdsl/language-design-patterns
>>
>> Cheers,
>>
>> James
>>
>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
>> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>>
>>
>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
>> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>
>