[antlr-interest] ANTLR3ea+ANTLWorks is *really cool*, but how do I insert PythonTokenStream.java

[Prashant Deva]

Elden,
 I cant understand your problem. Could you explain it a little more,
so that i can provide a nice solution in Antlr Studio?


 PRASHANT

On 7/24/05, Elden Crom <eldencrom at comcast.net> wrote:
> Same here for now (putting a class between lexor and parser, works
> fine), but with respect to your (Terence's) quote
> "Why program by hand in five days what you can spend five years of your
> life automating."
> are we out of time yet?  :-)
> perhaps it could be added to ANTLR 3's grammar some how....
> ======================================
> ======================================
> <example that covers most of my cases...(through it wouldn't pass coding
> standards!) (make sure its in a non-proportional font to read..).>
>   ...
>   for x (0,2,12 //no DENT
>  34,43):
>     y +=x*2;if(x>12):p+=y; i++;  //INDENT before 'p' but not before 'i'
>                      p/=z;
>       /*this too?*/  p/=2;  //all of the p assignments are in the
> if(...) block
>                              //comments don't gen dent errors or cause
> indention to be aquired
>                        // the  /*..*/ comment has no effect but the
> spacing is preserved
>     q*=p;
>   while(next()):; //gen INDENT ";" and DEDENT
>   func(SomeObnoxiouslyLongIdentifier,   //no DENT here
>        AnotherObnoxiouslyLongIdentifier);
>    func2();  //error -- alignment wrong
>    if(b):
>    c=1;   //error -- indent of 'c' must be greater than 'if'
> 
> ======================================
> ======================================
> 
> The first question is does it belong in the parser or the lexor? (it
> appears to usually sit in between, as a matter of necessity now)
> I know that for my grammar I'm designing, I only want to generate an
> error if the beginning of a statement is not aligned with the previous
> and it is not preceded with a ":".  Same for Python.  My tendency is to
> think it (control) belongs in the parser.
> Some options (certainly not exhaustive):
> 1) Have a grammar flag that says always generates a INDENT and DEDENT
> token after 'newline()' has been called.
>      CON: every statement that can contains a DENT token (arrays, list,
> between if and "(" etc) must have "(DENT)? {ignoreDent()}" all over it.
>      PRO: maybe not so bad just make a rule "ig_dent: (INDENT | DEDENT)?
> {ignoreDent()}" ... still ugly
> 2) Have a lexor functions that say look for new dent after a certain
> point (":" in my case) look for new indention
>      CON: Somewhat restricts how it can be used.
>      PRO: Relatively easy to describe in the lexor and does not put
> allot of jiberish in the parser chunk.
> 3) have the Parser mess with the Lexor's head, by saying where changes
> of indent are allowed
>      CON: parser and lexor are no-long stand alone (colorization for
> grammars becomes more difficult, etc.)
>      PRO: most flexible
> 
> (the grammar for my language.....)
> See 1) above -- in the parser
> statement: statment ';' {checkNextIndent=true;}
> block: ":" {aquireIndent();} compundStatement;
> if: "if" ig_dent expersion ig_dent block
> while: "while" ig_dent expersion ig_dent block
> ....
> 
> see 2) above -- in the lexor
> SEMI: ';' {checkIndent=true;};
> COLON: ':' {Indent(),looking_for_indent=true};
> ID:  IDENT {
> (looking_for_indent)?aquireIndent():(checkIndent)?verifysameline_or_same_column=true
> }
> 
> See 3) above
> {lexor.setIndentionMustBeEqual();}
> statement: statment ';' {lexor.setCheckNextIndent(true);}
> block: ":" {lexor.aquireIndent();} compundStatement;
> 
> Of coarse what we have now in 2.7.5 works, ANTLRWorks will need to allow
> for a thing to be inserted between the lexor and parser.
> Maybe just allow for this in the grammar
> <parser>
> { option: insertPostLexor="IndentionSensitiveLexor"}
> <lexor>
> 
> Just Musing........................
> (sorry for the length....)
> Elden
> 
> 
> 
> Terence Parr wrote:
> 
> >
> > On Jul 11, 2005, at 2:40 PM, Rodrigo B. de Oliveira wrote:
> >
> >> On 7/11/05, Terence Parr <parrt at cs.usfca.edu> wrote:
> >>
> >> Boo also needs virtual indent/dedent tokens. Our current approach is
> >> to insert a IndentTokenStreamFilter that preprocesses white space
> >> tokens and generates indent/dedent virtual tokens as necessary. Will
> >> this approach still be supported in antlr 3.0?
> >
> >
> > Yes, as long as ANTLR's lexer sees imaginary indent/dedent tokens,
> > it's cool :)  We'll have to find a way to have this automatically
> > detected and added to the input stream.
> >
> > Ter
> > --
> > CS Professor & Grad Director, University of San Francisco
> > Creator, ANTLR Parser Generator, http://www.antlr.org
> > Cofounder, http://www.jguru.com
> >
> >
> >
> 
> 
>