[antlr-interest] ANTLR3ea+ANTLWorks is *really cool*, but how do I insert PythonTokenStream.java

Prashant Deva prashant.deva at gmail.com
Sat Jul 23 13:32:24 PDT 2005


Oops, Please ignore my last post. I thought what u said was related to
the ide and not antlr itself.

PRASHANT

On 7/24/05, Prashant Deva <prashant.deva at gmail.com> wrote:
> Elden,
>  I cant understand your problem. Could you explain it a little more,
> so that i can provide a nice solution in Antlr Studio?
> 
> 
>  PRASHANT
> 
> On 7/24/05, Elden Crom <eldencrom at comcast.net> wrote:
> > Same here for now (putting a class between lexor and parser, works
> > fine), but with respect to your (Terence's) quote
> > "Why program by hand in five days what you can spend five years of your
> > life automating."
> > are we out of time yet?  :-)
> > perhaps it could be added to ANTLR 3's grammar some how....
> > ======================================
> > ======================================
> > <example that covers most of my cases...(through it wouldn't pass coding
> > standards!) (make sure its in a non-proportional font to read..).>
> >   ...
> >   for x (0,2,12 //no DENT
> >  34,43):
> >     y +=x*2;if(x>12):p+=y; i++;  //INDENT before 'p' but not before 'i'
> >                      p/=z;
> >       /*this too?*/  p/=2;  //all of the p assignments are in the
> > if(...) block
> >                              //comments don't gen dent errors or cause
> > indention to be aquired
> >                        // the  /*..*/ comment has no effect but the
> > spacing is preserved
> >     q*=p;
> >   while(next()):; //gen INDENT ";" and DEDENT
> >   func(SomeObnoxiouslyLongIdentifier,   //no DENT here
> >        AnotherObnoxiouslyLongIdentifier);
> >    func2();  //error -- alignment wrong
> >    if(b):
> >    c=1;   //error -- indent of 'c' must be greater than 'if'
> >
> > ======================================
> > ======================================
> >
> > The first question is does it belong in the parser or the lexor? (it
> > appears to usually sit in between, as a matter of necessity now)
> > I know that for my grammar I'm designing, I only want to generate an
> > error if the beginning of a statement is not aligned with the previous
> > and it is not preceded with a ":".  Same for Python.  My tendency is to
> > think it (control) belongs in the parser.
> > Some options (certainly not exhaustive):
> > 1) Have a grammar flag that says always generates a INDENT and DEDENT
> > token after 'newline()' has been called.
> >      CON: every statement that can contains a DENT token (arrays, list,
> > between if and "(" etc) must have "(DENT)? {ignoreDent()}" all over it.
> >      PRO: maybe not so bad just make a rule "ig_dent: (INDENT | DEDENT)?
> > {ignoreDent()}" ... still ugly
> > 2) Have a lexor functions that say look for new dent after a certain
> > point (":" in my case) look for new indention
> >      CON: Somewhat restricts how it can be used.
> >      PRO: Relatively easy to describe in the lexor and does not put
> > allot of jiberish in the parser chunk.
> > 3) have the Parser mess with the Lexor's head, by saying where changes
> > of indent are allowed
> >      CON: parser and lexor are no-long stand alone (colorization for
> > grammars becomes more difficult, etc.)
> >      PRO: most flexible
> >
> > (the grammar for my language.....)
> > See 1) above -- in the parser
> > statement: statment ';' {checkNextIndent=true;}
> > block: ":" {aquireIndent();} compundStatement;
> > if: "if" ig_dent expersion ig_dent block
> > while: "while" ig_dent expersion ig_dent block
> > ....
> >
> > see 2) above -- in the lexor
> > SEMI: ';' {checkIndent=true;};
> > COLON: ':' {Indent(),looking_for_indent=true};
> > ID:  IDENT {
> > (looking_for_indent)?aquireIndent():(checkIndent)?verifysameline_or_same_column=true
> > }
> >
> > See 3) above
> > {lexor.setIndentionMustBeEqual();}
> > statement: statment ';' {lexor.setCheckNextIndent(true);}
> > block: ":" {lexor.aquireIndent();} compundStatement;
> >
> > Of coarse what we have now in 2.7.5 works, ANTLRWorks will need to allow
> > for a thing to be inserted between the lexor and parser.
> > Maybe just allow for this in the grammar
> > <parser>
> > { option: insertPostLexor="IndentionSensitiveLexor"}
> > <lexor>
> >
> > Just Musing........................
> > (sorry for the length....)
> > Elden
> >
> >
> >
> > Terence Parr wrote:
> >
> > >
> > > On Jul 11, 2005, at 2:40 PM, Rodrigo B. de Oliveira wrote:
> > >
> > >> On 7/11/05, Terence Parr <parrt at cs.usfca.edu> wrote:
> > >>
> > >> Boo also needs virtual indent/dedent tokens. Our current approach is
> > >> to insert a IndentTokenStreamFilter that preprocesses white space
> > >> tokens and generates indent/dedent virtual tokens as necessary. Will
> > >> this approach still be supported in antlr 3.0?
> > >
> > >
> > > Yes, as long as ANTLR's lexer sees imaginary indent/dedent tokens,
> > > it's cool :)  We'll have to find a way to have this automatically
> > > detected and added to the input stream.
> > >
> > > Ter
> > > --
> > > CS Professor & Grad Director, University of San Francisco
> > > Creator, ANTLR Parser Generator, http://www.antlr.org
> > > Cofounder, http://www.jguru.com
> > >
> > >
> > >
> >
> >
> >
>


More information about the antlr-interest mailing list