[antlr-interest] ANTLR3ea+ANTLWorks is *really cool*,
but how do I insert PythonTokenStream.java
Prashant Deva
prashant.deva at gmail.com
Sat Jul 23 13:32:24 PDT 2005
Oops, Please ignore my last post. I thought what u said was related to
the ide and not antlr itself.
PRASHANT
On 7/24/05, Prashant Deva <prashant.deva at gmail.com> wrote:
> Elden,
> I cant understand your problem. Could you explain it a little more,
> so that i can provide a nice solution in Antlr Studio?
>
>
> PRASHANT
>
> On 7/24/05, Elden Crom <eldencrom at comcast.net> wrote:
> > Same here for now (putting a class between lexor and parser, works
> > fine), but with respect to your (Terence's) quote
> > "Why program by hand in five days what you can spend five years of your
> > life automating."
> > are we out of time yet? :-)
> > perhaps it could be added to ANTLR 3's grammar some how....
> > ======================================
> > ======================================
> > <example that covers most of my cases...(through it wouldn't pass coding
> > standards!) (make sure its in a non-proportional font to read..).>
> > ...
> > for x (0,2,12 //no DENT
> > 34,43):
> > y +=x*2;if(x>12):p+=y; i++; //INDENT before 'p' but not before 'i'
> > p/=z;
> > /*this too?*/ p/=2; //all of the p assignments are in the
> > if(...) block
> > //comments don't gen dent errors or cause
> > indention to be aquired
> > // the /*..*/ comment has no effect but the
> > spacing is preserved
> > q*=p;
> > while(next()):; //gen INDENT ";" and DEDENT
> > func(SomeObnoxiouslyLongIdentifier, //no DENT here
> > AnotherObnoxiouslyLongIdentifier);
> > func2(); //error -- alignment wrong
> > if(b):
> > c=1; //error -- indent of 'c' must be greater than 'if'
> >
> > ======================================
> > ======================================
> >
> > The first question is does it belong in the parser or the lexor? (it
> > appears to usually sit in between, as a matter of necessity now)
> > I know that for my grammar I'm designing, I only want to generate an
> > error if the beginning of a statement is not aligned with the previous
> > and it is not preceded with a ":". Same for Python. My tendency is to
> > think it (control) belongs in the parser.
> > Some options (certainly not exhaustive):
> > 1) Have a grammar flag that says always generates a INDENT and DEDENT
> > token after 'newline()' has been called.
> > CON: every statement that can contains a DENT token (arrays, list,
> > between if and "(" etc) must have "(DENT)? {ignoreDent()}" all over it.
> > PRO: maybe not so bad just make a rule "ig_dent: (INDENT | DEDENT)?
> > {ignoreDent()}" ... still ugly
> > 2) Have a lexor functions that say look for new dent after a certain
> > point (":" in my case) look for new indention
> > CON: Somewhat restricts how it can be used.
> > PRO: Relatively easy to describe in the lexor and does not put
> > allot of jiberish in the parser chunk.
> > 3) have the Parser mess with the Lexor's head, by saying where changes
> > of indent are allowed
> > CON: parser and lexor are no-long stand alone (colorization for
> > grammars becomes more difficult, etc.)
> > PRO: most flexible
> >
> > (the grammar for my language.....)
> > See 1) above -- in the parser
> > statement: statment ';' {checkNextIndent=true;}
> > block: ":" {aquireIndent();} compundStatement;
> > if: "if" ig_dent expersion ig_dent block
> > while: "while" ig_dent expersion ig_dent block
> > ....
> >
> > see 2) above -- in the lexor
> > SEMI: ';' {checkIndent=true;};
> > COLON: ':' {Indent(),looking_for_indent=true};
> > ID: IDENT {
> > (looking_for_indent)?aquireIndent():(checkIndent)?verifysameline_or_same_column=true
> > }
> >
> > See 3) above
> > {lexor.setIndentionMustBeEqual();}
> > statement: statment ';' {lexor.setCheckNextIndent(true);}
> > block: ":" {lexor.aquireIndent();} compundStatement;
> >
> > Of coarse what we have now in 2.7.5 works, ANTLRWorks will need to allow
> > for a thing to be inserted between the lexor and parser.
> > Maybe just allow for this in the grammar
> > <parser>
> > { option: insertPostLexor="IndentionSensitiveLexor"}
> > <lexor>
> >
> > Just Musing........................
> > (sorry for the length....)
> > Elden
> >
> >
> >
> > Terence Parr wrote:
> >
> > >
> > > On Jul 11, 2005, at 2:40 PM, Rodrigo B. de Oliveira wrote:
> > >
> > >> On 7/11/05, Terence Parr <parrt at cs.usfca.edu> wrote:
> > >>
> > >> Boo also needs virtual indent/dedent tokens. Our current approach is
> > >> to insert a IndentTokenStreamFilter that preprocesses white space
> > >> tokens and generates indent/dedent virtual tokens as necessary. Will
> > >> this approach still be supported in antlr 3.0?
> > >
> > >
> > > Yes, as long as ANTLR's lexer sees imaginary indent/dedent tokens,
> > > it's cool :) We'll have to find a way to have this automatically
> > > detected and added to the input stream.
> > >
> > > Ter
> > > --
> > > CS Professor & Grad Director, University of San Francisco
> > > Creator, ANTLR Parser Generator, http://www.antlr.org
> > > Cofounder, http://www.jguru.com
> > >
> > >
> > >
> >
> >
> >
>
More information about the antlr-interest
mailing list