[antlr-interest] [CSharp3] rule visibility in composite grammars

Douglas Godfrey douglasgodfrey at gmail.com
Thu Jun 9 01:49:07 PDT 2011


Almost all complex languages require a SymbolTable to resolve semantics and
grammar ambiguities.
Some languages only need a global scope while many need a NameSpaces and
nested Scopes.
Antlr itself has a SymbolTable with NameSpaces and nested Scopes. My
proposal would be to
generalize the Antlr SymbolTable and make it available to the grammar and
the generated parsers.

A Generalized call mechanism would help for code generation and combined
with a built-in SymbolTable
could eliminate most target specific code in productions.

To completely eliminate the need for target specific productions Antlr would
need a target independent
way to manipulate the Antlr class objects [Tokens, Streams, AST nodes,
etc.], basic assignment
statements, expressions and I/O. The underlying code to do the these
functions already exists in each
target.

What would need to be added to the Antlr grammar would be the equivalent of
a Java or C++ interpreter.
There are many available interpreter grammars with the required BSD license
that could be integrated
with Antlr. The interpreter grammar itself would be built for each target
and the generated code would be
part of the target runtime. All of the added functionality would be in the
interpreter grammar.



On Thu, Jun 9, 2011 at 1:53 AM, Sam Harwell <sharwell at pixelminegames.com>wrote:

> I’m also very interested in ways to make ANTLR grammars
> target-language-agnostic. I know ANTLR version 4 will provide more
> consistent semantics across a number of language features, but I’m not sure
> if a specific symbol table support like you mention will be included. The
> problem with such a feature is it increases the complexity of the grammar
> language specification, increases the size of the runtime and/or the
> complexity of the code generation templates (making things more difficult
> for target developers), and only meets the semantic language features of a
> small number of users.
>
>
>
> One idea I had is for a target-agnostic “call” syntax that could be used in
> semantic predicates and actions. It could use a form such as the following:
>
>
>
> @{FunctionName(arg1,arg2,arg3)}
>
>
>
> Where an argument can be one of the following:
>
>
>
> ·         A reference to a token, rule return value, or label in the rule.
>
> ·         A reference to an argument passed to the rule.
>
> ·         A reference to a value in an attribute scope.
>
>
>
> The target would then declare a user-definable method with the appropriate
> parameter types. The Java and CSharp2 targets could create an abstract
> method, the CSharp3 target could create a partial method, and a C/C++
> targets could declare the method in a generated header.
>
>
>
> Sam
>
>
>
> *From:* Douglas Godfrey [mailto:douglasgodfrey at gmail.com]
> *Sent:* Tuesday, June 07, 2011 8:09 AM
> *To:* Ranco Marcus
> *Cc:* Sam Harwell; antlr-interest at antlr.org
> *Subject:* Re: [antlr-interest] [CSharp3] rule visibility in composite
> grammars
>
>
>
> 2 months ago I submitted a feature request for an Antlr built-in symbol
> table to support the common
> requirements of the majority of block-structured languages. By making the
> SymbolTable part of the
> Antlr grammar language the interface can be much cleaner. The
> implementation of the SymbolTable
> classes would be part of the target runtime(s).
>
> i.e.
>
> new_variable_name:
>     (Identifier.IsNewSymbol()) => Identifier.AddSymbol();
>
> SymbolTable -> NameSpace -> SymbolScope -> Symbol -> Attribute-List {
> optional for structs -> NameSpace }
>
> On Tue, Jun 7, 2011 at 5:56 AM, Ranco Marcus <ranco.marcus at epirion.nl>
> wrote:
>
> Yes, that sounds like a good idea. I would definitely be in favour of
> merging the grammars before generating the code. My only concern is that
> this approach would deviate from the general ANTLR approach.
>
> In general, I have found ANTLR to be a great tool for parser generation,
> but never really liked the way target specific actions are mixed with the
> grammar definition. Ideally, I would like my grammars to be _completely_
> target agnostic (no actions, no visibility modifiers, members, headers,
> superClass definitions, etc.) and have an abstract mechanism that we can use
> to hookup actions and implementation specific stuff to the generated
> grammar.
>
> Do you know if there are plans to redesign the composite grammar feature in
> v4?
>
> Best regards,
>
> Ranco
>
>
> > -----Original Message-----
> > From: Sam Harwell [mailto:sharwell at pixelminegames.com]
> > Sent: Sunday, May 29, 2011 11:08 AM
> > To: Ranco Marcus; antlr-interest at antlr.org
> > Subject: RE: [antlr-interest] [CSharp3] rule visibility in composite
> grammars
> >
> > I'm not going to be able to address this issue until the second week of
> June.
> >
> > That said, it seems the best way to handle all these issues with delegate
> > grammars is to inline their rules before code generation. Suppose you
> have
> > grammar C importing A and B, and you also have D importing A and B. The
> > code generation will result in classes C, C_A, C_B, D, D_A, and D_B.
> Clearly
> > the independent generation of C_A and D_A during code generation does
> > not allow a single instance of the imported A grammar to be shared by C
> and
> > D. If we instead "flatten" the imported grammar hierarchy and only
> generate
> > classes C and D, then everything behaves like it was written in a single
> > grammar. Do you see any immediate problems with this potential approach?
> >
> > -----Original Message-----
> > From: Ranco Marcus [mailto:ranco.marcus at epirion.nl]
> > Sent: Wednesday, May 25, 2011 4:26 PM
> > To: Sam Harwell; antlr-interest at antlr.org
> > Subject: RE: [antlr-interest] [CSharp3] rule visibility in composite
> grammars
> >
> > Hi Sam/all,
> >
> > When a (tree) grammar C imports (tree) grammars A and B, where grammar
> > A calls a rule R from grammar B, a call is being made from delegate
> parser C_A
> > to a delegate rule R (targeting C_B) in the composite parser C (its
> parent).
> >
> > Now that the visibility of the delegate rules in C match the visibility
> of the
> > imported grammar, the rule R has to be made public for the above to work.
> > In our grammars, we build up internal structures that are subsequently
> > processed. In our case, that means that all those internal structures
> have to
> > be made public as well. This could be solved by allowing ANTLR rules to
> have
> > 'internal' visibility. Also, imported grammars can probably remain
> internal as
> > well.
> >
> > What are your thoughts on this?
> >
> > Thanks, Ranco
> >
> > > -----Original Message-----
> > > From: Sam Harwell [mailto:sharwell at pixelminegames.com]
> > > Sent: Thursday, April 07, 2011 3:57 PM
> > > To: Ranco Marcus; antlr-interest at antlr.org
> > > Subject: RE: [antlr-interest] [CSharp3] rule visibility in composite
> > > grammars
> > >
> > > The visibility of delegated rules now matches the way they're declared
> > > in the imported grammar. Since some of them return nested classes
> > > declared in the delegate grammar, I had to make the delegate grammar
> > > 'public' and all its rules 'internal'.
> > >
> > > I fixed the reserved names issue right after releasing 3.3.1, so it'll
> > > be included when I release 3.3.2. Note that rule parameters and return
> > > values need to include an explicit '@' in their declarations in the
> > > grammar, as follows. Labels and rule names do not need the '@'.
> > >
> > > namespace[int @in] returns [int @out] : as=A B;
> > >
> > > Sam
> > >
> > > -----Original Message-----
> > > From: Ranco Marcus [mailto:ranco.marcus at epirion.nl]
> > > Sent: Thursday, April 07, 2011 6:49 AM
> > > To: Sam Harwell; antlr-interest at antlr.org
> > > Subject: RE: [antlr-interest] [CSharp3] rule visibility in composite
> > > grammars
> > >
> > > Sounds great, Sam!
> > >
> > > What will be the visibility of the methods in the 'Delegated rules'
> > > region now that the class of the imported grammar is internal? Private
> > > or
> > internal?
> > >
> > > Another small suggestion is to prefix variable names (at least those
> > > that are reserved in C#) with an at-sign. The generated class for the
> > > grammar below has a compile error since 'as' is not a valid variable
> > > name
> > while '@as' is.
> > >
> > > name : as=firstName lastName;
> > >
> > > Are you planning to release new binaries any time soon?
> > >
> > > Thanks again,
> > >
> > > Ranco Marcus
> > >
> > > > -----Original Message-----
> > > > From: Sam Harwell [mailto:sharwell at pixelminegames.com]
> > > > Sent: woensdag 6 april 2011 17:20
> > > > To: Ranco Marcus; antlr-interest at antlr.org
> > > > Subject: RE: [antlr-interest] [CSharp3] rule visibility in composite
> > > > grammars
> > > >
> > > > Hi Ranco,
> > > >
> > > > I made the following changes so far:
> > > >
> > > > * An imported grammar is always generated as an internal class
> > > > instead of a public class.
> > > > * Rules within the imported grammar are always declared public since
> > > > they are only called by the root grammar.
> > > >
> > > > How does that sound?
> > > >
> > > > Sam
> > > >
> > > > -----Original Message-----
> > > > From: antlr-interest-bounces at antlr.org
> > > > [mailto:antlr-interest-bounces at antlr.org] On Behalf Of Ranco Marcus
> > > > Sent: Wednesday, March 23, 2011 6:40 AM
> > > > To: antlr-interest at antlr.org
> > > > Subject: [antlr-interest] [CSharp3] rule visibility in composite
> > > > grammars
> > > >
> > > > Hi all/Sam,
> > > >
> > > > While using the CSharp3 target on a composite grammar, I came across
> > > > a few issues regarding rule visibility.
> > > >
> > > > If no modifier (private, protected, public, ...) is specified for a
> > > > rule, the Java target generates public methods (since it is not
> > > > implemented IIRC) and the
> > > > CSharp3 target generates private methods.
> > > >
> > > > I like the idea to explicitly specify which rules are callable (by
> > > > making them public). However, for composite grammars the generated
> > > > parser contains delegate rules for every rule of the imported
> > > > grammar, even for rules in the imported grammar that aren't public
> > > > (causing compile errors). This can solved this adding a public
> > > > modifier to every rule of the imported grammar, but I was wondering
> > > > if there's a more
> > > convenient way of doing this.
> > > >
> > > > Furthermore, the methods for the delegate rules in the composite
> > > > grammars are all public, regardless of the visibility of the rule in
> > > > the
> > > imported grammar.
> > > > IMHO, it would be better to make them private and use the composite
> > > > grammar to explicitly specify the rules that are externally visible.
> > > > What are your thoughts on this?
> > > >
> > > > Btw, the CSharp3 templates are also included in the original Java
> > > > tool. Does that mean that we don't need the .NET version of the tool
> > > anymore?
> > > >
> > > >
> > > > Best regards,
> > > >
> > > > Ranco Marcus
> > > > Epirion Knowledge Solutions B.V.
> > > >
> > > >
> > > > List: http://www.antlr.org/mailman/listinfo/antlr-interest
> > > > Unsubscribe:
> > > > http://www.antlr.org/mailman/options/antlr-interest/your-email-addre
> > > > ss
> > >
> >
>
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe:
> http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>
>
>


More information about the antlr-interest mailing list