[antlr-interest] Re : Re : How to get a list of all validoptions for the next token?

Niemeijer, R.A. r.a.niemeijer at tue.nl
Fri Aug 22 03:50:56 PDT 2008


I'm afraid that unless somebody else offers to help me I'm a bit stuck at the moment. As Kay Röpke said: "right now i don't see any way of getting at the necessary information, without altering the codegen templates.". By now I've more or less learned how to work with Antlr, but I'm afraid hacking the way Antlr itself works is still a bit beyond me. So if anyone wishes to implement this I'd be very grateful, but otherwise I fear I will have to abandon the idea.

-----Original Message-----
From: Edwards, Waverly [mailto:Waverly.Edwards at genesys.com] 
Sent: donderdag 21 augustus 2008 23:53
To: Niemeijer, R.A.
Subject: RE: [antlr-interest] Re : Re : How to get a list of all validoptions for the next token?


Mr. Niemeijer,

If you come up with a solution would you mind sharing it.  I'm am quite curious to see
a model of this.  In looking at the follow sets source it seems the follow sets don't
show all of the possibilities but rather the ones that are most likely to occur, a local
follow set versus a global set.  I played with extracting this information about a month
ago with very limited success.  It involved the manual extraction of the bitset from the
code source which at the time was acceptable, since my choices were static, not dynamic
like yours.  I would much rather have tested something that didn't involve copying bitsets
if I made coding changes.


I would be interested in what you come up with.


Thank you very much for your time,


W.

Waverly Edwards

-----Original Message-----
From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-bounces at antlr.org] On Behalf Of Niemeijer, R.A.
Sent: Friday, August 15, 2008 5:29 AM
To: Stanislas Rusinsky; Thomas Brandon
Cc: antlr-interest at antlr.org
Subject: Re: [antlr-interest] Re : Re : How to get a list of all validoptions for the next token?

Right, I think I've localized all the data I need. All you need is a bit of reflection. All the bitsets that determine the possible next tokens are called FOLLOW_{0}_in_{1}{2}, where {0} is the previous token (can be gotten from the Parser's state.following variable), {1} is the name of the method the NoViableAltException is generated in (use something like MethodBase.GetCurrentMethod().Name) and {2} is un unimportant 3-digit number. So get the list of all Bitsets of the Parser via reflection and choose the one that starts with FOLLOW_{0}_in_{1}.

However, in order to be able to use this I have to insert this code at the point the NoViableAltException is thrown so I can dump the list of tokens in its grammarDecisionDescription field. That means overriding all the bits of code that look like this:

NoViableAltException nvae_d1s2 =
    new NoViableAltException("", 1, 2, input);

    throw nvae_d1s2;

Now obviously this can be done manually, but then you'd lose it after every grammar generation. So my question: is there a way to override this part of the generated grammar?

Thanks.

-----Original Message-----
From: Niemeijer, R.A. 
Sent: donderdag 14 augustus 2008 17:39
To: 'Stanislas Rusinsky'; Thomas Brandon
Cc: antlr-interest at antlr.org
Subject: RE: Re : [antlr-interest] Re : How to get a list of all valid options for the next token?

The FOLLOW_ variables do indeed seem to contain all the data I need. I'm going to have a look at it tomorrow to see if I can convert to bitset data to something more useful to me.

-----Original Message-----
From: Stanislas Rusinsky [mailto:rusinskystanislas at yahoo.fr] 
Sent: woensdag 13 augustus 2008 19:27
To: Thomas Brandon; Niemeijer, R.A.
Cc: antlr-interest at antlr.org
Subject: Re : [antlr-interest] Re : How to get a list of all valid options for the next token?

oops, when making such attention mistakes I feel like hiding under the table :) they are LL of course

Mr. Niemeijer, could you please keep me informed if you reach a satisfactory solution with the "FOLLOW_" hint? I am interested too in that topic.

Stanislas.



----- Message d'origine ----
De : Thomas Brandon <tbrandonau at gmail.com>
À : "Niemeijer, R.A." <r.a.niemeijer at tue.nl>
Cc : antlr-interest at antlr.org
Envoyé le : Mercredi, 13 Août 2008, 16h32mn 28s
Objet : Re: [antlr-interest] Re : How to get a list of all valid options for the next token?

ANTLR grammars are LL not LR, but they are recursive descent and thus
don't use an explicit internal representation of the grammar to
control parse decisions. They do however have follow sets available at
parse time, used for error recovery and reporting. See the various
FOLLOW_ bitsets in generated parsers and the computeErrorRecoverySet
and computeContextSensitiveRuleFOLLOW methods in BaseRecognizer. You
may be able to get what you need from them. Though as they only
support a limited purpose they may collapse alternatives you need to
keep distinct.
Another option would be to create a custom generator that would then
have the richer information ANTLR uses at generation time.

Tom.

On Wed, Aug 13, 2008 at 10:48 PM, Niemeijer, R.A. <r.a.niemeijer at tue.nl> wrote:
> Well, so much for that idea then :(
>
> Guess the only way to do this then is to indeed duplicate the knowledge
> inherent in the parser to a different checker. Oh well.
>
>
>
> From: Stanislas Rusinsky [mailto:rusinskystanislas at yahoo.fr]
> Sent: woensdag 13 augustus 2008 14:42
> To: Niemeijer, R.A.; Foust
> Cc: antlr-interest at antlr.org
> Subject: Re : [antlr-interest] How to get a list of all valid options for
> the next token?
>
>
>
> I have found one series of 3 articles that might be interesting for such
> situation:
>
> http://www.ibm.com/developerworks/views/opensource/libraryview.jsp?search_by=Create+commercial-quality+eclipse+ide
>
> But I don't know if you could achieve that easily (if someone knows a way or
> tool I would be interested too in knowing it). ANTLR parsers generated from
> grammars are not 'self-aware' if I understood correctly: they are LR parsers
> but don't hold as such a graph based on the grammar that could determine the
> next allowed token.
>
> Yours sincerely,
>
> Stanislas Herman.
>



      _____________________________________________________________________________ 
Envoyez avec Yahoo! Mail. Une boite mail plus intelligente http://mail.yahoo.fr


More information about the antlr-interest mailing list