[antlr-interest] Re : Re : How to get a list of all valid options for the next token?

[Johannes Luber]

Niemeijer, R.A. schrieb:
> Right, I think I've localized all the data I need. All you need is a bit of reflection. All the bitsets that determine the possible next tokens are called FOLLOW_{0}_in_{1}{2}, where {0} is the previous token (can be gotten from the Parser's state.following variable), {1} is the name of the method the NoViableAltException is generated in (use something like MethodBase.GetCurrentMethod().Name) and {2} is un unimportant 3-digit number. So get the list of all Bitsets of the Parser via reflection and choose the one that starts with FOLLOW_{0}_in_{1}.
> 
> However, in order to be able to use this I have to insert this code at the point the NoViableAltException is thrown so I can dump the list of tokens in its grammarDecisionDescription field. That means overriding all the bits of code that look like this:
> 
> NoViableAltException nvae_d1s2 =
>     new NoViableAltException("", 1, 2, input);
> 
>     throw nvae_d1s2;
> 
> Now obviously this can be done manually, but then you'd lose it after every grammar generation. So my question: is there a way to override this part of the generated grammar?

The only way I know is to modify the generation templates, but you have 
then to adapt them after each ANTLR upgrade. Maybe Ter is willing to add 
another overwritable region, which can be referenced in a user template. 
This would require a change in the tool itself, so it does look for such 
a template file.

Johannes
> 
> Thanks.
> 
> -----Original Message-----
> From: Niemeijer, R.A. 
> Sent: donderdag 14 augustus 2008 17:39
> To: 'Stanislas Rusinsky'; Thomas Brandon
> Cc: antlr-interest at antlr.org
> Subject: RE: Re : [antlr-interest] Re : How to get a list of all valid options for the next token?
> 
> The FOLLOW_ variables do indeed seem to contain all the data I need. I'm going to have a look at it tomorrow to see if I can convert to bitset data to something more useful to me.
> 
> -----Original Message-----
> From: Stanislas Rusinsky [mailto:rusinskystanislas at yahoo.fr] 
> Sent: woensdag 13 augustus 2008 19:27
> To: Thomas Brandon; Niemeijer, R.A.
> Cc: antlr-interest at antlr.org
> Subject: Re : [antlr-interest] Re : How to get a list of all valid options for the next token?
> 
> oops, when making such attention mistakes I feel like hiding under the table :) they are LL of course
> 
> Mr. Niemeijer, could you please keep me informed if you reach a satisfactory solution with the "FOLLOW_" hint? I am interested too in that topic.
> 
> Stanislas.
> 
> 
> 
> ----- Message d'origine ----
> De : Thomas Brandon <tbrandonau at gmail.com>
> À : "Niemeijer, R.A." <r.a.niemeijer at tue.nl>
> Cc : antlr-interest at antlr.org
> Envoyé le : Mercredi, 13 Août 2008, 16h32mn 28s
> Objet : Re: [antlr-interest] Re : How to get a list of all valid options for the next token?
> 
> ANTLR grammars are LL not LR, but they are recursive descent and thus
> don't use an explicit internal representation of the grammar to
> control parse decisions. They do however have follow sets available at
> parse time, used for error recovery and reporting. See the various
> FOLLOW_ bitsets in generated parsers and the computeErrorRecoverySet
> and computeContextSensitiveRuleFOLLOW methods in BaseRecognizer. You
> may be able to get what you need from them. Though as they only
> support a limited purpose they may collapse alternatives you need to
> keep distinct.
> Another option would be to create a custom generator that would then
> have the richer information ANTLR uses at generation time.
> 
> Tom.
> 
> On Wed, Aug 13, 2008 at 10:48 PM, Niemeijer, R.A. <r.a.niemeijer at tue.nl> wrote:
>> Well, so much for that idea then :(
>>
>> Guess the only way to do this then is to indeed duplicate the knowledge
>> inherent in the parser to a different checker. Oh well.
>>
>>
>>
>> From: Stanislas Rusinsky [mailto:rusinskystanislas at yahoo.fr]
>> Sent: woensdag 13 augustus 2008 14:42
>> To: Niemeijer, R.A.; Foust
>> Cc: antlr-interest at antlr.org
>> Subject: Re : [antlr-interest] How to get a list of all valid options for
>> the next token?
>>
>>
>>
>> I have found one series of 3 articles that might be interesting for such
>> situation:
>>
>> http://www.ibm.com/developerworks/views/opensource/libraryview.jsp?search_by=Create+commercial-quality+eclipse+ide
>>
>> But I don't know if you could achieve that easily (if someone knows a way or
>> tool I would be interested too in knowing it). ANTLR parsers generated from
>> grammars are not 'self-aware' if I understood correctly: they are LR parsers
>> but don't hold as such a graph based on the grammar that could determine the
>> next allowed token.
>>
>> Yours sincerely,
>>
>> Stanislas Herman.
>>
> 
> 
> 
>       _____________________________________________________________________________ 
> Envoyez avec Yahoo! Mail. Une boite mail plus intelligente http://mail.yahoo.fr
>