[antlr-interest] Re : How to get a list of all valid options for the next token?

Wed Aug 13 05:48:52 PDT 2008

Well, so much for that idea then :(

Guess the only way to do this then is to indeed duplicate the knowledge inherent in the parser to a different checker. Oh well.

From: Stanislas Rusinsky [mailto:rusinskystanislas at yahoo.fr] 
Sent: woensdag 13 augustus 2008 14:42
To: Niemeijer, R.A.; Foust
Cc: antlr-interest at antlr.org
Subject: Re : [antlr-interest] How to get a list of all valid options for the next token?

I have found one series of 3 articles that might be interesting for such situation: 

http://www.ibm.com/developerworks/views/opensource/libraryview.jsp?search_by=Create+commercial-quality+eclipse+ide

But I don't know if you could achieve that easily (if someone knows a way or tool I would be interested too in knowing it). ANTLR parsers generated from grammars are not 'self-aware' if I understood correctly: they are LR parsers but don't hold as such a graph based on the grammar that could determine the next allowed token.

Yours sincerely,

Stanislas Herman.

----- Message d'origine ----
De : "Niemeijer, R.A." <r.a.niemeijer at tue.nl>
À : Foust <javafoust at gmail.com>
Cc : antlr-interest at antlr.org
Envoyé le : Mercredi, 13 Août 2008, 13h02mn 41s
Objet : Re: [antlr-interest] How to get a list of all valid options for the next token?

No problem, thanks anyway.

Does anyone else have any idea how to do this?

Thanks in advance.

From: Foust [mailto:javafoust at gmail.com] 
Sent: woensdag 13 augustus 2008 12:44
To: Niemeijer, R.A.
Subject: RE: [antlr-interest] How to get a list of all valid options for the next token?

> Yes, it’s certainly possible to make my own check. However, that means I’m basically duplicating 

> my grammar, which makes it more difficult to maintain. The reason I wanted to get the information 

> from Antlr is that it should already know all the possible token combinations.

OK, well it sounds like you know what you’re doing. 

Sorry I wasn’t able to help more,

Brent

From: Niemeijer, R.A. [mailto:r.a.niemeijer at tue.nl] 
Sent: Wednesday, August 13, 2008 3:06 AM
To: Foust
Cc: 
Subject: RE: [antlr-interest] How to get a list of all valid options for the next token?

Yes, it’s certainly possible to make my own check. However, that means I’m basically duplicating my grammar, which makes it more difficult to maintain. The reason I wanted to get the information from Antlr is that it should already know all the possible token combinations.

From: Foust [mailto:javafoust at gmail.com] 
Sent: woensdag 13 augustus 2008 11:58
To: Niemeijer, R.A.
Subject: RE: [antlr-interest] How to get a list of all valid options for the next token?

Yes, I didn’t understand exactly what you’re trying to do (and probably still don’t). If you’re thinking of it like an Intellisense problem, then there are other ways of solving that kind of problem without using a parser-generator like Antlr. Have you already considered those? 

It sounds like your problem is a bit different than the code editor’s – simpler, actually. Whatever rules you use to decide what pieces can go where may be more easily expressed with a custom solution.  You could use Antlr to parse any text input, but call custom code to make sense of it and decide what the set of following pieces is. This would not only be more efficient, but would be far less frustrating to develop, since you have full control over the decision mechanism and can customize the underlying data structures to fit the task. Depending upon how simple your sequencing rules are, you might even be able to use a regular expression solution, if you can choose the text representation of the pieces.

Brent   

From: Niemeijer, R.A. [mailto:r.a.niemeijer at tue.nl] 
Sent: Wednesday, August 13, 2008 12:13 AM
To: Foust
Cc: antlr-interest at antlr.org
Subject: RE: [antlr-interest] How to get a list of all valid options for the next token?

Hello Brent,

It seems my question wasn’t clear enough. I have no intention of having a dynamic grammar. What I meant by disabling puzzle pieces is this: on the left side of the screen there is a list of all the possible puzzle pieces, a library if you will. From there the user drags pieces to the right side of the screen to form sequences of puzzle pieces. What I want to do is disable the pieces in the library. An alternative way of looking at it is seeing it as Intellisense/code completion. While the user is typing (dragging pieces), I want to display a popup list of all the possible words (pieces) that can come at the current position in the sentence. The grammar itself is completely fixed, just like the grammar of programming languages that allow code completion.

From: Foust [mailto:javafoust at gmail.com] 
Sent: dinsdag 12 augustus 2008 22:14
To: Niemeijer, R.A.
Subject: RE: [antlr-interest] How to get a list of all valid options for the next token?

> I want to disable all the pieces that cannot come at that point in the sentence.

> I want to disable all the pieces that cannot come at that point in the sentence.

My guess is that this will be pretty difficult to implement using even an augmented context-free grammar. Usually, you would parse any valid sentence (a list of puzzle pieces, in your case), then do the semantic analysis as a later pass (iterate over the actual puzzle pieces entered and check each one for validity). 

Multiple passes should make the code easier to write (as well as maintain). And error reporting should be more intuitive. It’s pretty difficult to enforce all semantic meaning at the parse phase and still output meaningful error messages.

If you really want to experiment with changing the grammar dynamically, though, you could generate an AST in your parser for any valid sentence, and then use semantic predicates in your tree grammar to turn on and off certain rules. But I think you’ll find that just adding validity checks in your tree grammar actions will be the most straightforward, providing all the advantages of a multi-pass interpreter.

Brent

From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-bounces at antlr.org] On Behalf Of Niemeijer, R.A.
Sent: Tuesday, August 12, 2008 2:57 AM
To: antlr-interest at antlr.org
Subject: [antlr-interest] How to get a list of all valid options for the next token?

Hello everyone,

I’m making a program where users construct short sentences by combining puzzle pieces. This series of puzzle pieces is then converted to text and parsed by Antlr. Currently the user has access to all the different pieces at any point in the sentence, even if the grammar would not be able to parse this. I want to disable all the pieces that cannot come at that point in the sentence.

My idea was to do this with the help of the exceptions the parser generates. For example, when it throws a MismatchedTokenException I can find the expected token through the Expecting property of the exception. Unfortunately this only works when there’s only one possibility. When there are multiple it instead throws a NoViableAltException or a MismatchedSetException. I haven’t been able to find a list of valid following tokens in their properties.

Can anyone tell me if there is any way of determining the list of valid tokens after the current token in the sentence?

Thanks in advance.

________________________________

Envoyé avec Yahoo! Mail <http://us.rd.yahoo.com/mailuk/taglines/isp/control/*http:/us.rd.yahoo.com/evt=52423/*http:/fr.docs.yahoo.com/mail/overview/index.html> .
Une boite mail plus intelligente. 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20080813/b8e0ae32/attachment-0001.html 