[antlr-interest] Another update to Honey Badger

Terence Parr parrt at cs.usfca.edu
Tue Jan 24 13:00:07 PST 2012


"more" is used when you want the lexer is merge the current token's text with the next token. It basically says to continue without throwing out the current text. Useful for mode stuff.
T
On Jan 24, 2012, at 10:03 AM, Peter Boughton wrote:

> Is there a definition of what the "more" action is for?
> 
>> From experimenting, it seems to merge the marked token with the next
> token, but only if it is placed AFTER any "type" actions - placed
> before it is just ignored. However, the merged token always takes the
> type of the final token (without the more action), not the original
> one.
> 
> 
> What I personally would like "more" to do is to only merge tokens of
> the same type (or perhaps to specify a whitelist of token types which
> can be merged).
> 
> This would mean I can have a simplified catch-all rule that produces a
> single token, like so:
> 
> EVERYTHING_ELSE : . -> more(EVERYTHING_ELSE);
> 
> At the moment, if I do a more on that rule I end up with tokens like
> "abc  " because the more doesn't stop until it reaches a rule without
> more - so I get (in this example) appended whitespace and the wrong
> type.
> 
> 
> However, I don't know if that is the intended use of "more" or if
> there's a different way to achieve what I'm after?
> 
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address



More information about the antlr-interest mailing list