[antlr-interest] Whitespace: More than meets the eye?

Loring Craymer lgcraymer at yahoo.com
Wed Aug 5 17:41:11 PDT 2009


ANTLR makes no assumptions about whitespace; ANTLR lexers just slice up a stream (array) of characters into tokens according to a lexer grammar.  For most programming language grammars, whitespace can be ignored (and lexer grammars tend to put whitespace tokens on a "hidden" channel) since it basically serves to separate meaningful tokens.

If you want to require or forbid whitespace in parser rules, you either have to not ignore whitespace or insert semantic predicates to look for whitespace or its absence.  This gets messy fast, so modern languages avoid the old FORTRAN "interesting" lexical behavior, and you will not find much expertise in dealing with really outre lexer problems here--it is unusual to find a need to solve such problems.

--Loring



----- Original Message ----
> From: Graham Wideman <gwlist at grahamwideman.com>
> To: antlr-interest at antlr.org
> Sent: Wednesday, August 5, 2009 5:10:18 PM
> Subject: [antlr-interest] Whitespace: More than meets the eye?
> 
> Hi folks:
> 
> Could someone clue me in to the zen of whitespace (or where this is discussed in 
> The Books or online)?  By that I mean as follows: 
> 
> I see that grammars generally have a lexer rule that specifies the pattern for 
> whitespace, but what's the correspondence between that and what ANTLR does with 
> lexer rules, where the WHITESPACE token is generally not used.
> 
> I'm assuming that ANTLR makes assumptions that whitespace either may or must 
> appear between tokens that appear in lexer rules, right?  Either that or it's 
> picking up something from lexer rules that's subtler than I've noticed.
> 
> To pick a simple example, suppose you're defining what a function call should 
> look like:
> 
> functionCall
>     : Identifier LParen args RParen -> ....
> 
> ... what variations does one need in order to assert that whitespace is, or is 
> not, permitted or required between Identifier and LParen?
> 
> Thanks!
> 
> Graham
> 
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: 
> http://www.antlr.org/mailman/options/antlr-interest/your-email-address



      


More information about the antlr-interest mailing list