[antlr-interest] Number tokenizer vs. number grammar

[Sam Harwell]

The problem you'll encounter is order of operations. If the following is
valid in your language:

2 * 3+2i

Then is the result supposed to be 6+2i or 6+4i? What about 2*3+2i?

How the language treats order of operations will determine your possible
approaches.

Sam

-----Original Message-----
From: antlr-interest-bounces at antlr.org
[mailto:antlr-interest-bounces at antlr.org] On Behalf Of Todd O'Bryan
Sent: Saturday, November 15, 2008 10:36 AM
To: antlr-interest at antlr.org
Subject: [antlr-interest] Number tokenizer vs. number grammar

 (One warning--I'm new to ANTLR so I haven't wrapped my brain around
syntactic/semantic predicates and may have missed something obvious
there that would make this trivial.)

I'm parsing a language that allows a great range of number constructs.
1, -3, 3.14, -5.8, 2/3, -3/-5, 3+2i, -4+i, 5-i, -3/5+2/3i, 3/5-2/3i,
7+2.3i and some other things are all legal numbers.

For the language grammar itself, it would be very nice to just have a
NUMBER token, but it would also be nice to have the ability to parse
numbers into component parts since creating a complex number involves
calling a constructor with the two real parts. The other tricky thing
is that whitespace is irrelevant in the language grammar, but relevant
in parsing numbers. For example, 3+ 2i (with a space before the 2) is
not a legal number.

I've thought about identifying a NUMBER token in the grammar and then
calling a number parser on that token as part of processing, but what
I'm afraid of is keeping the number parser and the NUMBER token in
sync. What I'm feeling like is that I need token fragments in the
language grammar that aren't fragments but are actual tokens in the
number grammar.

Am I missing a simple way to deal with this or is this just a nasty
problem?

Todd

List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-addr
ess