[antlr-interest] How can I parse length defined tokens?

[Dave Dutcher]

You have to remember that the lexer "automatically" chooses which token to
parse next.  It doesn't know anything about your block_1 grammar rule, so
when it sees 10 digits in a row, it won't know to lex a session number
before a sequence number.  I believe if the lexer has two rules with the
same left prefix it will try to lex the one which matches the most
characters.
(http://www.antlr.org/wiki/pages/viewpage.action?pageId=4882470)
 
I think your best option is to make application_id, service_id, address,
session_number, and sequence number grammar rules.
 
Dave
 
 



  _____  

From: antlr-interest-bounces at antlr.org
[mailto:antlr-interest-bounces at antlr.org] On Behalf Of
loic.lefevre at bnpparibas.com
Sent: Wednesday, December 02, 2009 11:15 AM
To: antlr-interest at antlr.org
Subject: [antlr-interest] How can I parse length defined tokens?



Hello, 
I'm relatively new to antlr v3. 

I'm trying to write a parser that should be able to parse the following
(SWIFT block 1) input: 

{1:F01PARBDEFFXXXX0001000000} 


I wish the parser to be able to then work with the following rule: 

block_1: '{1:' 
         APPLICATION_ID 
         SERVICE_ID 
         ADDRESS 
         SESSION_NUMBER 
           SEQUENCE_NUMBER 
         '}' 
       ; 

with tokens: 

APPLICATION_ID: 'F' | 'A' | 'L'; 

SERVICE_ID: '01' | '02'; 

fragment CHAR: 'A' .. 'Z' | '0' .. '9'; 

ADDRESS: CHAR CHAR CHAR CHAR CHAR CHAR CHAR CHAR CHAR CHAR CHAR CHAR; 

fragment DIGIT: '0' .. '1'; 

SESSION_NUMBER: DIGIT DIGIT DIGIT DIGIT; 

SEQUENCE_NUMBER: DIGIT DIGIT DIGIT DIGIT DIGIT DIGIT; 

However it doesn't work. Do you know an elegant way to work with this kind
of 
tokens? I mean to define length defined tokens and to be able to work
without 
"token recognition overlapp". 

(I'm open to combined grammar or not solution) 

Could the "filter" Lexer option help me there? 


Thanks in advance. 

Regards, 
Loïc






This message and any attachments (the "message") is

intended solely for the addressees and is confidential. 

If you receive this message in error, please delete it and 

immediately notify the sender. Any use not in accord with 

its purpose, any dissemination or disclosure, either whole 

or partial, is prohibited except formal approval. The internet

can not guarantee the integrity of this message. 

BNP PARIBAS (and its subsidiaries) shall (will) not 

therefore be liable for the message if modified. 

Do not print this message unless it is necessary,

consider the environment.



                ---------------------------------------------



Ce message et toutes les pieces jointes (ci-apres le 

"message") sont etablis a l'intention exclusive de ses 

destinataires et sont confidentiels. Si vous recevez ce 

message par erreur, merci de le detruire et d'en avertir 

immediatement l'expediteur. Toute utilisation de ce 

message non conforme a sa destination, toute diffusion 

ou toute publication, totale ou partielle, est interdite, sauf 

autorisation expresse. L'internet ne permettant pas 

d'assurer l'integrite de ce message, BNP PARIBAS (et ses

filiales) decline(nt) toute responsabilite au titre de ce 

message, dans l'hypothese ou il aurait ete modifie.

N'imprimez ce message que si necessaire,

pensez a l'environnement.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20091202/43f1b222/attachment.html