[antlr-interest] recombining tokens

Davyd Madeley davyd at fugro-fsi.com.au
Mon Jul 28 01:03:15 PDT 2008


Hi all,

I'm currently writing a grammar in which '/' is used to append a
qualifier to a token. Unfortunately it is also used in path parameters.

I am trying to figure out how I can recombine tokens in the case where I
determine I'm reading a path.

e.g.

// these are my token delimiters
TOKEN
	: ~(','|'>'|'*'|'/'|'('|')'|LINE_TERMINATOR)+
	;

At one point in the state machine, I expect to be able to start reading
parameters ('LINE' is a special token at the start of the file, but
after that is just a regular token):

parameter
	: a=TOKEN	-> PARAMETER[$a]
	| a='LINE'	-> PARAMETER[$a]
	| path		-> ^(PATH path)
	;

path
	: ('/' TOKEN)+
	;

Every so often, a path will be provided. Currently this will be
tokenised around the '/', which is undesirable.

e.g.
      PATH (9) .......................... PATH
        '/' (20) ........................ /
        TOKEN (11) ...................... path
        '/' (20) ........................ /
        TOKEN (11) ...................... to
        '/' (20) ........................ /
        TOKEN (11) ...................... my.file

What I want to do is be able to recombine this into a
PARAMETER["/path/to/my.file"].

Someone spoke about a concatenation operator, but I can't find any info
about it.

Regards,
--davyd

-- 
Davyd Madeley        Software Engineer
Fugro Seismic Imaging, Perth Australia



More information about the antlr-interest mailing list