[antlr-interest] "dynamic" whitespace?

[Skip Sailors]

Dear List,

 

N00b.

 

I have a stream of characters that can be chunked into lines, and the
characters in a line can be chunked into tokens.  My problem is that the
character denoting the end of line and the character delimiting a token
changes from message to message.  For any message I know that the third
character will be used as a token delimiter and the 100th will be the
line delimiter, or something like that.  The idea is that I can't
generally reserve a character that will not appear as content in the
message, but I can reserve a character on a message-by-message basis.

 

I would like to learn to build some grammar tools to process this kind
of stream, but I don't know where to start.  All the books and articles
I have read so far on language start with a fixed, static list of
delimiters.  Can you help me learn how to design language tools when
whitespace tokens vary?

 

TIA

 

Skip Sailors 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20091209/94cec01d/attachment.html