[antlr-interest] spaces between tokens

[Anakreon]

silverio.di at qem.it wrote:
> 
> 
> 
> 
> Hi,
> I've a big problem.
> 
> In my grammar, how in many others, the whitespaces are skipped in lexer,
> but I've some circumstances in which I need to check that not any spaces
> are present between tokens.
> 
> Example :
> WeekJobHour at Monday = 8
> 
> would mean assign 8 (hours) to parameter Monday of structure WeekJobHour.
> I would like my lexer extract following tokens:
> 
> IDENT ATSIGN IDENT
> 
> but my problem is to check than not any WS are present between
> IDENT and ATSIGN and between ATSIGN and IDENT so
> 
> WeekJobHour at Monday = 8        // is OK
> WeekJobHour @Monday = 8       // is BAD
> WeekJobHour@ Monday = 8       // is BAD
> WeekJobHour  @ Monday = 8           // is BAD too !
> 
> I could use following lexer rule:
> 
> STRUCT_PARAMETER
>       :     ('A'..'Z' | 'a..z')+
>             '@'
>             ('A'..'Z' | 'a..z')+
>       ;
> 
> but in parser how can I extract the structure name (WeekJobHour)
> and the structure parameter (Monday) form STRUCT_PARAMETER
> token ?
> 
> I think a similar issue is present in C/C++ structure construct
> 
> Thank you for your suggestions about
> Silverio Diquigiovanni
Make a class wich implements TokenStream wich uses the Lexer.
In the nextToken method, if the lexer returns a token of type 
STRUCT_PARAM, split the token in 3 tokens where the first would be
of type STRUCT_NAME the second STRUCT_AT and the third STRUCT_DAY
and the text of the tokens WeekJobHour, @, Monday respectively.
return the first token from the method and store the other 2.
In the next 2 calls of nextToken return the stored ones.

Pass the implementor of TokenStream instead of your Lexer to the
parser.

Anakreon


 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
    antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/