Re: Réf. : Re: Réf. : Re: [antlr-interest] java.lang.OutOfMemoryError
Alexey Demakov
demakov at ispras.ru
Mon Jul 25 07:11:12 PDT 2005
Of course it works. But what if your input file contains '\uffff'?
For example, 5 files in sun jdk 1.5.0 sources contain it in string literals.
The only solution I see is to make additional check of input
when '\uffff' occurs in lexer - to distinguish between real '\uffff'
and end of stream.
Regards,
Alexey
-----
Alexey Demakov
TreeDL: Tree Description Language: http://treedl.sourceforge.net
RedVerst Group: http://www.unitesk.com
----- Original Message -----
From: <loic.lefevre at bnpparibas.com>
To: <antlr-interest at antlr.org>
Sent: Monday, July 25, 2005 5:56 PM
Subject: Réf. : Re: Réf. : Re: [antlr-interest] java.lang.OutOfMemoryError
> Indeed, I was just finishing running all my testcases but I've found a
> solution slightly similar than yours:
>
> // Single-line comments
> SL_COMMENT
> : "//"
> (~('\uffff'|'\n'|'\r'))* ('\n'|'\r'('\n')?)?
> {$setType(Token.SKIP); newline();}
> ;
>
> I've just added this and now it works.
> I also added it here:
>
> // string literals
> STRING_LITERAL
> : '"'! (REGEXP_ESC|~('\uffff'|'"'|'\\'|'\n'|'\r'))* '"'!
> ;
>
> Thanks a lot!
> Loïc
>
>
>
>
> Internet
> demakov at ispras.ru@antlr.org - 25/07/2005 15:31
>
>
> Envoyé par : antlr-interest-bounces at antlr.org
>
>
>
> Pour : antlr-interest
>
> cc :
>
>
> Objet : Re: Réf. : Re: [antlr-interest] java.lang.OutOfMemoryError
>
> antlr casts -1 read from input reader:
> CharBuffer.fill(i): queue.append( (char)input.read() );
>
> to FFFF
>
> When charVocabulary includes '\uFFFF':
>
> charVocabulary='\u0003'..'\uFFFF';
>
> FFFF is included in the set ~('\n'|'\r') and is successfully matched in the
> loop.
>
> You can remove FFFF from vocabulary:
>
> charVocabulary='\u0003'..'\uFFFE';
>
> But I don't like this solution because it doesn't handle unicode '\uFFFF'.
>
> Regards,
> Alexey
>
> -----
> Alexey Demakov
> TreeDL: Tree Description Language: http://treedl.sourceforge.net
> RedVerst Group: http://www.unitesk.com
>
>
> ----- Original Message -----
> From: <loic.lefevre at bnpparibas.com>
> To: <antlr-interest at antlr.org>
> Sent: Monday, July 25, 2005 4:28 PM
> Subject: Réf. : Re: [antlr-interest] java.lang.OutOfMemoryError
>
>
>
> Seems I found the origin of the problem but I can't find how to resolve it
> yet:
>
> All comes from the implementation of the Reader I use, if you change your
> code with:
>
> ---8
>
<-----------------------------------------------------------------------------------------------------------------------------------
>
> ----------
> public static void main( String[] args ) throws TokenStreamException
> {
> RosterExpressionLexer lexer = new RosterExpressionLexer( new
> java.io.StringReader("//") );
> while( true )
> {
> Token t = lexer.nextToken();
> if( t.getType() == RosterExpressionTokenTypes.EOF ) break;
> System.out.println( "'" + t.getText() + "'" );
> }
> }
> ---8
>
<-----------------------------------------------------------------------------------------------------------------------------------
>
> ----------
>
> then you'll get the OutOfMemoryError!
>
> Argh, I'm investigating on the behaviour of this java class!
>
> Loïc
>
>
>
>
> Internet
> demakov at ispras.ru@antlr.org - 25/07/2005 13:19
>
>
> Envoyé par : antlr-interest-bounces at antlr.org
>
>
>
> Pour : antlr-interest
>
> cc :
>
>
> Objet : Re: [antlr-interest] java.lang.OutOfMemoryError
>
> Here is shortened version of your parser with main() reading System.in
> I don't see any problems with it
>
> Regards,
> Alexey
>
> -----
> Alexey Demakov
> TreeDL: Tree Description Language: http://treedl.sourceforge.net
> RedVerst Group: http://www.unitesk.com
>
>
> ----- Original Message -----
> From: <loic.lefevre at bnpparibas.com>
> To: <antlr-interest at antlr.org>
> Sent: Monday, July 25, 2005 2:33 PM
> Subject: [antlr-interest] java.lang.OutOfMemoryError
>
>
>
> Hello,
> My grammar to handle expressions is now finished but now I'm facing a
> strange behaviour:
> When I try to validate hand typed expressions, I get the terrible
> java.lang.OutOfMemoryError when
> parsing the expression: // (two chars)
>
> Here is my grammar:
> (See attached file: RosterExpression.g)
>
> It appears that the lexer loops forever:
>
> public final void mSL_COMMENT(boolean _createToken) throws
> RecognitionException, CharStreamException, TokenStreamException {
> int _ttype; Token _token=null; int _begin=text.length();
> _ttype = SL_COMMENT;
> int _saveIndex;
>
> match("//");
> {
> _loop59:
>
>
> //--> begin of loop forever
> do {
> if ((_tokenSet_0.member(LA(1)))) {
> {
> match(_tokenSet_0);
> }
> }
> else {
> break _loop59;
> }
>
> } while (true);
> //--> end of loop forever
>
>
> }
> {
> switch ( LA(1)) {
> case '\n':
> {
> match('\n');
> break;
> }
> case '\r':
> {
> match('\r');
> {
> if ((LA(1)=='\n')) {
> match('\n');
> }
> else {
> }
>
> }
> break;
> }
> default:
> {
> }
> }
> }
> if ( inputState.guessing==0 ) {
> _ttype = Token.SKIP; newline();
> }
> if ( _createToken && _token==null && _ttype!=Token.SKIP ) {
> _token = makeToken(_ttype);
> _token.setText(new String(text.getBuffer(), _begin,
> text.length()-_begin));
> }
> _returnToken = _token;
> }
>
>
> Thanks in advance for your help.
> Loïc
>
> This message and any attachments (the "message") is
> intended solely for the addressees and is confidential.
> If you receive this message in error, please delete it and
> immediately notify the sender. Any use not in accord with
> its purpose, any dissemination or disclosure, either whole
> or partial, is prohibited except formal approval. The internet
> can not guarantee the integrity of this message.
> BNP PARIBAS (and its subsidiaries) shall (will) not
> therefore be liable for the message if modified.
>
> ---------------------------------------------
>
> Ce message et toutes les pieces jointes (ci-apres le
> "message") sont etablis a l'intention exclusive de ses
> destinataires et sont confidentiels. Si vous recevez ce
> message par erreur, merci de le detruire et d'en avertir
> immediatement l'expediteur. Toute utilisation de ce
> message non conforme a sa destination, toute diffusion
> ou toute publication, totale ou partielle, est interdite, sauf
> autorisation expresse. L'internet ne permettant pas
> d'assurer l'integrite de ce message, BNP PARIBAS (et ses
> filiales) decline(nt) toute responsabilite au titre de ce
> message, dans l'hypothese ou il aurait ete modifie.
>
> (See attached file: RosterExpression.g)
>
> This message and any attachments (the "message") is
> intended solely for the addressees and is confidential.
> If you receive this message in error, please delete it and
> immediately notify the sender. Any use not in accord with
> its purpose, any dissemination or disclosure, either whole
> or partial, is prohibited except formal approval. The internet
> can not guarantee the integrity of this message.
> BNP PARIBAS (and its subsidiaries) shall (will) not
> therefore be liable for the message if modified.
>
> ---------------------------------------------
>
> Ce message et toutes les pieces jointes (ci-apres le
> "message") sont etablis a l'intention exclusive de ses
> destinataires et sont confidentiels. Si vous recevez ce
> message par erreur, merci de le detruire et d'en avertir
> immediatement l'expediteur. Toute utilisation de ce
> message non conforme a sa destination, toute diffusion
> ou toute publication, totale ou partielle, est interdite, sauf
> autorisation expresse. L'internet ne permettant pas
> d'assurer l'integrite de ce message, BNP PARIBAS (et ses
> filiales) decline(nt) toute responsabilite au titre de ce
> message, dans l'hypothese ou il aurait ete modifie.
>
>
>
>
>
> This message and any attachments (the "message") is
> intended solely for the addressees and is confidential.
> If you receive this message in error, please delete it and
> immediately notify the sender. Any use not in accord with
> its purpose, any dissemination or disclosure, either whole
> or partial, is prohibited except formal approval. The internet
> can not guarantee the integrity of this message.
> BNP PARIBAS (and its subsidiaries) shall (will) not
> therefore be liable for the message if modified.
>
> ---------------------------------------------
>
> Ce message et toutes les pieces jointes (ci-apres le
> "message") sont etablis a l'intention exclusive de ses
> destinataires et sont confidentiels. Si vous recevez ce
> message par erreur, merci de le detruire et d'en avertir
> immediatement l'expediteur. Toute utilisation de ce
> message non conforme a sa destination, toute diffusion
> ou toute publication, totale ou partielle, est interdite, sauf
> autorisation expresse. L'internet ne permettant pas
> d'assurer l'integrite de ce message, BNP PARIBAS (et ses
> filiales) decline(nt) toute responsabilite au titre de ce
> message, dans l'hypothese ou il aurait ete modifie.
>
More information about the antlr-interest
mailing list