antlr3.TokenSource Class Reference
Abstract baseclass for token producers. More...
Public Member Functions | |
| def | nextToken |
| Return a Token object from your input stream (usually a CharStream). | |
| def | __iter__ |
| The TokenSource is an interator. | |
| def | next |
| Return next token or raise StopIteration. | |
Detailed Description
Abstract baseclass for token producers.A source of tokens must provide a sequence of tokens via nextToken() and also must reveal it's source of characters; CommonToken's text is computed from a CharStream; it only store indices into the char stream.
Errors from the lexer are never passed to the parser. Either you want to keep going or you do not upon token recognition error. If you do not want to continue lexing then you do not want to continue parsing. Just throw an exception not under RecognitionException and Java will naturally toss you all the way out of the recognizers. If you want to continue lexing then you should not throw an exception to the parser--it has already requested a token. Keep lexing until you get a valid one. Just report errors and keep going, looking for a valid token.
Definition at line 3457 of file antlr3.py.
Member Function Documentation
| def antlr3.TokenSource.nextToken | ( | self | ) |
Return a Token object from your input stream (usually a CharStream).
Do not fail/return upon lexing error; keep chewing on the characters until you get a good one; errors are not passed through to the parser.
Reimplemented in antlr3.Lexer.
| def antlr3.TokenSource.__iter__ | ( | self | ) |
The TokenSource is an interator.
The iteration will not include the final EOF token, see also the note for the next() method.
| def antlr3.TokenSource.next | ( | self | ) |
The documentation for this class was generated from the following file:
