[antlr-interest] Geting all tokens from lexer / token stream

[Lars von Wedel]

Hi again,

I came up with this now which made my tests happy:

            do
            {
                tokens.Consume();
            } while (tokens.LastToken.Type != MyLexer.EOF);

Afterwards I can go forward with tokens.GetTokens(). Is that how it is
supposed to be used ?

Lars

On Wed, May 11, 2011 at 3:22 PM, Lars von Wedel <lars.vonwedel at gmail.com>wrote:

> Thanks for the answer. However, calling GetTokens() yields an empty list
> for me. This is my code (C#):
>
>             var inputStream = new ANTLRStringStream(input);
>             var lexer = new MyLexer(inputStream);
>             var tokens = new CommonTokenStream(lexer,
> BaseRecognizer.DefaultTokenChannel);
>
>             foreach (var token in tokens.GetTokens())
>             {
>                 System.Console.WriteLine(token);
>             }
>
> Interestingly, it works, if I inspect the token stream variable in the VS
> debugger.
>
> Lars
>
>
> On Wed, May 11, 2011 at 3:03 PM, Bart Kiers <bkiers at gmail.com> wrote:
>
>> CommonTokenStream inherits getTokens() which returns a List of Tokens.
>> You'll need to cast them to a Token (or something that extends a Token),
>> since it's a non-generics List list:
>>
>> CommonTokenStream tokens = new CommonTokenStream(lexer);
>> for(Object o : tokens.getTokens()) {
>>   Token t = (Token)o;
>>   System.out.println(t);
>> }
>>
>>
>> Regards,
>>
>> Bart.
>>
>> On Wed, May 11, 2011 at 1:40 PM, Lars von Wedel <lars.vonwedel at gmail.com>wrote:
>>
>>> Hello,
>>>
>>> I am writing an interactive interpreter and I would like to obtain all
>>> tokens from a lexer of token stream to test, whether the input is
>>> complete
>>> or continued on the next line.
>>>
>>> What is the easiest approach to do this ? I tried a CommonTokenStream but
>>> I
>>> am not sure how to tell it to pull all tokens from the lexer.
>>>
>>> Thanks and Regards,
>>> Lars
>>>
>>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
>>> Unsubscribe:
>>> http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>>>
>>
>>
>