[antlr-interest] Geting all tokens from lexer / token stream

[Lars von Wedel]

Thanks for the answer. However, calling GetTokens() yields an empty list for
me. This is my code (C#):

            var inputStream = new ANTLRStringStream(input);
            var lexer = new MyLexer(inputStream);
            var tokens = new CommonTokenStream(lexer,
BaseRecognizer.DefaultTokenChannel);

            foreach (var token in tokens.GetTokens())
            {
                System.Console.WriteLine(token);
            }

Interestingly, it works, if I inspect the token stream variable in the VS
debugger.

Lars

On Wed, May 11, 2011 at 3:03 PM, Bart Kiers <bkiers at gmail.com> wrote:

> CommonTokenStream inherits getTokens() which returns a List of Tokens.
> You'll need to cast them to a Token (or something that extends a Token),
> since it's a non-generics List list:
>
> CommonTokenStream tokens = new CommonTokenStream(lexer);
> for(Object o : tokens.getTokens()) {
>   Token t = (Token)o;
>   System.out.println(t);
> }
>
>
> Regards,
>
> Bart.
>
> On Wed, May 11, 2011 at 1:40 PM, Lars von Wedel <lars.vonwedel at gmail.com>wrote:
>
>> Hello,
>>
>> I am writing an interactive interpreter and I would like to obtain all
>> tokens from a lexer of token stream to test, whether the input is complete
>> or continued on the next line.
>>
>> What is the easiest approach to do this ? I tried a CommonTokenStream but
>> I
>> am not sure how to tell it to pull all tokens from the lexer.
>>
>> Thanks and Regards,
>> Lars
>>
>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
>> Unsubscribe:
>> http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>>
>
>