Behind the scenes of the C# yield keyword

After reading the great article about the code-saving yield keyword “Give way to the yield keyword” by Shay Friedman I thought it could be interesting to know how the yield keyword works behind the scenes.

…it doesn’t really end the method’s execution. yield return pauses the method execution and the next time you call it (for the next enumeration value), the method will continue to execute from the last yield return call. It sounds a bit confusing I think… (ShayF)

By using yield return within a method that returns IEnumerable or IEnumerator the language feature is activated.

Note: IEnumerable is kind of a stateless factory for Enumerators. IEnumerable.GetEnumerator() is thread safe and can be called multiple times, while the returned stateful Enumerator is just a helper for enumerating contained values once. By contract IEnumerator offers a Reset() method, but many implementations just throw a NotSupportedException.

Lets create an enumerator method that yields some Fibonacci nubmers.

public class YieldingClass
{
    public IEnumerable<int> GetFibonachiSequence()
    {
        yield return 1;
        yield return 2;
        yield return 3;
        yield return 5;
    }
}

Note: Yield is not a feature of the .Net runtime. It is just a C# language feature which gets compiled into simple IL code by the C# compiler.

The compiler now generates a inner class with following signature (Reflector + some renaming):

[CompilerGenerated]
private sealed class YieldingEnumerator : 
   IEnumerable<object>, IEnumerator<object>
{
    // Fields
    private int state;
    private int current;
    public YieldingClass owner;
    private int initialThreadId;

    // Methods
    [DebuggerHidden]
    public YieldingEnumerator(int state);
    private bool MoveNext();
    [DebuggerHidden]
    IEnumerator<int> IEnumerable<int>.GetEnumerator();
    [DebuggerHidden]
    IEnumerator IEnumerable.GetEnumerator();
    [DebuggerHidden]
    void IEnumerator.Reset();
    void IDisposable.Dispose();

    // Properties
    object IEnumerator<object>.Current 
    { [DebuggerHidden] get; }

    object IEnumerator.Current 
    { [DebuggerHidden] get; }
}

The original method GetFibonachiSequence() only returns a new instance of the YieldingEnumerator, passing the initial state –2 as well as itself as the owner.

Each enumerator holds a state indicating:

  • -2: Initialized as Enumerable. (Not yet an Enumerator)
  • -1: Closed
  • 0: Initialized as Enumerator. 
    If a new Enumerator is requested on the same instance, GetEnumerator() returns another new instance of YieldingEnumerator.
  •  1-n: Index of the yield return in the original GetFibonachiSequence()method. In case of nested enumerators or other more complex scenarios one yield return consumes more than one index.

The content of GetFibonachiSequence() is translated into YieldingEnumerator.MoveNext().

In our very simple scenario the code looks like this:

bool MoveNext()
{
    switch (state)
    {
        case 0:
            state = -1;
            current = 1;
            state = 1;
            return true;

        case 1:
            state = -1;
            current = 2;
            state = 2;
            return true;

        case 2:
            state = -1;
            current = 3;
            state = 3;
            return true;

        case 3:
            state = -1;
            current = 5;
            state = 4;
            return true;

        case 4:
            state = -1;
            break;
    }
    return false;
}

Quite easy, isn’t it?

So far we easily could have created the classes and methods used to enable the yield keyword ourselves, too.

But in more complex scenarios Microsoft does some tricks, which won’t compile as C# – at least not how Reflector translates the resulting IL code.

Lets have a look at some code with a nested enumeration…

foreach(int i in new int[] {1, 2, 3, 5, 8})
{
    yield return i;
}

This compiles into:

private bool MoveNext()
{
    try
    {
        switch (state)
        {
            case 0:
                state = -1;
                state = 1;
                this.values = new int[] { 1, 2, 3, 5, 8 };
                this.currentPositionInValues = 0;
                while (this.currentPositionInValues < this.values.Length)
                {
                    current_i = this.values[this.currentPositionInValues];
                    current = current_i;
                    state = 2;
                    return true;
                Label_007F:
                    state = 1;
                    this.currentPositionInValues++;
                }
                this.Finally2();
                break;

            case 2:
                goto Label_007F;
        }
        return false;
    }
    fault
    {
        this.System.IDisposable.Dispose();
    }
}
[/sourcecode]

Now the states 1 and 2 are used to indicate whether the enumerator actually is at some point (2), or wether it is trying to retrieve the next value (1).

Two things would not compile:

  • goto Label_007F is used to jump back into the iteration over int[] values. The C# goto statement is not able to jump into another statements context. But in IL this is totally valid as a while statement in MSIL is nothing but some gotos either.
  • The fault is proper MSIL, but not supported in C#. Basically it acts as a finally which just is executed in case of an error.

Attention: As in anonymous delegates, parameters as well as local and instance variables are passed to the YieldingEnumerator only once. Read this great post on this: Variable Scoping in Anonymous Delegates in C#

Thanks for your attention!

kick it on DotNetKicks.com

Advertisements

26 thoughts on “Behind the scenes of the C# yield keyword

  1. Hello Wesner,

    hi, i have not checked the MSIL spec, but i think fault does not catch an exception. The code is just excecuted when some exception occured. So it’s pretty much like:

    catch
    {
    /// some code
    throw;
    }

    My source:
    http://weblogs.asp.net/kennykerr/archive/2004/09/15/230167.aspx

    They write: The fault exception handler is similar to the finally block except that it is invoked only if it’s associated try block is left as a result of an exception. After the fault handler has been given an opportunity to execute, the exception continues on its way in search or a handler that is willing to catch it.

  2. ‘fault’ is a third type of exception handling block (after ‘catch’ and ‘finally’). It exists in MSIL, but can not be expressed in C#.

    It is similar to what larscorneliussen described above, but is slightly different in that the exception is not rethrown. Instead, the stack unwind from the original exception throw continues at the end of the fault block. (Exactly like a finally block that only runs when an exception was thrown.)

    It’s a subtle difference, but it ensures completely transparent exception handling, where in the case of a catch { throw; }, some things like the stack trace of the exception might be altered.

    Fault blocks were added into the CLR spec to support Managed C++ (where they’re used to implement deterministic finalization), and exception filters in VB.Net (where you can specify an expression as your catch condition, as opposed to just a type as in C#).

  3. Pingback: Weekly Link Post 46 « Rhonda Tipton’s WebLog

  4. Very interesting.

    This is actually the approach taken by the CCR (Currency and Coordination Runtime) guys to allow a procedural-esque programming style that executes asynchronously.

    I talk a little bit about it here
    http://thevalerios.net/matt/2008/06/multithreading-and-concurrency-in-net/
    but the real discussion can be found in this video
    http://channel9.msdn.com/shows/Going+Deep/CCR-Programming-Jeffrey-Richter-and-George-Chrysanthakopoulos/
    and this article
    http://msdn.microsoft.com/en-us/magazine/cc163556.aspx

  5. Pingback: Code Heaven's Required Reading - June 22, 2008 - Code Heaven

  6. Hello,

    Good article, excellent work.

    Is the yield thread safe? What happens when you have 2 threads hitting the same static method? Is the state maintained for each separate thread?

    Jacobson

  7. Pingback: C# tips « Bobobobo’s Weblog

  8. Pingback: Using C# Yield for Readability and Performance

  9. Pingback: yield naredba u C# « Bahrudin Hrnjica Web Page

  10. Pingback: What is the problem? « Struggles by Lars C.

  11. Pingback: Stream: Lazy Iteration with Continuation | Vjeux

  12. Pingback: Run Serial and Parallel Asynchronous Tasks in Unity3D and C# | Seba's Lab

  13. Pingback: The c# “yield” keyword explained « Jack's Sleazy Hacks

  14. Pingback: Ivan Krivyakov's Blog » PLinq and source IEnumerable thread safety

  15. Good article, Lars 😉
    IEnumerable/IEnumerator pair is the implementation of the Iterator Pattern in C# and thus many language features are built on that paradigm, such as LINQ, foreach, collections, async features (before the latest async “CTP” abilities some developers even had created their own async APIs making use of the enumerators and the iterator block).

    As if to the thread safety, when we extend the IEnumerable the compiler generates a special extra variable that keeps the unique ID of the current thread in the “enumerable” type’s constructor and then creates new instances of that object when another thread accesses the IEnumerable.GetEnumerator():

    //.ctor
    this.l__initialThreadId = Thread.CurrentThread.ManagedThreadId;

    // GetEnumerator()
    if ((Thread.CurrentThread.ManagedThreadId == this.l__initialThreadId) && (this.1__state == -2))
    {
    this.1__state = 0;
    return this;
    }
    return new Test.d__0(0);

  16. Pingback: .NET/C#: some starting posts on the `yield` keyword. « The Wiert Corner – irregular stream of stuff

  17. Pingback: 【转向Javascript系列】深入理解Generators | Web前端 腾讯AlloyTeam Blog | 愿景: 成为地球卓越的Web团队!

  18. Pingback: 【转向Javascript系列】深入理解Generators — 好JSER

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s