Dudi Keleti

Writing IL according to the specification

Mon, 14 Mar 2022 00:00:00 GMT

I have to deal with IL quite a bit, whether it’s to write IL code or a C# code that simulates running IL instructions according to C# spec.

In order to write an IL, of course, you need to have a basic knowledge of IL instructions, stack operations, and maybe other certain things.

But there is really no need to write a large method from scratch because you can always write it in C# and see through a decompiler tool how it looks in IL.

So if for example, you need to emit some code (whether with Reflection.Emit or with Cecil, or any other framework), you can write the C# code that emulates the method and then write the decompiled IL.

On the other hand, sometimes the thing is that I do not have to write the IL (because it can be copied from an existing method), but to write code that simulates the execution of the code in the sense of how it will run according to C# spec. (e.g. if you writing an interpreter or asserting generating IL).

And this requires deep knowledge, not only in IL but in CLR and C# spec.

What does that mean to write it according to spec?

Assume we have these types:(All examples are on x64 in release mode).

interface MyInterface{} class MyGenericObject<T>{}

And there’s the following code: (1)

static void IsInterfaceGeneric(T t)
{
    System.Console.WriteLine(t is MyInterface);
}

Pretty simple, we got a generic parameter and we check it against another type. If you just want to write this piece of code in IL, it’s very simple:

IL_0000: ldarg.0
IL_0001: box !T 
IL_0006: isinst MyInterface
IL_000b: ldnull
IL_000c: cgt.un
IL_000e: call void [System.Console]System.Console::WriteLine(bool)
IL_0013: ret

Why the box? Because if the object is a generic parameter (or a value type), the object is boxed first.

Let’s see another example: (2)

private static void AsInterfaceGeneric(T t)
{
    System.Console.WriteLine(t as MyInterface);
}

We got a generic parameter and we treat it as another type. The code is almost identical, all of us see is and as in C# every day. So what about IL?

IL_0000: ldarg.0
IL_0001: box !T
IL_0006: isinst MyInterface
IL_000b: call void [System.Console]System.Console::WriteLine(object)
IL_0010: ret

As you see, it’s two instructions less although both of them are translated to isinst instruction. And if you trying to be identical to what the C# compiler generating, it’s important to know the differences.

The isinst instruction check object against type token, but in the is example, we want to know if that check succeeded, and in the as example, we don’t care about the success result, just about the returning isinst result (null or target object)

Now let’s see another very similar code: (3)

private static void CastInterfaceGeneric(T t)
{
    System.Console.WriteLine((MyInterface)t);
}

What will be the difference? Maybe you already know that the difference between as and a direct cast is if the runtime will throw an exception in case of failure.

IL_0000: ldarg.0
IL_0001: box !T
IL_0006: castclass MyInterface
IL_000b: call void [System.Console]System.Console::WriteLine(object)
IL_0010: ret

As you guessed, or not, the result is indeed exactly the same.

Now let’s do it a little more complicated by using constraints

What will be in each example if the type was written like this:

interface MyInterface{
class MyGenericObjec<T> where T : MyInterface {}

Now the generic type has a constraint, is it change the generated IL?

Let’s start with the 1st example.

IL_0000: ldarg.0
IL_0001: box !T
IL_0006: ldnull
IL_0007: cgt.un
IL_0009: call void [System.Console]System.Console::WriteLine(bool)
IL_000e: ret

Note that the isinst instruction has been removed.

And for the 2nd the 3rd examples:

IL_0000: ldarg.0
IL_0001: box !T
IL_0006: call void [System.Console]System.Console::WriteLine(object)
IL_000b: ret

Yes, they will have the exact IL although as translated to isinst and direct cast translated to castclass.

The reason for this change is that we added the constraint on the generic type, this let the compiler optimize the output because he now understands that T type must be from MyInterface type, so no need to cast ( castclass ) or check ( isinst ).

So why the 1st example still has the isinst?

This is because as I wrote, the is C# keyword returns not the type itself, but the result of the check, and we need that result.

And more complicated when considering optimizations

But this rise another question if the compiler knows this info and is considering it in the generating process, why he is not taking it another step farther and removing ldnull and cgt.un and simply pushing 1 to stack?

And again it returns us to read the specification, where is specified that in the isinst instruction, we should return false in case of the object is null so we can’t remove this check because we must test it against cases that t is null.

Make sense, so what will be here?

static void IsWithNotNullObject()
{
    System.Console.WriteLine(new MyObject() is MyInterface);
}

In this case, the compiler can see that obj is not null so no need to check, right?

But as we can see the check still exists in the IL:

IL_0000: newobj instance void MyObject::.ctor()
IL_0005: ldnull
IL_0006: cgt.un
IL_0008: call void [System.Console]System.Console::WriteLine(bool)
IL_000d: ret

But in this case, if we will look at the machine code we will see the difference:

In the “maybe null” example it will look like this:

L0000: test rdx, rdx
L0003: setne cl

And in the “not null” example, it will look like this:

L0004: mov ecx, 1

A bit more with constriants

Now that we saw the optimization that can be obtained with generic constraints, lest see another constraint example.

class MyGenericObject<T> where T : class, MyInterface
void IsInterfaceGeneric(T t)
{
    System.Console.WriteLine(t is MyInterface);
}

Now the compiler can know that T must be a reference type, so no box needed?

IL_0000: ldarg.1
IL_0001: box !T
IL_0006: ldnull
IL_0007: cgt.un
IL_0009: call void [System.Console]System.Console::WriteLine(bool)
IL_000e: ret

No, the box still there.

So I’m sure that this example will not change anything, right?

struct MyGenericObject<T> where T : struct, MyInterface
void IsInterfaceGeneric(T t)
{
    System.Console.WriteLine(t is MyInterface);
}

Is exactly the same but with struct instead of class

And the answer is…

IL_0000: ldc.i4.1
IL_0001: call void [System.Console]System.Console::WriteLine(bool)
IL_0006: ret

What???

Yes, now the compiler can be smart enough because T can’t be null - it is a value type - and it must be `MyInterface` because it is constrained, so we can just push 1 to the stack!

Summary

If we need to run code like the C# compiler will generate it (or like how the CLR will run it), we must know, not just how to write it in IL, but, which IL to write in each case and how to run this IL in each case.

The point that I wanted to make is, writing IL is one thing - you can copy/paste - but writing an IL interpreter, even in C# code, that is compatible with the spec is another thing because you need to know every corner of the spec.

Here is the spec for isinst and castclass if you are interested (or read it from the source).

How to inspect a MethodSpec signature without code

Sun, 13 Mar 2022 00:00:00 GMT

So much time has passed since the last time I wrote here. It’s not that I did not have anything to write, but it probably was not important enough to me. I hope that I will continue writing more often.

This time I will write something short that may help some people.

Why do you ever need it and why without code?

For the first part, I can think of various reasons, but I’m sure that if you read it, you know why you need it. Still, I will write two reasons why you might need it. One is for learning purposes and two, for writing these signatures yourself if is part of your work project.

For the second part, I find it easier to have a UI tool for debugging sessions when you want to understand what signatures look like, instead of using libraries like dnlib to do that. But, if you prefer the code way, it’s ok.

But first, let’s start with some background.

What is a MethodSpec?

A MethodSpec is a method that represents a specific instantiation of a generic method.

In the metadata, there are two tables for methods (among other tables) one is the MethodDef table - for methods that are defined in the assembly, and the other one is a MemberRef table for methods that are defined in a referenced assembly.

Now let’s talk about generic methods. If there is a method that defined in a generic form, for example:

void Foo<T>(T t){}

The method will have an entry in the appropriating method table in the open form (MethodDefOrRef), meaning that this method does not define who is the generic parameter T.

Once in someplace in the code, there will be a usage of this generic method, e.g.

Foo<string>(myString)

Then it will create an entry to the MethodSpec table. (There is a difference if the generic parameter is a value type or reference type, but is not important for this example).

MethodSpec table contains two columns. One for the parent method (that is, a MethodDef or MemberRef (MethodDefOrRef) that represent the open form of this generic method), and the second for the specific method instantiation. I’ll explain it in a bit.

What is a MethodSpec signature?

A method signature is a way to represent the identity of the method. It contains some info (it’s depending on the signature type) like return type, generic parameters, regular parameters, and more.

A MethodSpec signature is, as the name says, a signature for a specific generic method. It contains a special value for the calling convention that says that this method is a generic method instantiation, then the number of generic parameters, and then the generic parameter types. (Other info exists in the MethodDefOrRef signature).

The instantiation that I wrote above is the signature of a specific generic method.

So to summarize, a MethodSpec represents a specific generic method instantiation, by piecing together a reference to the open generic method (the MethodDefOrRef) and the type information for this specific generic method instantiation.

But it is a little tricky to get the instantiation because it’s not written in the column itself, but the column contains an index to the Blob stream, which is a binary stream, so no easy way to understand what a MethodSpec instantiation looks like.

And as I wrote at the beginning, the whole point is that you want to see what the signature looks like, whether for learning or whether you are debugging a signature you wrote yourself.

How to use a tool to inspect a MethodSpec signature?

There are various disassembler tools that you can use to learn about metadata, Ildasm, ILSpy, dotPeek, and more.

But the maximum valuable info that those tools will give you related to MethodSpec instantiation is the representable text of the signature. i.e. Foo<string>(string), and not the bytes from the blob stream, so all of those tools are not so helpful for this task.

There are other types of tools that can read and understand PE’s and .NET file structure so you can use them to look at the blob stream. It will work, but it’s not a quick and easy way to find the right offset.

Other tools are debugging tools, and we are not talking here about this, but on static assembly tools.

The only two tools (that I’m familiar with it) that will give you the full info in an easy way are dnSpy and CFF Explorer!

They are pretty amazing tools (you probably already knew that if you ever used them) and must-have tools for this kind of project.

It’s very similar to inspecting metadata in both of them.

To use dnSpy to find anything in the blob stream (yes, I talked about MethodSpec signature, but it’s just for the example, you can find whatever is important to you), find the required offset, go to the blob stream under the PE, then right-click and choose “Show Data in HEX Editor”, then ctrl+g, write the required offset and choose in the box, “Current Position”.

For MethodSpec it will be like the following:

Find the method in the MethodDef or MemberRef table
Go to the relevant MethodSpec entry (where ‘method’ column point to the right method)
Copy the instantiation column value (is the offset into the blob stream)
Go to the Blob stream and right-click + “Show Data in HEX Editor”
ctrl+g and paste the offset value, then choose “Current Position”
You will jump to the place of the bytes that represent the specific generic method instantiation!

And in CFF Explorer:

Find the method in the MethodDef or MemberRef table
Go to the relevant MethodSpec entry (where ‘method’ column point to the right method)
Copy the instantiation column value (is the offset into the blob stream)
Go to the Blob stream
Click on the arrow icon (Go To Offset) and paste the offset value
You will jump to the place of the bytes that represent the specific generic method instantiation!

Let’s see the instantiation for an example method and what each byte represents.

Foo<struct>(myStruct s)

In the Blob stream it will look like this:

05 - length of signature 0A - IMAGE_CEE_CS_CALLCONV_GENERICINST 01 - number of generic parameters 11 - ELEMENT_TYPE_VALUETYPE 80 A4 - coded token of the TypeDefOrRef ‘myStruct’

The object header gets complicated

Wed, 10 Jan 2018 00:00:00 GMT

*Object header, sync block, thin lock vs fat lock, hashcode - all of this and more will be explained in the following post.*First, let’s make a brief overview on the object header in the CLR.What is the object header?

An header is - like it sound - header “above” the object. If I try to illustrate it, it will be like this ––>

Every object in CLR has an header that contains info related to the object. The run -time use this info as we’ll see later in this post.

In memory, the header will sits in a negative offset from the object pointer: in a x86 (32 bit) process, the offset will be -4 bytes, and in a x64 process the offset will be -8 bytes.(Keep in mind the although the offset changes from 4 bytes to 8 bytes, the sync block itself is always 4 bytes even in a 64 bit process.)**Why do we need the header?**The CLR uses that header to store a few things the run-time needs. Let’s start to reveal them, bit by bit.

0 - 15: Thin lock information - 16 bits that are composed from 10 bits for thread id that’s holding the lock (0 mean no thread holding the lock) and 6 bits for recursion level (0 mean no lock was taken or only taken once by the same thread).Thin lock meaning that all the lock info sits on the object header and that the sync block won’t create it (will explain the details later in this post)
16 - 26: App domain index - 11 bits (2048 indices) index of the app domain that the object belongs to. (Set only when needed, e.g. for COM interop)
0 - 25: Sync block index - 26 bits: the index in the array of sync blocks (0 for shared sync block for all objects)
0 - 25: Hash code - 26 bits for object hashcode if not overridden
26: BIT_SBLK_IS_HASHCODE - 1 bit that marks if the rest of the word is a hash code or sync block index
27: BIT_SBLK_IS_HASH_OR_SYNCBLKINDEX - 1 bit that tell us if hash code or sync block index are set
28: BIT_SBLK_SPIN_LOCK - 1 bit that is set if we want to lock the header for changing it value
29: BIT_SBLK_GC_RESERVE - 1 bit that is set only during the gc process, if the object is pinned
30: BIT_SBLK_FINALIZER_RUN - 1 bit that marks the object is finalized already (via GC.SuppressFinalize for example). When the finalizer thread is run, if this bit is turned on, it skip it (but always set it back to 0 to support “GC.RegisterForFinalization()”. Note that if it will remain turned on, the finalizer will skip it even if we ask to RegisterForFinalization)
31: BIT_SBLK_AGILE_IN_PROGRESS - Actually I’m not sure that I understand completely what this bit means, but it’s less important. In general this 1 bit is set only in debug build and it checks for infinite loop between two objects (object 1 has a ref to object 2 and object 2 has a ref to object 1)

There are 3 more markers that are used only for string type (0x80000000, 0x40000000 and 0xC0000000 if both are set). Note that in case of strings we use the BIT_SBLK_AGILE_IN_PROGRESS and BIT_SBLK_FINALIZER_RUN and its OK because we don’t need them in case of string (no meaning for infinite loop or finalizer in strings).

Sync Block

What’s the use of sync block index?As you probably saw, in the above list there are more bytes than the header can store, and that’s why we have the bits flags that let us know which info is currently stored in the header.

If we want to store more info than the header can store (e.g. lock info and hash code for the object), there is another object called SyncBlock.

SyncBlock is a data structure that mainly stores info about thread synchronization (Monitor.Enter, Exit and Monitor.Wait, Pulse). But it also stores other info like hash code and app domain index. So, in case we don’t have enough space in the header, a SyncBlock will be created and all the info will be saved there. All sync blocks are stored in an array that’s accessible by index.SyncBlock will be created in an additional case, when a kernel event object is involved and we need to store the handle to it in the SyncBlock.

This lock vs Fat lock

II’ll explain it in a few words. When we lock on some object, this lock called “thin lock” and we store only the thread id and the recursion level (how much times we enter the monitor from the same thread). Thin lock is stored in the object header if it is possible, if not (there is a computed hash code for this object), it will be store in sync block object and the lock will become a “fat lock”. Even if there isn’t a computed hash code for the object, there is a case when the lock becomes a “fat lock”. The cases are when contention occurs or when condition variable is involved (via Wait, Pulse etc.). Because for these cases, additional info need to be store - such as a handle to the kernel object or a list of events that are associated with the lock - and this is stored in a sync block.

Note that when using the thin lock, the lock operation is very fast. If it the first lock, just set the thread id and when contention occurs in most cases it just spins till the thread id in object header is zero then set to the new thread id. If it doesn’t succeed, sync block is created.

Return to sync block indexSo the use of the sync block index is to point to the index of the sync block object in the array of sync blocks.

Sync block implementationThe actual implementation is a bit complicated (and I think also the comments in the code don’t always match the code itself and it makes it more difficult to understand).

If I simplify it, I can say that there is an array of sync blocks (SyncBlockArray) and there is a table of SyncTableEntry and a cache manager SyncBlockCache. Each sync block index points to the sync table entry that in turn points to the actual sync block and to the object. The cache responsibility is to allocate and release sync blocks in the array itself and in the entries table. Sync block is reclaimed by the GC when not needed but they are reusable.

Example to something that isn’t very clear to me is the following comments:

“many objects have a SyncTableEntry but no SyncBlock. That’s because someone (e.g. HashTable) called Hash() on them”

or:

“a common case (need a hash code for an object) you just need a syncTableEntry”

I don’t understand it because as I see it in the code, the hash code is stored in the header itself when possible or in the sync block itself, so how will a sync table entry will help here? Maybe I’m missing something here.As a side note, what will happen if for example we need to create too many sync blocks entries? As you saw there are some limited bits to hold it. As other cases in software design, we need to decide how to handle each case (unlimited growth, override old, throw exception) here is an example code from theSyncBlockCache::Grow method:

// Compute the size of the new synctable. Normally, we double it - unless
// doing so would create slots with indices too high to fit within the
// mask. If so, we create a synctable up to the mask limit. If we're
// already at the mask limit, then caller is out of luck.
DWORD newSyncTableSize;
if (m_SyncTableSize <= (MASK_SYNCBLOCKINDEX >> 1))
{
newSyncTableSize = m_SyncTableSize * 2;
}
else
{
newSyncTableSize = MASK_SYNCBLOCKINDEX;
}
if (!(newSyncTableSize > m_SyncTableSize)) // Make sure we actually found room to grow!
{
COMPlusThrowOM();
}

Hash Code

Let’s examine the different cases of GetHashCode. The first one is the case overridden method and I skip on it.If we remember the start of this post, the hash code is a 26-bit length block (really 26? we will see) that’s stored in the object header. So, in the common case, when no lock is taken on the object, the hash code will be computed and saved in object header.But if there is a need to create a sync block the hash code will move to the sync block (is already computed) and it will still be a 26 bit hash code or will be computed when required and saved directly in sync block, then it will save in a 32-bit field.How is the hash code computed?

I can’t be sure, but from the source it’s look like this:

DWORD multiplier = GetThreadId()*4 + 5;
m_dwHashCodeSeed = m_dwHashCodeSeed*multiplier + 1;
return m_dwHashCodeSeed >> (32-HASHCODE_BITS);

And the method must enforce to get result bigger then 0, because 0 means no hash code has been set yet.When I call GetHashCode what happening?

We search in the object header for BIT_SBLK_IS_HASH_OR_SYNCBLKINDEX & BIT_SBLK_IS_HASHCODE if both set, the hash code is the followed 26 bits in the header.

if just BIT_SBLK_IS_HASH_OR_SYNCBLKINDEX is set, we need to search in the sync block or create a new one if it doesn’t exist.

if BIT_SBLK_IS_HASH_OR_SYNCBLKINDEX is not set, if we are in thin lock, we need to create a sync block and save the hash code there, if not, we need to compute new hash code and save it in the header.SummarizeObject header is some meta data for the run-time that stores info mainly about lock and hashcode, but also for more things as we saw. The header is limited in size, so when we need, the header contains sync block index to the sync block structure that store all thread synchronization info and more (hash code, app domain index).Locks can be handled in the header itself (thin lock) or in sync block (fat lock).Hash code is stored in header itself or in sync block.

for further reading, just look mainly in syncblk.h and syncblk.cpp files in coreclr github. There is more source files the can give info but if you serach you will get them (threading.md, object.cpp, gc.cpp and more).

C# behind the scenes

Wed, 19 Jul 2017 00:00:00 GMT

I see a lot of beginners programmers, but certainly not just beginners, who are not sure how things work in C#.

I’m not talking about the CLR but about the language itself. And mostly specific on all sorts of keywords which are compiler sugar.

There are simple things that probably everyone understands what it means, such as this or const. But what about yield or async and what happens when we use lambda expression or write Linq in the form of query comprehension?

So I decided to do a series of articles (hopefully series …) about all kinds of words like this and what happens behind the scenes when you write a particular word in c #.

I do not like long articles unnecessarily, so I’ll try to keep the pattern short and permanent. An example of code in c #, an example of the code that was actually generated from it, and a brief explanation of what happens there.

Hope it will be useful.

C# behind the scenes: Local functions

Wed, 19 Jul 2017 00:00:00 GMT

The second “Behind the scenes” article, will be about local function.

This is much simple from previous post about yield keyword. In short, local function translated to a method of the class.

Example 1

void M(int o)
{
    int Multiply(int n) => n * 2;

    foreach (var i in new List<int>; { 1, 2, 3, 4, 5 })
    {
        Console.WriteLine(Multiply(i));
    }
}

Multiply will translate to static method in M class:

static int Multiply(int n)
{
    return n*2;
}

OK this is very simple, but what if M class has already method with the same signature (Multiply(int n))?

The answer is that the compiler will generate name that not confuse with existing class method. In this case, the name will be the parent method name and then the method name _Multiply (This is not what will be in reality and there is no commitment to what will be in reality, it is only for the purpose of explanation of how in generally the compiler handle this case.)

Example 2

void M(int o)
{
    int Multiply(int n) => n * 2;

    foreach (var i in new List<int> { 1, 2, 3, 4, 5 })
    {
        Console.WriteLine(Multiply(i));
    }
}
void M(bool o)
{
    int Multiply(int n) =>; n * 2;

    foreach (var i in new List<int> { 1, 2, 3, 4, 5 })
    {
        Console.WriteLine(Multiply(i));
    }
}

Note, there is a two local functions that their method parents called M. For this case, the compiler will add index for the name and the methods names will be _Multiply1, _Multiply2 etc.

Example 3

void M(int o)
{
    int Multiply(int n) => obj == null ? 0 : n * 2;

    foreach (var i in new List<int> { 1, 2, 3, 4, 5 })
    {
        Console.WriteLine(Multiply(i));
    }
}

Note, obj is a class member, so Multiply can’t be static. Indeed in this case the generated method will be an instance method with access to this.

Example 4

void M(int o)
{
    int Multiply(int n) => o * n;

    foreach (var i in new List<int> { 1, 2, 3, 4, 5 })
    {
        Console.WriteLine(Multiply(i));
    }
}

Do you see the problem? we use the o parameter inside the local function, how this can be done? Remember that Multiply is a different method (static or instance) in the class, so how is know about parameter passed to M?

The solution for this (and not just for this, we will see in later posts that C# compiler use it in more cases) is to generate a new nested type named for example ‘ParentClass_DisplayClass_0’ (see below) and this object has member to hold the parameter o.

private struct <>c__DisplayClass0_0
{
    public int o;
}

Now in M method, the struct is created and the parameter is saved and when the local function code is run, this struct is sent as by ref parameter and this how the local function can use it.

Example 5

void M(int o)
{
    int Multiply(int n) => obj == null ? 0 : o * n;

    foreach (var i in new List<int> { 1, 2, 3, 4, 5 })
    {
        Console.WriteLine(Multiply(i));
    }
}

Now we use parameter and instance member. For this ‘ParentClass_DisplayClass_0’ also generated and it holds two fields, one for parameter and the second for this. The local function will generated as static, and not as we saw in example 3, even that it use an instance member, because the generated type is hold for us the this. The rest is same as example 4.

Homework: What do you think will be the behavior if the M method is within a struct and not within a class and inside the local function we want to use obj instance member?

See more C# behind the scenes articles

C# behind the scenes: yield

Wed, 19 Jul 2017 00:00:00 GMT

The first “Behind the scenes” article will be about yield keyword.

Lets take for example a very simple code that return numbers 1,2,3:

public IEnumerable<int> ExampleEnumerable()
{
    var arr = new int[] { 1, 2, 3 };
    foreach (var i in arr)
    {
        yield return i;
    }
}

The compiler translates this 4 lines code into an entire class.

(I will not put here the actually generated code, because it will show unclear to those who do not familiar with generated code. And it would be useless. Instead, I’ll put here code that is close enough to the generated code.)

The ExampleEnumerable method will be like this:

public IEnumerable<int> ExampleEnumerable()
{
    Yield.YeildGeneratedCode yeildExample = new Yield.YeildGeneratedCode(-2);
    yeildExample._this = this;
    return (IEnumerable<int>)yeildExample;
}

And this is the YeildGeneratedCode class: (with explanation comments)

class YeildGeneratedCode : IEnumerable<int>, IEnumerable, IEnumerator<int>, IDisposable, IEnumerator
{
    private int _state; // The state of the enumerator. see explanation later.
    private int _current; // The current item
    private readonly int _initialThreadId; // For thread safety. see GetEnumerator later.
    public Yield _this; // This is the variable that saving the instance of the original code. In this case its Yield object and not YeildGeneratedCode object
    private int[] _collection; //The collection with items to enumerate
    private int _index; // Index of the current enumeration

    int IEnumerator<int>.Current => _current; // Return the current item (last item the has set in the last MoveNext call)

    object IEnumerator.Current => (object)_current;

    public YeildGeneratedCode(int state)
    {
        // Initialize a new Enumerable. The state set to -2, meaning the
        // enumerator hasn't create yet and set the thread id for thread safety
        _state = state;
        _initialThreadId = Environment.CurrentManagedThreadId;
    }

    void IDisposable.Dispose()
    {
        // Use if needed
    }

    bool IEnumerator.MoveNext()
    {
        switch (_state)
        {
            case 0: // Initialized. Change to running and initialize the collection and the index
                _state = -1;
                _collection = new int[] { 1, 2, 3 };
                _index = 0;
                break;
            case 1: // Suspend. Change to running and increase the index
                _state = -1;
                _index++;
                break;
            default: // Other states
                return false;
        }
        if (_index < _collection.Length)
        {
            // If there is a items, set current and change state to suspend
            _current = _collection[_index];
            _state = 1;
            return true;
        }
        _collection = null;
        return false;
    }

    void IEnumerator.Reset()
    {
        // This is not the real implementation,
        // I just show you an example of what you can do here.
        // The regular implementation is to throw NotSupported exception
        _state = 0;
        _index = 0;
    }

    IEnumerator<int> IEnumerable<int>.GetEnumerator()
    {
        YeildGeneratedCode yeildGeneratedCode;
        if (_state == -2 && _initialThreadId == Environment.CurrentManagedThreadId)
        {
            // If the enumerator yet not created and the the
            // original thread is same as the current calling thread,
            // change the state to initialized and return this
            _state = 0;
            yeildGeneratedCode = this;
        }
        else
        {
            // In other case, return a new Enumerator
            yeildGeneratedCode = new YeildGeneratedCode(0);
            yeildGeneratedCode._this = _this;
        }
        return yeildGeneratedCode;
    }

    IEnumerator IEnumerable.GetEnumerator()
    {
        return ((IEnumerable<int>)this).GetEnumerator();
    }
}

When a method return an IEnumerable interface, the method does not return the actual values immediately, but an enumerator block is created and return. When the MoveNext is invoked (via foreach for example) the actual operation done.

Enumerator is a state machine with states: Initialized (before) = 0 Running = -1 Suspend = 1 (can be also a close or after state)

In each MoveNext call, if the state is in initialized or suspend, the state is changing to running and then the next item (if any) will return. (‘running’ and ‘close’ states will not return value).

The Reset method can potentially reset the enumerator to some state but it’s not implemented in yield.

To make the picture complete, lets see the IL code of the original method:

IL_0000: ldc.i4.s -2 // Load the state. 2 for uninitialized yet
IL_0002: newobj instance void Yield.YeildGeneratedCode'::.ctor(int32) // create the generated object
IL_0007: dup
IL_0008: ldarg.0 // load 'this'
IL_0009: stfld class Yield Yield.YeildGeneratedCode'::'<>_this' // store 'this' in this field of the generated class
IL_000e: ret

As you see, in the method we don’t see any loop. Just create the object and return it. If someone later will write foreach(var item in enumerableObject) then the enumerator will be created and MoveNext will be call in each iteration.

One more example of the same code but that return the Ienumerable like this:

public IEnumerable<int> ExampleEnumerable()
{
    yield return 1;
    yield return 2;
    yield return 3;
}

In this case, the switch in the MoveNext will look like this:

switch (_state)
{
  case 0:
    _state = -1;
    _current = 1;
    _state = 1;
    return true;
  case 1:
    _state = -1;
    _current = 2;
    _state = 2;
    return true;
  case 2:
     _state = -1;
     _current = 3;
     _state = 3;
     return true;
  case 3:
     _state = -1;
     return false;
  default:
     return false;
}

Here there is no need to check if there is item to return, all is known from the start, so just return the item in the order of the yield statements in the original method (that will be the cuurent item in each MoveNext call). Note that here the state veritable is a kind of index.

If you want to investigate a more complex examples, try to understand what will happen in this case:

foreach (var i in new int[] { 1, 2 })
{
    foreach (var i1 in new int[] { 1, 2, 3 })
    {
        yield return i1 * i;
    }
}

Have fun until the next post

See more C# behind the scenes articles

DumpMiner: A UI for exploring ClrMD

Thu, 24 Nov 2016 00:00:00 GMT

ClrMDis a library built by Lee Culver. This is a live process and crash dump introspection library. It allows you to write tools and debugger plugins which can do thing similar to SOS and PSSCOR.

I’m not going to write here about ClrMD, there are plenty of explanations on the net, but I want to talk about tool called DumpMiner I built using ClrMD.

When ClrMD came out, maybe three years ago, I started playing with it and I was very impressed. It gives you fairly easily, with managed code, to build tools which analyze dumps or live process (like WinDbg with SOS).

The main drawback of WinDbg for .NET developers is the is the need to remember a plenty of commands and when to use each of them is definitely not an easy task.

So after I wrote a few examples with ClrMD, I decided to build a simple UI tool to demonstrate the operations you can do with ClrMD with easy to use UI, just click the operation and you have what you search for.

I think, beside the use you can do with this tool, is also a good start point to learn about ClrMD.

The tool is written in C# and of course with ClrMD and his UI infrastructure built with MUI (Modern UI for WPF) library.

(I must warn you that I do not an expert UI designer, so forgive me about the look of the tool ;) )

This is not a complete tool, but a small project to work with ClrMD , however he certainly has quite a lot of examples.

Additionally, it was built in a way that you can easily add more ClrMD operations to UI.

Lets see whats inside DumpMiner.

The main classes are:

DebuggerSession - Manage the processes of attach\detach, load dump, symbol path, creating runtime etc.
ClrObject - Represents a live object from dump or process.
IDebuggerOperation - Represents an operation (i.e. dump heap, dump stack, etc.)
BaseOperationViewModel - The base view model of operation

There are more, but these are the most important to know.

Operation may looks like this:

public async Task<IEnumerable<object>> Execute(OperationModel model, CancellationToken token, object customeParameter)
{
List<string> types = model.Types?.Split(';').ToList();
return await DebuggerSession.Instance.ExecuteOperation(() =>
{
var generation = (int)customeParameter;
var heap = DebuggerSession.Instance.Heap;
var results = new List<object>();
foreach (var obj in heap.EnumerateObjectAddresses().Where(ptr => (generation != -1 && generation == heap.GetGeneration(ptr)) || generation == -1))
{
if (token.IsCancellationRequested)
break;

var type = heap.GetObjectType(obj);
if (type == null)
continue;

if (types?.Any(t => type.Name.ToLower().Contains(t.ToLower())) ?? true)
results.Add(new { Address = obj, Type = type.Name, Generation = heap.GetGeneration(obj), Size = type.GetSize(obj) });
}
return results;
});
}

To add a new operation, you need to write the operation itself, and a user control that you want to represent the results.

For all the rest it’s copy paste from another operation. For a simple operation, you even not required to write anything except the operation itself.

The operations already implemented:

DumpClrStack
DumpDelegateMethod
DumpDisassemblyMethod
DumpExceptions
DumpFinalizerQueue
DumpGcHandles
DumpHeap
DumpHeapSegments
DumpHeapStat
DumpLargeObjetcs
DumpMemoryRegions
DumpMethods
DumpModules
DumpObject
DumpSyncBlock
GetObjectRoot
GetObjectSize
TargetProcessInfo

There is more that I want to implement among others improvements and capabilities, but I need to find time for this…

To work with this, the process may be like that:

Dump heap to search for type
Dump heap stat to search for type with a lot of instances or in big size
Maybe dump large object heap to search for something that should not be there
Once you have a type, search for specific one. Dump object to see values, Dump size
Maybe dump methods to search for a specific one and investigate is stack objects
When you have a suspect object get is root to check who keep it a live

The interface (as I wrote, I’m not a great UI designer )

Attach\Detach, Load dump

Dump heap - strings from generation 1

Dump object (string in address 169807932)

BTW, because I use MUI here, you can easily change the UI theme. Check the settings page in the application. For example here is the dark theme version:

I’ll be happy if you try it, and if you love it that great! If something not work, you can contact me for help.

Mono.Cecil: SimplifyMacros and OptimizeMacros

Mon, 14 Nov 2016 00:00:00 GMT

You may have seen it, but what does it actually do? Do I need to add them?

When we write IL macros, we emit instruction after instruction to the method body. Instruction, is operation to do (e.g. math operations, load\store, invocations etc.) and it may have operand. The instruction is 1-byte or 2-bytes length.

Lets look on an example of loading method argument:

ldarg 1

ldarg is a 2-bytes instruction - FE 09 - that takes 2-bytes operand with the argument index to load.

But methods are rarely have more than a few arguments and we don’t really need a 2-bytes operand.

For this we have ldarg.s . meaning, a short form that takes only 1-byte for the argument index.

(actually there is a 1-byte ldarg.0, ldarg.1, ldarg.2, ldarg.3 because they are used very often).

Cecil takes care for this optimization by calling OptimizeMacros``SimplifyMacros in the other hand, is the opposite. You need it when you rewrite an existing method. Because you may change the number of arguments\local variables\methods bytes, and it may affect the short form instruction.

So before you start to rewrite, call SimplifyMacros and when you finish, call OptimizeMacros.

That all for now :)

Mono.Cecil: Generics

Sun, 13 Nov 2016 00:00:00 GMT

Till now we saw an overview on Cecil and two simple examples. Today I want to show how you can handle generics in Cecil.

For the example I’m going to use EqualityComparer<>

// importing the generic type
var eq = mainModule.Import(typeof(EqualityComparer<>));

// importing the type of T we want to instantiate the generic type
var t = mainModule.Import(typeof(object));
var genericEq = new GenericInstanceType(eq);
genericEq.GenericArguments.Add(t);
var importedGenericEq = mainModule.Import(genericEq);

// getting the method we want to call on the generic instance
var defaultMethodDef = importedGenericEq.Resolve().
Methods.Single(m => m.Name == "get_Default");
var methodRef =  mainModule.Import(defaultMethodDef);

// Important - setting the method declaring type to the correct instantiated type
methodRef.DeclaringType = importedGenericEq;

So what we saw here is a regular type importing but with a little additional code, because we need to instantiate the type with the correct T and last, we must to set the declaring type to the correct instance of the generic type.

This is because there is a difference between the TypeDef representation of the class - that there is just a one row for our generic class - and the TypeSpec representation - that has one row for each instantiation. (I’m not going into details of difference between value type and reference type of T and sharing code implementation.)

Mono.Cecil: IAssemblyResolver

Sun, 13 Nov 2016 00:00:00 GMT

One of the important thing you need to know when you start to do real work with Cecil is the IAssemblyResolver

public interface IAssemblyResolver
{
AssemblyDefinition Resolve(AssemblyNameReference name);

AssemblyDefinition Resolve(AssemblyNameReference name, ReaderParameters parameters);

AssemblyDefinition Resolve(string fullName);

AssemblyDefinition Resolve(string fullName, ReaderParameters parameters);
}

The role of this interface, is to handle module resolving.

As you know, in Cecil there is a difference between TypeDef and TypeRef, and every time toy resolve a type - i.e. get a TypeRef - the IAssemblyResolver work to find the assembly that defines the reference.

By default, each module has its own default resolver. This may work in most of the times, but even if it works, potentially you will keep more than one resolver in memory. And beside that, sometimes you need a custom resolver with custom paths to search in so the default will not work.

So the recommendation is to give to assembly\module reader a custom resolver. You can find an example of IAssemblyResolverhere.

Mono.Cecil: Nothing stops you (well, almost)

Fri, 11 Nov 2016 00:00:00 GMT

One of the advantages of Cecil (like Reflection.Emit or writing IL directly) is that it based on the ECMA CLI standard. So the only limitation is if it meets the standard or not. Unlike, C#.

For example, in C#, there is a set of rules of how you can call to variable, type, assembly etc. But these rules is not a CLI rule. So if you need to do something illegal in C# but is legal from the CLI point of view, Cecil is your friend.

(BTW, C# expression evaluator itself generating “illegal” code, for example, when it generate class for lambda closure) .

Here is an example of give an illegal C# name to a method.

void ChangeMethodName()
{
//Before changing the method name, this will print "Start"
var assem = Assembly.LoadFile(@"C:\temp\ClassLibrary1.dll");
Console.WriteLine(
assem.GetType("ClassLibrary1.Class1").
GetMethod("Start", BindingFlags.Static | BindingFlags.Public).
Invoke(null, null));

// Changing the name to illegal name
var module = ModuleDefinition.ReadModule(@"C:\temp\ClassLibrary1.dll");
TypeDefinition class1 =
module.Types.First(type => type.Name == "Class1");
var method = class1.Methods.First(m => m.Name == "Start");
method.Name = "###Start_v1.4.3.0";
module.Write(@"C:\temp\ClassLibrary1_new.dll");

//After changing the method name, this will print "###Start_v1.4.3.0"
assem = Assembly.LoadFile(@"C:\temp\ClassLibrary1_new.dll");
Console.WriteLine(
assem.GetType("ClassLibrary1.Class1").
GetMethod("###Start_v1.4.3.0",
BindingFlags.Static|BindingFlags.Public).
Invoke(null, null));
}

class Class1
{
static string Start()
{
return $"my name is {MethodBase.GetCurrentMethod().Name}";
}
}

Value type methods: call, callvirt, constrained, and hidden boxing

Fri, 11 Nov 2016 00:00:00 GMT

Long time ago I wrote a posthere on call vs callvirt and the needed of the this null check. Here I want to wrote about this topic but on value types.

Look on these three ToString() calls:

struct Struct1
{
public override string ToString() { return "";}
}

struct Struct2{}

class Program
{
static void Main(string[] args)
{
// Case 1
var s1 = new Struct1();
s1.ToString();

// Case 2
var s2 = new Struct2();
s2.ToString();

// Case 3
int i = 5;
i.ToString();
}
}

In each ToString() call, what will be called, call or callvirt?

In case 1 and case 2, callvirt will be generated by the compiler. But this is not the all story. Before callvirt, another instruction will be generated, called constrined. And this instruction cause to different behaviour between case 1 and case 2. And what will happen in case 3?

Lets start by see what is constrainedMSDN

The constrained prefix is permitted only on a callvirt instruction.

The state of the MSIL stack at this point must be as follows:

A managed pointer, ptr, is pushed onto the stack. The type of ptr must be a managed pointer (&) to thisType. Note that this is different from the case of an unprefixed callvirt instruction, which expects a reference of thisType.

And

When a callvirtmethod instruction has been prefixed by constrainedthisType, the instruction is executed as follows:

If thisType is a value type and thisType implements method then ptr is passed unmodified as the ‘this’ pointer to a callmethod instruction, for the implementation of method by thisType.

If thisType is a value type and thisType does not implement method then ptr is dereferenced, boxed, and passed as the ‘this’ pointer to the callvirtmethod instruction.

Case 1:

It’s a value type that implement the method. Roslyn (C# compiler) generating constrained and callvirt but in the end, call is execute.

Case 2:

It’s a value type that doesn’t implement the method. Roslyn generating constrained and callvirt and then ptr is dereferenced and boxed and callvirt is execute.

Here happening what I wrote in title “hidden boxing”. Note that the ToString() call is made on boxed object and not on the one you mean to call it.

Case 3:

Here surprisingly, Roslyn generating call instruction. What the different between int and our custom value types?

The answer is, that in principle, for value type that override the method, call need to be emitted, and when it’s doesn’t override the method, an extra operation need to be done to be able to emit callvirt. Beside that there is another problem for value types that override method and the override has removed or the opposite.

To simplify this, as we saw, Roslyn emit, callvirt with constrained prefix.

But in the last case, Roslyn know that int override the method, and int is a special type (among other system types), so it can assume that override Tostring() is not will change, and emitting a call directly to the right method!

Mono.Cecil: Hello, World

Thu, 10 Nov 2016 00:00:00 GMT

In previous post, I explained what is Cecil, now lets write some code.

This is a minimal program that show how to crate an assembly, add one type, and one method that print “Hello World”.

Before you start, you need to get the latest Mono.Cecil from nuget, create a new console application and add the necessary using statements.

static void Main(string[] args)
{
// Crate a new assenbly
var assembly = AssemblyDefinition.CreateAssembly(
new AssemblyNameDefinition("HelloWorld", new Version()),
"HelloWorld",
ModuleKind.Console);

// This is the main module you need to work with
var module = assembly.MainModule;

// Create a new type called "Program" and add it to main module
var programType = new TypeDefinition(
"HelloWorld",
"Program",
TypeAttributes.Class | TypeAttributes.Public,
module.Import(typeof(object)));

module.Types.Add(programType);

// create a new method called 'Main' method and add it to 'Program' type
var mainMethod = new MethodDefinition(
"Main",
MethodAttributes.Public | MethodAttributes.Static,
module.Import(typeof(void)));

programType.Methods.Add(mainMethod);

// Get ILProcessor for the method body
var ilProcessor = mainMethod.Body.GetILProcessor();

// Load the string "Hello World" to stack
ilProcessor.Emit(OpCodes.Ldstr, "Hello World");

// Call Console.WrtieLine(string)
var writline = module.Import(
typeof(Console).GetMethod("WriteLine", new[] { typeof(string) }));
ilProcessor.Emit(OpCodes.Call, writline);

// Call Console.ReadKey()
var readKey = module.Import(typeof(Console).GetMethod("Read"));
ilProcessor.Emit(OpCodes.Call, readKey);

// You must pop out the return value before you leave the method
ilProcessor.Emit(OpCodes.Pop);

// Return
ilProcessor.Emit(OpCodes.Ret);

// Because this is an executable assembly, you must define an entry point
assembly.EntryPoint = mainMethod;

// Save the assembly to disk
assembly.Write(@"c:\temp\HelloWorld.exe");
}

This code, generate a class equivalent to this class:

namespace HelloWorld
{
public class Program
{
public static void Main()
{
Console.WriteLine("Hello World");
Console.Read();
}
}
}

If you run the output exe, you will see in the console Hello World and the program will wait to input.

I think the comments on the code explain everything, well almost…

What is `Import` method?

Import is the Cecil way to get a reference (type, method etc.). As you know, void and object are not define in out assembly, so we need to import them as a type reference from mscorlib.dll. Import accepting parameter from Cecil types or .Net types. The opposite is Resolve, which is return a definition. We will see more on this in next posts.

Mono.Cecil: An overview

Thu, 10 Nov 2016 00:00:00 GMT

I got the feeling that C# programmers kind of fear of using Mono.Cecil.I think it’s just because they are didn’t try it. Cecil is a powerful tool for rewriting an assembly. And it does it in pretty simple way.

You can use Cecil for various tasks. One of the main usage is, AOP libraries. In general it can replace Reflection.Emit and sometimes ExpressionTrees. Those are also a very powerful tools, but I prefer Cecil.

So what is Cecil?

Cecil is a library written by Jb Evain to generate and inspect programs and libraries in the ECMA CIL format. It has full support for generics, and support some debugging symbol format.

In simple English, with Cecil, you can load existing managed assemblies, browse all the contained types, modify them on the fly and save back to the disk the modified assembly. (The quote taken from mono project website)

As I wrote, it can be Reflection.Emit alternative, but it has two important additional capabilities.

Cecil has support for extracting the CIL bytecodes.
Cecil does not need to load the assembly or have compatible assemblies to introspect the images.

Also at Cecil, in contrast to Emit, there is a difference between Reference and Definition.

You can think of cases where it’s important. I’ll show you these cases in a later post.

**Recommendation:**Cecil has a good wiki page to learn more about it, and a good google group to ask and get help (mostly from Jb Evain himself).

Cecil type system

The most important types you need to know to know, is the following:

AssemblyDefinition
ModuleDefinition
MemberReference
- TypeReference
  - TypeDefinition
- MethodReference
  - MethodDefinition
- Field, Property, Event etc.
MethodBody
ILProcessor

AssemblyDefinition and ModuleDefinition (specially MainModule property in the AssemblyDefinition), used to read\write assembly\module. From this you start to work with Cecil.

MemberReference and his inherited children, is the metadata representation of type, method etc.

MethodBody and specially his ILProcessor, is for rewrite an existing method, or write a new one.

Good to know: Cecil has library, called Cecil.Rocks that has a plenty extension methods that make your work even easier.

How Cecil do its magic?

To understand this, you need to know what is exist in your compiled managed assembly, specially the metadata section. I’m not going to deep into the subject now, only adding a small overview. (For more info I’m very recommending to read Simon Cooper post series, Anatomy of a .NET assembly)

All types, methods etc. saved in the assembly file in binary representation of 45 tables that has rows with the relevant information. To reference a row within a table, there is a something called token. Token is a four bytes value that point to a row index in a specific table.

Example of existing tables:

TypeDef - Store types in the same assembly
TypeRef - Store types in a referenced assembly
TypeSpec - Store instantiations of a generic type

Cecil is a Metadata Reader\Writer. It knows how to read metadata info and how to write it.

In the next post we will see how to write our first HelloWorld program with Cecil.

Making a method static with Roslyn

Mon, 27 Jun 2016 00:00:00 GMT

Before Roslyn days, this was not easy to do that, like others re factoring.

Fortunately we have R# for this.

But now it’s not so hard to re factor an instance method into static method.

I wrote a rewriter that convert any instance method to a static method.

The code is in GitHub

What is the @ sign before a variable name?

Mon, 27 Jun 2016 00:00:00 GMT

You probably saw this at least once

var @delegate = ...

Or especially in generated code (like if you let R# to do some rewriting for you) you can see

var @t = ...

So, what is it? the simple answer is the ‘@’ sign can be added to a reserved word and then you can define for example variable that with the name “@delegate” but you can not define “delegate” variable.

But what special with ‘@’ ? you can add any other characters like “_delegate”?

It’s indeed different because when you add the ‘@’ sign, in the assembly the variable name is saved without the ‘@’ sign, so “@delegate” will be just “delegate”.

It can see meaningless until you will be in situation that it is matter, and this is exactly what happened to me.

I have a code that create a type with property named “@t”, then I try to get the property by name with reflection.

All of this is part of a huge generated code that is very hard to read or debug.

When I got a NRE I figured it out that is because there is no “@t” property, but this is weird because I know the name is “@t”!

With the knowing the above, I get the property with the name excluding the ‘@’ sign and everything works fine.

Converting LINQ query syntax to fluent syntax with Roslyn

Thu, 09 Jun 2016 00:00:00 GMT

In c# you can write LINQ queries in two ways. One is the fluent invocation syntax (extension method) and the other is query comprehension syntax.

Fluent example:

Enumerable.Range(1, 5).
Where(i => i % 2 == 0).
Select(i => i * 2);

Comprehension example:

from i in Enumerable.Range(1, 5)
where i % 2 == 0
select i * 2;

I was needed the ability to take the comprehension syntax and convert it to fluent syntax. So I wrote a rewriter with Roslyn that do exactly this.

It’s not complete implementation (its not do any syntax validation checks), but its good point to start with.

You can find it in GitHub

If you test it and you find a bug, please let me know.

Here you can find a very helpful query expression translation cheat sheet.

A low-level global keyboard hook in C#

Mon, 16 Feb 2015 00:00:00 GMT

I had to implement a mechanism for a global low-level keyboard hook.

After a brief search on the internet, I understood the principle and I went out with a solution in hand.

First I show you how to use it, it’s very simple.

To register a new hook, just write the following:

hookId = GlobalKeyboardHook.Instance.Hook(new List { Key.LeftCtrl, Key.D }, SomeMethod, out message);

And to remove hook registration, write the following:

GlobalKeyboardHook.Instance.UnHook(hookId);

Here is the full code. You can use it as is if you want.

I tried to make the code formatted in the right way but it seems impossible so you can grab it from*gist*

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Runtime.InteropServices;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using System.Windows.Input;

namespace KeyboardUtils
{
///

<summary>
    /// Provide a way to handle a global keybourd hooks
/// This hook is called in the context of the thread that installed it.
/// The call is made by sending a message to the thread that installed the hook.
/// Therefore, the thread that installed the hook must have a message loop.
/// </summary>    public sealed class GlobalKeyboardHook : IDisposable
{
private const int WH_KEYBOARD_LL = 13;
private const int WM_KEYDOWN = 0x0100;
private const int WM_KEYUP = 0x0101;
private LowLevelKeyboardProc _proc;
private readonly IntPtr _hookId = IntPtr.Zero;
private static GlobalKeyboardHook _instance;
private Dictionary<int, KeyValuePair> _hookEvents;
private bool _disposed;
private KeyCombination _pressedKeys;

///

<summary>
        /// Return a singleton instance of
/// </summary>        public static GlobalKeyboardHook Instance
{
get
{
Interlocked.CompareExchange(ref _instance, new GlobalKeyboardHook(), null);
return _instance;
}
}

private GlobalKeyboardHook()
{
_proc = HookCallback;
_hookEvents = new Dictionary<int, KeyValuePair>();
_hookId = SetHook(_proc);
_pressedKeys = new KeyCombination();
}

///

<summary>
        /// Register a keyboard hook event
/// </summary>        /// The short keys. minimum is two keys
/// The action to run when the key ocmbination has pressed
/// Empty if no error occurred otherwise error message
/// True if the action should execute in the background. -Be careful from thread affinity- Default is false
/// An action to run when unsubscribing from keyboard hook. can be null
/// Event id to use when unregister
public int Hook(List keys, Action execute, out string message, bool runAsync = false, Action dispose = null)
{
if (_hookEvents == null)
{
message = "Can't register";
return -1;
}

if (keys == null || execute == null)
{
message = "'keys' and 'execute' can't be null";
return -1;
}

if (keys.Count  Task.Run(() => execute);

_hookEvents[id] = new KeyValuePair(kc, new HookActions(asyncAction ?? execute, dispose));
message = string.Empty;
return id;
}

private bool ValidateKeys(IEnumerable keys)
{
return keys.All(t => IsKeyValid((int)t));
}

private bool IsKeyValid(int key)
{
// 'alt' is sys key and hence is disallowed.
// a - z and shift, ctrl.
return key >= 44 && key = 116 && key <= 119;
}

///

<summary>
        /// Un register a keyboard hook event
/// </summary>        /// event id to remove
/// parameter to pass to dispose method
public void UnHook(int id, object obj = null)
{
if (_hookEvents == null || id < 0 || !_hookEvents.ContainsKey(id)) return;

var hook = _hookEvents[id];

if (hook.Value != null && hook.Value.Dispose != null)
{
try
{
hook.Value.Dispose(obj);
}
catch (Exception)
{
// neet to be define if we need to throw the exception
}
}

_hookEvents.Remove(id);
}

private IntPtr SetHook(LowLevelKeyboardProc proc)
{
using (Process curProcess = Process.GetCurrentProcess())
using (ProcessModule curModule = curProcess.MainModule)
{
return SetWindowsHookEx(WH_KEYBOARD_LL, proc, GetModuleHandle(curModule.ModuleName), 0);
}
}

private delegate IntPtr LowLevelKeyboardProc(int nCode, IntPtr wParam, IntPtr lParam);

private IntPtr HookCallback(int nCode, IntPtr wParam, IntPtr lParam)
{
if (nCode = 2)
{
var keysToAction = _hookEvents.Values.FirstOrDefault(val => val.Key.Equals(_pressedKeys));
if (keysToAction.Value != null)
{
keysToAction.Value.Exceute();
// don't try to get the action again after the execute becasue it may removed already
result = new IntPtr(1);
}
}
}
else if (wParam == (IntPtr)WM_KEYUP)
{
_pressedKeys.Clear();
}

// in case we processed the message, prevent the system from passing the message to the rest of the hook chain
// return result.ToInt32() == 0 ? CallNextHookEx(_hookId, nCode, wParam, lParam) : result;
return CallNextHookEx(_hookId, nCode, wParam, lParam);
}

#region extern
[DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
private static extern IntPtr SetWindowsHookEx(int idHook, LowLevelKeyboardProc lpfn, IntPtr hMod, uint dwThreadId);

[DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
private static extern bool UnhookWindowsHookEx(IntPtr hhk);

[DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
private static extern IntPtr CallNextHookEx(IntPtr hhk, int nCode, IntPtr wParam, IntPtr lParam);

[DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
private static extern IntPtr GetModuleHandle(string lpModuleName);
#endregion

#region IDsiposable
private void Dispose(bool dispose)
{
try
{
if (_disposed)
return;

UnhookWindowsHookEx(_hookId);
if (dispose)
{
_proc = null;
_hookEvents = null;
_pressedKeys = null;
GC.SuppressFinalize(this);
}
_disposed = true;
}
// ReSharper disable once EmptyGeneralCatchClause
catch
{
}
}

public void Dispose()
{
Dispose(true);
}

~GlobalKeyboardHook()
{
Dispose(false);
}
#endregion

private class HookActions
{
public HookActions(Action excetue, Action dispose = null)
{
Exceute = excetue;
Dispose = dispose;
}

public Action Exceute { get; set; }
public Action Dispose { get; set; }

}
private class KeyCombination : IEquatable
{
private readonly bool _canModify;
public KeyCombination(List keys)
{
_keys = keys ?? new List();
}

public KeyCombination()
{
_keys = new List();
_canModify = true;
}

public void Add(Key key)
{
if (_canModify)
{
_keys.Add(key);
}
}

public void Remove(Key key)
{
if (_canModify)
{
_keys.Remove(key);
}
}

public void Clear()
{
if (_canModify)
{
_keys.Clear();
}
}

public int Count { get { return _keys.Count; } }

private readonly List _keys;

public bool Equals(KeyCombination other)
{
return other._keys != null && _keys != null && KeysEqual(other._keys);
}

private bool KeysEqual(List keys)
{
if (keys == null || _keys == null || keys.Count != _keys.Count) return false;
for (int i = 0; i < _keys.Count; i++)
{
if (_keys[i] != keys[i])
return false;
}
return true;
}

public override bool Equals(object obj)
{
if (obj is KeyCombination)
return Equals((KeyCombination)obj);
return false;
}

public override int GetHashCode()
{
if (_keys == null) return 0;

//http://stackoverflow.com/a/263416
//http://stackoverflow.com/a/8094931
//assume keys not going to modify after we use GetHashCode
unchecked
{
int hash = 19;
for (int i = 0; i < _keys.Count; i++)
{
hash = hash * 31 + _keys[i].GetHashCode();
}
return hash;
}
}

public override string ToString()
{
if (_keys == null)
return string.Empty;

var sb = new StringBuilder((_keys.Count - 1) * 4 + 10);
for (int i = 0; i < _keys.Count; i++)
{
if (i <span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span>< _keys.Count - 1)
sb.Append(_keys[i] + " , ");
else
sb.Append(_keys[i]);
}
return sb.ToString();
}
}
}
}

Localization with ResourceManager

Mon, 13 Oct 2014 00:00:00 GMT

Multiple language support is a very important thing to think of it in early software development process to avoid unnecessary work later on when we realize that we need it.

So how is it done? There are several ways to support this and here I will describe one option supported in .NET framework using .resx files In principle, it is possible to insert any text required resx file and then access it in the following way:

Properties.Resources.Cars.Nissan;

So instead of writing hard-coded strings, we will get them from resource. This approach suffers from several drawbacks.

To understand this, lets define some requirements that we want to support them:

We want that localization will be loosely coupled with the app itself. Namely, to support other language or to change resource values (add / remove / change), we do not want to re-compile the application and to re-deploy.
We want the ability to change a particular resource file reference. That is, if to a certain stage, to get the name of a vehicle, we go to Cars resource file, now we want to take values from a file called Vehicles.
We want full control over access to the resource, so that we can add values that tried to get them and they were missing, we do validation on a new value, and the most important thing we can do Logging the missing value.
We want one place to handle with everything. Debugging / maintenance

We can’t use the above approach and support all the above requirements.

To support this, we offer the following solution:

Resource project will be created one for each system. It will create a file for each module / subject or all the selected logical way.

We will create a resource file for each language we want to support it. The files should be marked as Access modifier = public.

We will define configuration file (xml). Each module name will be map to a file resource. So if we decide to go car name from “Vehicles” instead of “Cars”, we need to change only the name.

We will create a class (infrastructure) that will envelope the ResourceManager. it will keep a collection of resources as defined by the user in the configuration file.

In this class we will create a two methods, GetString and GetObject (for non-text resource). We can add value to the resource file in case that does not exist and we can make sure this value is used as a key can actually be used as such. For example: “Some_key” This is the right value. However “some-key” is invalid. And, of course, we can write logs.

The usage will be by sending the key and file name where the resource value should be:

MyResourceManager.GetString (key, CarsModuleResource);

CarsModuleResource will be as defined in the file config.

And from xaml we can call this method via MarkupExtension that will get the key and file name;

The XML config can be like this:

ResourceConfig Name="CarsModule" Path="Cars\Path.resx" Assembly="MyResorceAssembly.dll"

IL call vs. callvirt, part two

Thu, 24 Apr 2014 00:00:00 GMT

In the first part I wrote about what is call and callvirt, what the differences and why should I care. We saw that even non-virtual methods called via callvirt instruction because the nullcheck reason. But with this sense when this is not in use or when thiscan’t be null there is no need to check for *null.*for example: static class. And indeed for those cases the compiler emit call instruction.

Here is the cases that call instruction will emit:

static class methods
struct methods
new MyClass().SampleMethod();

The C# compiler / jit compiler know that it can’t be null so it generating a call instruction

What do you think happening here?

var c = new MyClass();
c.MyMethod();

Is this same as

new MyClass().MyMethod();

Check it yourself with reflectoror with Ildasm :-)

For who that want to know how all of those seem in ‘real life’ (it’s also answering on the above question), given the following code:

C#:

static void Main(string[] args)
{
var a = new myclass();
a.mymethod(args.Length);
}

IL: .method private hidebysig static void Main(string[] args) cil managed { .entrypoint // Code size 17 (0x11) .maxstack 2 .locals init ([0] class sealedvsnon.myclass a) IL_0000: newobj instance void sealedvsnon.myclass::.ctor() IL_0005: stloc.0 IL_0006: ldloc.0 IL_0007: ldarg.0 IL_0008: ldlen IL_0009: conv.i4 IL_000a: callvirt instance int32 sealedvsnon.myclass::mymethod(int32) IL_000f: pop IL_0010: ret } // end of method Program::Main

Disassembly:00CD0099 cmp dword ptr [ecx],ecx 00CD009B call dword ptr ds:[0BE380Ch]

As you can see callvirtand cmphas used.

And with the following code: C#:

static void Main(string[] args)
{
new myclass().mymethod(args.Length);
}

IL:.method private hidebysig static void Main(string[] args) cil managed { .entrypoint // Code size 15 (0xf) .maxstack 8 IL_0000: newobj instance void sealedvsnon.myclass::.ctor() IL_0005: ldarg.0 IL_0006: ldlen IL_0007: conv.i4 IL_0008: call instance int32 sealedvsnon.myclass::mymethod(int32) IL_000d: pop IL_000e: ret } // end of method Program::Main

Disassembly:01220088 mov ecx,dword ptr [ebp-0Ch] 0122008B call dword ptr ds:[11C380Ch]

And here you can see that non callvirt or cmphas used.

IL call vs. callvirt

Tue, 22 Apr 2014 00:00:00 GMT

.NET languages compiles to ILcode (aka msilor cil) The IL code contains instructions that in run time the jitcompiler translate it to machine code.

In this post I want to talk about method call instructions.

When we writing the following in C#

MyClass().MyMethod();

The code will translate to one of the following: callor callvirt. Theoretically there is one more option named callibut c# compiler won’t generate calli instruction so I’ll skip this.

What is the difference and why should I care?

The difference is if the code will translating to direct call to supplied address (call) or to a late-bound call to specific address that we have to find via the object vtable(callvirt).

From MSDN:

Callvirt: The callvirt instruction calls a late-bound method on an object.

Call: The call instruction calls the method indicated by the method descriptor passed with the instruction.

And why should I care? The first reason came up is performance, but actually you don’t need to care about this. First because you can’t do anything about this (except of writing your own CLR) and second because when you write your code you don’t really need to think “how I make it to emit call instead of callvirt” this is not so serious performance problem in .NET section. If you want to tweak these things you need to change language.

So as I see it, the reason is just a curiosity of what going on under the hood and why this happen and not this etc. And for me this is the most important reason. Because certainly knowing what happen under the hood give a lot in way of understanding what you do in your code and give a good debugging abilities,.

Back to topic, I think it’s reasonably to say that “direct call” mean staticmethods, sealed classmethods, because these methods can be knowing at compile time and late-bound call mean virtual methods because these type of methods can’t be known at compile time and need to be determined in run time.

But this is not the true…

Again from MSDN:

Note that a delegate’s Invoke method can be called with either the call or callvirt instruction.

It is valid to call a virtual method using call (rather than callvirt).

So we saw that even virtual methods can be called via call instruction. But more surprising it that a non virtualmethods will be called via callvirt instruction, It was understate if regular non virtual method called via callvirtbecause there is a change to them to being overridden, but for sealedclasses it’s completely don’t make sense. Sealed class methods never been overridden. So why is this behaviour?

The answer is that callvirt do one more thing that we need even if of non virtual methods*.* The thing is a null check.

In disassembly its look like this: cmp dword ptr [ecx], ecx This will raise a NullReferenceException if the pointer in ecxregister is invalid.

Note: If you want to read more about this, read thispost at The Old New Thing. Why we need the null check?

When we call to method on object the this pointer pass as the first argument, this can be a null.

What will be happen without the null check?Case one: We try to use the this and we will get a NullReferenceException . This is an expected behaviour.

Case two: We don’t use this in the method and everything work just great. Some people will say that this is a little wired but after all this is a legal.

Case three: We don’t use this in the method but we call other method and there we try to use this. Then we get a NullReferenceException from the second method and this is more problematic from case two because its a little confusing that we doesn’t get the exception in the first when we use the object instance to call the first method..

So the design decision was to check for null every time we call an instance method. (as Eric Gunnerson explained here)

So to summarize it, methods calls generate a callvirt instruction because we need to make sure that the this pointer isn’t null.

Do I miss something here?

Virtual calls is not just about null check, its a different way to call methods as we see from MSDN. So just because the nullcheck we use vtablefor non virtual methods? Why to search for direct method in vtable…

Fortunately Iv’e founded this mail in mono mailing list that said that indeed we not really do virtual call for non-virtual method but we just relay on the nullcheck of the callvirtinstruction and then we call the method directly.

There is also the constrainedopcode that can change the behavior from callvirt to call. Part two

In the next post I’ll show that not all methods actually use the callvirt instruction and also put some ILexamples to demonstrate it.

Update

In C# 6 there is a new null check operator ‘?’When you use ‘?’ on an non virtual method, C# will generate call instruction because the ‘?’ ensure that is not null by check for null and copy the reference to a local value. See this question in SO

One thing you can do only with a reference type

Tue, 08 Apr 2014 00:00:00 GMT

What in your opinion possible to do only with reference type and no with value type without explicit boxing?

What the fact of derived from object that give all reference type some ability that value types don’t have?

You can say ToString method or GetHashCode or Equals or GetType but you actually know that you can do it also with value type via implicit boxing.

So maybe the ability to pass by reference? Also no. you can use the c# ref or out to pass value type by ref.

What about store value types on the heap? It’s not the programmer business at all where the memory allocated but anyway value types can be in the heap (actually most of your value type memory exist in the heap…)

So what I mean is of course using lock (Monitor). You can do it only with reference type and this is because the SyncBlock. (object header)

The SyncBlock is a four bytes that holds an index into SyncTableEntry table, that table contains several things like ComData, AppDomainID, Hashcode and Monitor.

Every time you do:

lock(myObject);

The SyncBlock structure is initialized and the object SyncBlock get an index to the table that contains the data.

When your thread blocked and wait there is an event handle associate with his SyncBlock and that why the SyncBlock is needed.

Deadlock vs. livelock

Mon, 07 Apr 2014 00:00:00 GMT

when more than one thread involve in our application we need to synchronize our shared data access.

This synchronization can look like a simple thing in the first but if we look close at the details, we will see the problems that we need to be aware of them. I want to talk about two things that can happen when we synchronize data access.

Before i explain what each means, I want to mention that thread can being in either two modes: User-Mode and Kernel-Mode. User-mode means that the OS not involved in the process, in the other side, the kernel-mode means that the OS kernel code managed the process of the thread. Each of the method has his upsides and downsides. In general, the user-mode is more efficient.

Deadlock:

This is the more known problem than the other (livelock). Deadlock including transition to kernel mode (because the use of kernel object) and then sleep. It means that first thread access to a critical section and second thread sleep and wait to wake up when the first thread exit the critical section. The problem occur if the first thread never exit from the critical section (if he wait on a second thread lock for example) and then the second thread will keep sleeping forever. Because we talk about kernel-mode this mean that the second thread does not consume CPU resources.

Livelock:

This is like first example in the sense that two threads will block each other but here the two threads doesn’t sleep. They try again and again to do their job but because non of them wait and let the second to do is work, both of them keeping to consume CPU resources without success to finish their job.

So the two concepts mean that our threads are threads are unable to make further progress and this is bad, the difference is if the threads waste CPU resources - livelock; or not - deadlock.

Rendering XAML as an image for a custom Windows Phone 8 live tile

Mon, 07 Apr 2014 00:00:00 GMT

One of the most beautiful feature of WP is thelive tiles. Live tiles can display simple icon or background and content on it.

Tiles can be used just for beauty purpose but the more useful reason is to inform the user about relevant information.

For example: weather, open missions, new requests etc.

The regular way to create tile is pretty simple, you need to add icon/background image to your project and define some details in the manifest and the tile will display in the home screen (if the user choose to do that).

If you want to update the tile content then in the code you can do like the follow:

ShellTile.ActiveTiles.First().Update(new FlipTileData{Count=1});

But what if we want to do more than that? lets say we have a UI that has some text blocks that has important information and we want to display it in the tile in periodic time.

The template of the standard tile data is a constant, it has content, count and backgrounds and we can’t modify it as we want.

Another example is if we want to display randomly images from a local storage images or from the web but each image is in different size and the tiles size must be one of three defined size.

For the periodic mission we need to use ScheduledAgent and you can read about this here and for the tile itself the idea is to use WriteableBitmap.

Here is a great post that explain in details how to do that and also had source code to download. I not going to copy paste the solution… so read this post and enjoy :-)

One thing to comment is the example use ManualResetEvent to block the scheduler thread from finish before the thread that work on the tile finish his work but you can use Task based programming and use TaskCompilationSource to achieve the same result.

Update: Ariel Ben Horesh sent me this link http://msdn.microsoft.com/en-us/library/windows/apps/dn633914. So now there is a new way to do that.

Showing help for the focused WPF control with F1

Sun, 06 Apr 2014 00:00:00 GMT

We want to be able to display Help text when we focus on some FrameworkElement and pressing F1.

We have two constraints:

the solution must be in MVVM.
The solution need to allow to add more controls easily.

The solution is to use CommandBinding to execute the command and AttachedProperty to set the Help description.

Comment: Here we use ToolTip to show the help, of course this is can do in any other way. Also the way to get the Help description can be do in any other way.

static class HelpManager{

private static Predicate hasDescription;

static HelpManager() {
CommandManager.RegisterClassCommandBinding(
typeof(FrameworkElement),
new CommandBinding(
ApplicationCommands.Help,
new ExecutedRoutedEventHandler(Executed),
new CanExecuteRoutedEventHandler(CanExecute)));

hasDescription = element =>
GetHelpDescription(element) != null &&
Properties.Resources.ResourceManager.
GetString(GetHelpDescription(element)) != null;
}

public static readonly DependencyProperty HelpDescriptionProperty =
DependencyProperty.RegisterAttached(
"HelpDescription",
typeof(string),
typeof(HelpManager));

public static string GetHelpDescription(DependencyObject obj) {
return (string)obj.GetValue(HelpDescriptionProperty);
}

public static void SetHelpDescription(DependencyObject obj, string value) {
obj.SetValue(HelpDescriptionProperty, value);
}

static private void CanExecute(object sender, CanExecuteRoutedEventArgs e){
var element = sender as FrameworkElement;
if (element == null) return;
e.CanExecute = hasDescription(element);
}

static private void Executed(object sender, ExecutedRoutedEventArgs e) {
var element = sender as FrameworkElement;
if (element == null) return;
ShowHelp(element);
}

private static void ShowHelp(FrameworkElement element) {
var tooltip = ((ToolTip)element.ToolTip) ?? new ToolTip();
tooltip.IsOpen = true;
tooltip.StaysOpen = false;
tooltip.Content =
Properties.Resources.ResourceManager.GetString(GetHelpDescription(element));
}

}

How to use?

Normal:

<Window x:Class="Example.MainWindow"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:local="clr-namespace:Example"
Title="MainWindow" Height="350" Width="525">
<Button Height="30" local:HelpManager.HelpDescription="SampleHelp"/>
</Window>

With style setter

<Window x:Class="Example.MainWindow"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:local="clr-namespace:Example"
Title="MainWindow" Height="350" Width="525">
<Window.Resources>

<Setter Property="local:HelpManager.HelpDescription" Value="Help2"/>
</Style>
</Window.Resources>
<TextBox />
</Window>

With external control

<Window x:Class="Example.MainWindow"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:local="clr-namespace:Example"
xmlns:external="clr-namespace:WpfControlLibrary1;assembly=WpfControlLibrary1"
Title="MainWindow" Height="350" Width="525">
<external:UserControl1 local:HelpManager.HelpDescription="ExternalHelp"/>
</Window>

You can use it also in nested control (e.g. button within button)

Dudi Keleti

Writing IL according to the specification

What does that mean to write it according to spec?

Now let’s do it a little more complicated by using constraints

And more complicated when considering optimizations

A bit more with constriants

Summary

How to inspect a MethodSpec signature without code

Why do you ever need it and why without code?

What is a MethodSpec?

What is a MethodSpec signature?

How to use a tool to inspect a MethodSpec signature?

The object header gets complicated

Sync Block

Hash Code

C# behind the scenes

C# behind the scenes: Local functions

C# behind the scenes: yield

DumpMiner: A UI for exploring ClrMD

(I must warn you that I do not an expert UI designer, so forgive me about the look of the tool ;) )

Mono.Cecil: SimplifyMacros and OptimizeMacros

Mono.Cecil: Generics

Mono.Cecil: IAssemblyResolver

Mono.Cecil: Nothing stops you (well, almost)

Value type methods: call, callvirt, constrained, and hidden boxing

Case 1:

Case 2:

Case 3:

Mono.Cecil: Hello, World

What is Import method?

Mono.Cecil: An overview

So what is Cecil?

Cecil type system

How Cecil do its magic?

Making a method static with Roslyn

What is the @ sign before a variable name?

Converting LINQ query syntax to fluent syntax with Roslyn

A low-level global keyboard hook in C#

Localization with ResourceManager

IL call vs. callvirt, part two

Check it yourself with reflectoror with Ildasm :-)

IL call vs. callvirt

One thing you can do only with a reference type

Deadlock vs. livelock

Rendering XAML as an image for a custom Windows Phone 8 live tile

Showing help for the focused WPF control with F1

What is `Import` method?