Thursday, 20 May 2010

A Primer on exposing .Net to COM

Four days ago, I jumped in a time machine, dialled back two millennia, and emerged in the medieval period of programming history, sometime during the reign of King COM. Now, returning, I present to you my rough notes on how you, a citizen of the brave new .Net utopia can communicate with the denizens of the dark ages.

Brace yourself

You’ll want to start by understanding the basic principles on which COM is built. There is a series of articles on CodeProject which form a good introduction. Here are links to Part 1 and Part 2. A post on StackOverflow provides an index to the remainder. Reading those articles, I am, once again, amazed at the ingenuity of the ancients!

Consuming COM in .Net

Consuming COM in .Net - well “it’s trivial” (as my old Maths lecturer used to say, having written a partial differential equation on the blackboard, looking to us expectantly for a solution). And in .Net 4.0 it has just got trivialler thanks to dynamic types and No PIA. You just right click on your project, click  Add Reference…, then select the appropriate library in the COM tab. All the types will then be available to you as if they were regular .Net types

Exporting .Net to COM

Where it gets interesting is when you want to make a .Net type available to COM. Here’s what I’ve learnt.

  • Create yourself a new Class library project. In the AssemblyInfo file, make sure that the ComVisible attribute is given the value false – you can then be selective in what is visible. Also, make sure that a Guid has been created.
  • Start by defining an interface defining the methods that you want your object to expose to COM. Decorate this interface with the attribute ComVisible(true).
  • Implement the interface in your object, and decorate the class with [ComVisible(true)] and [ClassInterface(ClassInterfaceType.None)]
  • Open the project properties, go to the Build tab, and tick Register for COM interop. This will ensure that the the necessary Type Library (.tlb) file gets created in the /bin directory
  • To check how your type will appear to COM, you can browse the tlb file in the Visual Studio Object Browser – but note that this will lock the tlb file, so you won’t be able to rebuild your project whilst it remains open in the browser.
  • I’ve noticed in VS2010 that the tlb file sometimes doesn’t seem to get updated: a Clean and a Rebuild soon fixed that.

An example:

using System;
using System.Runtime.InteropServices;

namespace ComLibrary
{
    [ComVisible(true)]
    public interface IMainType
    {
        int GetInt();

        void StartTiming();

        int StopTiming();
    }

    [ComVisible(true)]
    [ClassInterface(ClassInterfaceType.None)]
    public class MainType : IMainType
    {
        private Stopwatch _stopWatch;

        public int GetInt()
        {
            return 42;
        }

        public void StartTiming()
        {
            _stopWatch = new Stopwatch();
            _stopWatch.Start();
        }

        public int StopTiming()
        {
            return (int)_stopWatch.ElapsedMilliseconds;
        }
    }
}

Using the exported COM object

I last touched C++ about 15 years ago. I’ve never programmed with COM in C++. With that valuation of my advice, here is how I went about calling my COM object from C++.

  • Remember to call CoInitialize(NULL) on each thread from which you call COM objects. Before the thread exits you should also call CoUninitialize()
  • Copy the tlb from the /bin directory of your .Net project into your C++ project. At the top of the file where you want to call your COM object put #import “<name of your tlb file>.tlb”.
  • Define a smart pointer to your type: this takes care of all the AddRef/ReleaseRef stuff which you should have read about in the introductory article I pointed you to: for example
    COMLibrary::IMainTypePtr myType;
  • Create an instance of your object:
    myType.CreateInstance(__uuidof(COMLibrary::MainType));
  • Use it:
    myType->GetInt();
  • During development, you might find that, having updated your tlb file, new or changed members don’t appear on the C++ side. There are two things to try here:
    1. In the Debug folder of your C++ project look for the two files <your typelib name>.tlh and <your typelib name>.tli and delete them. Then rebuild your project.
    2. If the project compiles but you get intellisense errors, try closing down the solution and deleting the intellisense cache files. These are located next to the solution file, and have extension sdf (for VS 2010) or ncb (for VS 2008 and earlier)

Here’s a fuller sample for the C++ side

#include "stdafx.h"
#include <iostream>
#import "ComLibrary.tlb"


int _tmain(int argc, _TCHAR* argv[])
{
    CoInitialize(NULL);

    ComLibrary::IMainTypePtr myType;
    myType.CreateInstance(__uuidof(ComLibrary::MainType));

    
    myType->StartTiming();

    for (long long i=0; i < 1000000; i++)
    {
        myType->GetInt();
    }

    long timeInMilliseconds = myType->StopTiming();

    printf("%d", timeInMilliseconds);
    std::cin.get();

    CoUninitialize();
    return 0;
}

Other notes

  • If you try using your exported .Net type in VBA or VB6 and you get the error “Function or interface marked as restricted, or the function uses an Automation type not supported in Visual Basic”, then check the parameter and return types on the interfaces you are exposing. I got this error when trying to return a .Net long – a 64-bit integer. VBA can’t count that far: it’s longs are only 32-bit.
  • Primitive types appear to translate fairly straight-forwardly between .Net and COM. Things start to get hairier when arrays become involved. On the COM side these become SAFEARRAYS, and look like being a right pain to deal with, somewhat mitigated by the CComSafeArray wrapper class.

Wednesday, 21 April 2010

Quick C# Tip: You can use Collection Initializers without creating a Collection

Thanks to some code I saw on Ayende's blog, I’ve just learnt something about C# 3.0 that I never knew before: you don’t have to be creating a brand new collection in order to use a collection initializer.

Here’s what I mean. Collection initializers let you populate a collection without the noise of repeated “Add(…)” calls. Like this:

var myStrings = new List<string> { "A", "B", "C" };

Or if you are fleshing out a collection that is part of another object, you can do this

var instance = new MyType 
               { 
                    MyStrings = new List<string> { "A", "B", "C" }
               };

What I have only just realised is that if the collection property you are assigning to is read only or already has a value, you can still use a collection initializer - you just miss out the construction of the collection type:

var instance = new MyType { MyStrings = { "A", "B", "C" } };

When I first saw this, I didn’t think it was real C# until I checked it myself in LinqPad. But it’s right there in the C#3.0 Specification.

Thanks Ayende!

Monday, 12 April 2010

Back from a bug hunt [or, Don’t Call Excel on a Background Thread]

I’m just back from a two-day bug hunt: an exhausting experience, but with a successful outcome that I’d like to share in case anybody else comes across this critter.

Photoxpress_8059464As he was testing our Excel Add-in, our Tester, Rich, noticed some rather odd behaviour. Whenever he used the Scan Tables feature,  then subsequently closed Excel, Excel.exe would refuse to vacate its slot in Task Manager until forcibly evicted.

The feature in question had my grubby fingerprints all over it, so I signed up to fix the bug. But where to start?

There are abundant examples of this kind of behaviour when automating Excel from the outside – usually caused by managed code not releasing its references to the COM objects. But my Add-in is inside Excel: any references it holds will be trashed when Excel is shutdown.

What then? I clearly remembered testing for just this kind of problem a couple of weeks back, and I was sure the application closed cleanly then. This gave me my starting point for a spot of Binary-Search debugging.

The Find

I began by grabbing a version of the code as it was at the time I thought the code was bug-free. Running it confirmed that was indeed the case. Then I pulled down a version that lay about half-way between then and now. That one exhibited all the symptoms of infestation. Hah! So the bug must have slipped in some time between the first version I checked and the half-way version. And thus I proceeded, wading through version after version, at each step slashing in half the list of changes in which the bug must have concealed itself.

After a couple of hours searching I narrowed it down to two versions: version 3320 had the bug, version 3319 was bug free. The check-in comment on version 3320 was rather revealing:

“Moved long-running table-scan operation to a background thread, and introduced progress reporting”

Those two words “background thread” were a smoking gun if ever I saw one, but who pulled the trigger? (I did mention that this bug was wanted for murder, right?). Now to adjust my binary-search-magnifying glass to zoom in, not on versions, but on lines of code.

I started by hacking the method where I launched the BackgroundWorker that did the table-scan so that, once again, it did its stuff in the UI thread. Sure enough, the bug went away, Excel closing cleanly on request.

Then, having restored the BackgroundWorker, I chopped its DoWork method in two, stubbing out one half at a time so that I could see which bunch of code was causing the problem. DoWork called out to a couple of other components, and I subject their methods to the same treatment. As method-call after method-call confirmed its innocence I closed in on the culprits: calls to the Excel Object model – specifically, calls to the Excel Object model made from a background thread.

Excel gave no indication that it had a problem being called in this way. When called on the background thread, it did exactly as when called on the foreground thread, with nary an exception. Seems it preferred to sulk, and protest in a more underhand way.

The Fix

Having found the problem, how was I to fix it?

I didn’t want to move all the work back on to the foreground thread: but clearly Excel wouldn’t listen if I talked to it from any other. Enter SynchronizationContext. SynchronizationContext is a handy little class that wraps up the complexities of executing code in another thread’s context. In the olden days of Windows Forms you’d use Control.BeginInvoke or Control.Invoke to do this kind of thing; these days WPF’s Dispatcher provides the same features. SynchronizationContext abstracts either Control or Dispatcher depending on what kind of message loop you’re running, and gives you two methods, Send and Post: Send executes code synchronously, Post asynchronously.

So all I needed to do was add a SynchronizationContext parameter to my two services which do all the talking to Excel (my IoC container kindly absorbed the consequences of this change – no ripple-down effect here).

Most places where I was calling Excel, I was querying some value. Like this, for example:

var result = (string)_application.Run("MyMacro")

Afterwards my change it looked like this:

var result = Query(() => (string)_application.Run("MyMacro"));

...

private TResult Query<TResult>(Func<TResult> query)
{
    return _synchronizationContext.SendQuery(query);
}

SendQuery is an extension method I created for SynchronizationContext that looks like this:

public static SynchronizationContextExtensions
{
    public static TResult SendQuery<TResult>(this SynchronizationContext context, Func<TResult> query)
    {
        var result = default(TResult);
        context.Send(parameter => result = query());
        return result;
    }
}

And that nailed it good and proper.

The Finale

So, Bing, if anybody asks you why their Excel Process doesn’t close after running an Add-in (managed or otherwise), tell them to make sure they are not calling into its object model from a background thread.

Saturday, 23 January 2010

Simulating Snakes and Ladders with a PLINQ Turbo boost

“Daddy, will you play a game with me?”

I’d been left in charge, whilst my wife had some well-earned time to herself. Baby brother had been bounced to sleep, and now big sister was wanting some attention.

“Sure! What shall we play?”

My daughter fished around in the cupboard, then thumped a box on the table. My heart sank. “Snakes and Ladders”, she announced.

Snakes and Ladders

For the benefit of any reader who has the good fortune never to have thus occupied their children, Snakes and Ladders is a game played on 10 x 10 board wherein players take it in turns to move across the board boustrophedonically (that is, left to right then right to left – I’ve been itching to work that word into a blog post), on each go moving the number of spaces indicated by the throw of a die. The winner is the first person to reach the last space on the board. The only modicum of excitement to be found in the game is provided by the eponymous snakes and ladders, which connect certain squares on the board. Land on a square at the foot of a ladder, and you zoom ahead to the square at its top. But beware the squares where snakes lie in wait: land here, and you must slide back down the board to the snake’s tail.

You see then, why I wasn’t thrilled at my daughter’s choice: I don’t expect much from a children’s game, but it is nice to be sure that it will finish before one’s brain goes into power-save mode. And no mathematician would be prepared to give such a guarantee about this game: it is perfectly possible to find yourself within a throw of winning, but the next moment unceremoniously snaking your way back to the beginning, to start all over again.

So as I fixed my grin in place and started rolling the die, I got to wondering: how long till we can play something else? How many turns does a game of Snakes and Ladders last, on average? By the time I’d slid down my first snake, I knew how I could find out: I would set my computer playing snakes and ladders, over and over, and instruct it to count how many turns each game took. Apply a little high-school Maths and Stats to the results, and I’d have my answer.

I will confess the game board received little of my attention from then on, as code began to assemble in the editor in my mind (which has syntax highlighting, but is lacking Intellisense - CommonSense too, my wife would tell you). When I realised that I could use PLINQ to parallelise the code and speed up the simulation, my fingers began itching to get at the keyboard.

Here’s how the code turned out, once I fired up Visual Studio.

Modelling the Game

First, a fluent interface for defining the game board:

var board = new BoardBuilder()
            .Length(100)
            .Ladder().From(1).To(38)
            .Ladder().From(4).To(14)
            .Ladder().From(9).To(31)
            .Snake().From(16).To(6)
            .Ladder().From(21).To(42)
            .Ladder().From(28).To(84)
            .Ladder().From(36).To(44)
            .Snake().From(47).To(26)
            .Snake().From(49).To(11)
   //...
   .Build()

Then the game logic. The GameBoard built by the BoardBuilder consists of a number of Places. Each Place has a reference to the Place following it on the board, and if it is at the foot of a ladder or the head of a snake then it has a reference to the Place at the other end of the link. Most of the logic of the game simulation is contained in the MoveOn method. This method is defined recursively. It is passed the numberOfPlaces that the player should move, and it returns a reference to the Place they end up at, taking into account any snakes or ladders leaving the final square. For simplicity, I’ve decided that players cannot move beyond the last space on the board – they simply get stuck there, whatever they throw.

    class Place
    {
        private Place _ladderTo;
        private Place _chuteTo;
        private Place _nextPlace;

  // ...

        public Place MoveOn(int numberOfPlaces)
        {
            Contract.Requires(numberOfPlaces >= 0, "numberOfPlaces must be positive");

            if (numberOfPlaces > 0)
            {
                if (IsLastSpace)
                {
                    return this;
                }
                else
                {
                    return _nextPlace.MoveOn(numberOfPlaces - 1);
                }
            }

            if (_ladderTo != null)
            {
                return _ladderTo;
            }
            else if (_chuteTo != null)
            {
                return _chuteTo;
            }
            else
            {
                return this;
            }
        }

        // ...
    }
}

Running the Simulation

At last, the simulation:

var gameLengths
    = 1.To(numberOfTrials)
    .Select(trial =>
        {
            var die = new Die();
            return board.
           FirstPlace
           .Unfold(place => place.MoveOn(die.Roll()))
           .TakeWhile(place => !place.IsLastPlace)
           .Count();
        })
    .ToList();

Not much to look at, is it? I start by generating a simple sequence, 1.To(numberOfTrials), and for each item in that sequence I instruct the computer to play a LINQified game of Snakes and ladders, and to count how many moves it takes to get to the end of the board. Each game starts by creating a new Die (would be handy if we could do that in real life – we’re always losing ours). Then, using the Unfold method, seeded with the first place on the board, we generate a sequence of all the spaces landed on during that game. We pull items out of this sequence until we hit the last place on the board, at which point we count how many moves the game took.

Analysing the Results

Of course, we want to do some analysis on the results. LINQ makes it trivial to find the average length of a game:

var averageLength = games.Average();

This turns out to be 35.8 moves.

More interesting is the frequency distribution of game lengths which tells us how often we can expect to play games of a given length.

var frequencyDistribution =
    gameLengths
   .GroupBy(gameLength => gameLength)
   .OrderBy(gameLengthGroup => gameLengthGroup.Key)
   .Select(gameLengthGroup => gameLengthGroup.Key + "," + gameLengthGroup.Count() + "," + gameLengthGroup.Count() / (double)numberOfTrials)
   .ToList();
System.IO.File.WriteAllLines("SnakesAndLadderResult.csv", frequencyDistribution);

gameLengths is the sequence containing the length of each simulated game. This query is counting how many games of each length were encountered in the simulation by grouping together all games of the same length. After ordering the groups, shortest game length first, it creates a string for each group, listing the length, the absolute number of games of that length and the frequency of that length of game. File.WriteAllLines is a neat method that (from .Net 4.0 onwards) takes an IEnumerable<string> and writes each string as a new line to a file, giving us an instant CSV file from which Excel can draw a pretty chart: SnakesAndLadders_FrequenciesOfGameLengths

As you can see at a glance, the shortest game I can hope for takes 7 moves, but that’s pretty unlikely to happen – about once in a thousand games. The most common length of game encountered in the simulation was 19. Now let’s transform this into a cumulative frequency chart (which shows how often games take a given number of moves or fewer): imageThis shows that in at least half of all games I’ll need to throw the dice 28 times or more before getting to the end. Worse, there’s a 10% chance that it’ll be 68 moves or more before I beat those slippery snakes.

The PLINQ Part

Where does PLINQ come into all this? And what is PLINQ anyway? PLINQ is an extension to LINQ, part of the Parallel Tasks library that is shipping as part of .Net 4.0. You know how you splashed out for that quad-core, hyper-threading enabled processor and geeked out over all those little processor utilisation charts in Task Manager – then noticed that your CPU rarely hits more than 12.5% total utilisation? Well PLINQ helps you put those lazy cores to work. MultiCoreTaskManagerWatch closely:

var gamesLengths
    = 1.To(numberOfTrials)
    .AsParallel()
    .Select(trial =>
        {
            var die = new Die();
            return board.
        FirstPlace
        .Unfold(place => place.MoveOn(die.Roll()))
        .TakeWhile(place => !place.IsLastPlace)
        .Count();
        })
    .ToList();

Spot it? Check out line 3. AsParallel(), that’s all you need to add. It takes a standard Linq-to-Objects query, and splits the work across multiple threads. On my bog-standard dual-core home laptop, that change took the time to run 1 million iterations down from 58 seconds to 36. That’s a 1.5x speed-up – by changing one line of code! And the best thing is, the speed-up increases as the number of cores in your machine goes up. On my Core i7 Quad core box at work, the simulation took 7 seconds in its parallelised form – and 9 seconds without the AsParallel magic! Still a 1.3x speed up – but overshadowed by the sheer awesomeness of the Nehalem architecture!

I should point out one subtlety here. You might have been wondering why it was necessary to create a new Die for every game? Why not share a die between games, like you do in the real world? The reason is the multiple threads introduced by the AsParallel() call. The Roll() method on Die calls through to Random.Next() under the covers, and the Next method isn’t thread-safe.

The easiest fix then, is to give each game its own Die. But there’s another thing: with all those threads beavering away simultaneously there’s a good chance that several Die will be created at the same time, each initializing their own instance of Random at the same time. Since Random by default seeds itself from the system clock, this would result in identical games being played – not exactly representative of the real world.

So what I do is have a static instance of Random which I use to seed all the others, taking care to take a lock on it of course:

class Die
{
    private Random _random;
    private static Random _seedGenerator = new Random();

    public Die()
    {
        lock (_seedGenerator)
        {
            _random = new Random(_seedGenerator.Value.Next());
        }
    }

    public int Roll()
    {
        return _random.Next(1, 7);
    }
}

Try it for yourself

As always the complete source code is available for download from MSDN Code Gallery. Try it out, and let me know what you think.