A quick guide to Registration-Free COM in .Net–and how to Unit Test it

A couple of times recently I’ve needed to set up a .Net application to use Registration-Free COM, and each time I’ve had to hunt around to recall the details. Further, just this week I needed to write some unit tests that involve instantiating these un-registered COM objects, and that wasn’t straight forward. So, as much for the benefit of my future self as for you, my loyal reader, I’m going to summarise my know-how in quick blog post before it becomes used-to-know-how.

What is Registration-Free COM?

If you’re still reading, I’ll assume you know all about COM, Microsoft’s ancient technology for enabling components written in different languages to talk to each other (I wrote a little about it here, with some links to introductory articles). You are probably also aware of DLL Hell. That isn’t a place where bad executables are sent when they are terminated. Rather, it was a pain inflicted on developers by the necessity of registering COM components (and other DLLs) in a central place in the OS. Since all components were dumped into the same pool, one application could cause all kinds of hell for others by registering different versions of shared DLLs. The OS doesn’t police this pool, and it certainly doesn’t enforce compatibility, so much unexpected weird and wonderful behaviour was the result.

Starting with Windows XP, it has been possible to more-or-less escape this hell by not registering components in a central location, and instead using Registration-Free COM. This makes it much easier to deploy applications, because you can just copy a bunch of files – RegSvr32 is not involved, and there are no Registry keys to be written. You can be confident that your application will have no impact on others once installed.

It is all done using manifests.

Individual Manifest Files

For each dll, or ocx file (or ax files in my case – I’m working with DirectShow filters) containing COM components you need to create a manifest.

Suppose your dll is called MyCOMComponent.dll. Your manifest file should be called MyCOMComponent.sxs.manifest, and it should contain the following:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">

<assemblyIdentity
    type="win32"
    name="MyCOMComponent.sxs"
    version="1.0.0.0" />

<file name="MyCOMComponent.dll">
    <comClass
        description="MyCOMComponent"
        clsid="{AB12C3D4-567D-4156-802B-40A1387ADE61}"
        threadingModel="Both" />
</file>
</assembly>

Obviously you need to make sure that the clsid inside comClass is correct for your component. If you have more than one COM object in your dll you can add multiple comClass elements. For those not wanting to generate these manifests by hand, a StackOverflow answer lists some tools that might help.

About Deployment

When you deploy your application you should deploy both the dll/ocx/ax file and its manifest into the same directory as your .Net exe/dlls. When developing in Visual Studio, I customise the build process to make sure all these dlls get copied into the correct place for running and debugging the application. I stole the technique for doing this from the way ASP.Net MVC applications manage their dlls.

Put all the dlls and manifests into a folder called _bin_deployableAssemblies alongside the rest of your source code. Then modify your csproj file and add the following Target at the end of it:

<!--
  ============================================================
  CopyBinDeployableAssemblies

  This target copies the contents of ProjectDir\_bin_deployableAssemblies to the bin
  folder, preserving the relative paths
  ============================================================
  -->
<Target Name="CopyBinDeployableAssemblies" Condition="Exists('$(MSBuildProjectDirectory)\_bin_deployableAssemblies')">
  <CreateItem Include="$(MSBuildProjectDirectory)\_bin_deployableAssemblies\**\*.*" Condition="Exists('$(MSBuildProjectDirectory)\_bin_deployableAssemblies')">
    <Output ItemName="_binDeployableAssemblies" TaskParameter="Include" />
  </CreateItem>
  <Copy SourceFiles="@(_binDeployableAssemblies)" DestinationFolder="$(OutDir)\%(RecursiveDir)" SkipUnchangedFiles="true" Retries="$(CopyRetryCount)" RetryDelayMilliseconds="$(CopyRetryDelayMilliseconds)" />
</Target>

To make sure that target is called when you build, update the AfterBuild target (uncomment it first if you’re not currently using it):

 <Target Name="AfterBuild" DependsOnTargets="MyOtherTarget;CopyBinDeployableAssemblies" />

The Application Manifest

Now you need to make sure your application declares its dependencies.

First add an app.manifest file to your project, if you haven’t already got one. To do this in Visual Studio, right click the project, select Add –> New Item … and then choose Application Manifest File. Having added the manifest, you need to ensure it is compiled into your executable. You do this by right-clicking the project, choosing Properties, then going to the Application tab. In the resources section you’ll see a Manifest textbox: make sure your app.manifest file is selected.

Now you need to add a section to the app.manifest file for each dependency.

By default your app.manifest file will probably already have a dependency for the Windows Common Controls. After that (so, nested directly inside the root element) you should add the following for each of the manifest files you created earlier:

<dependency>
  <dependentAssembly>
    <assemblyIdentity
        type="win32"
        name="MyCOMComponent.sxs"
        version="1.0.0.0" />
  </dependentAssembly>
</dependency>

Notice that we drop the “.manifest” off the end of the manifest file name when we refer to it here. The other important thing is that the version number here and the one in the manifest file should exactly match, though I don’t think there’s any reason to change it from 1.0.0.0.

Disabling the Visual Studio Hosting Process

There’s just one more thing to do before you try running your application, and that is to turn off the Visual Studio hosting process. The hosting process apparently helps improve debugging performance, amongst other things (though I’ve not noticed greatly decreased performance with it disabled). The problem is that, when enabled, application executables are not loaded directly- rather, they are loaded by an intermediary executable with a name ending .vshost.exe. The upshot is that the manifest embedded in your exe is ignored, and COM components are not loaded.

Disabling the hosting process is simple:  go to the Debug tab of your project’s Properties and uncheck “Enable the Visual Studio hosting process”

With everything set up, you’ll want to try running your application. If you got everything right first time, everything will go smoothly. If not you might see an error like this:

If you do, check Windows’ Application event log for errors coming from SideBySide. These are usually pretty helpful in telling you which part of your configuration has a problem.

Summary

To re-cap briefly, here are the steps to enabling Registration-Free COM for you application:

1. Create a manifest file for each COM dll
2. Make sure both COM dlls and manifest files are deployed alongside your main executable
3. Add a manifest file to your executable which references each individual manifest file
4. Make sure you turn off the Visual Studio hosting process before debugging

Unit Testing and Registration-Free COM

And now, as promised, a word about running Unit Tests when Registration-Free COM is involved.

If you have a Unit Test which tries to create a Registration-Free COM object you’ll probably get an exception like

Retrieving the COM class factory for component with CLSID {1C123B56-3774-4EE4-A482-512B3AB7CABB} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).

If you don’t get this error, it’s probably because the component is still registered centrally on your machine. Running regsvr32 /u [Path_to_your_dll] will unregister it.

Why do Unit Tests fail, when the application works? It is for the same reason that the Visual Studio hosting process breaks Registration-Free COM: your unit tests are actually being run in a different process (for example, the Resharper.TaskRunner), and the manifest file which you so carefully crafted for your exe is being ignored. Only the manifest on the entry executable is taken into account, and since that’s a generic unit test runner it says nothing about your COM dependencies.

But there’s a workaround. Win32 has some APIs –the Activation Context APIs- which allow you to manually load up a manifest for each thread which needs to create COM components. Spike McLarty has written some code to make these easy to use from .Net, and I’ll show you a technique to incorporate this into your code so that it works correctly whether called from unit tests or not.

Here’s Spike’s code, with a few minor modifications of my own:

/// <remarks>
/// Code from http://www.atalasoft.com/blogs/spikemclarty/february-2012/dynamically-testing-an-activex-control-from-c-and
/// </remarks>
class ActivationContext
{
    static public void UsingManifestDo(string manifest, Action action)
    {
        UnsafeNativeMethods.ACTCTX context = new UnsafeNativeMethods.ACTCTX();
        context.cbSize = Marshal.SizeOf(typeof(UnsafeNativeMethods.ACTCTX));
        if (context.cbSize != 0x20)
        {
            throw new Exception("ACTCTX.cbSize is wrong");
        }
        context.lpSource = manifest;

        IntPtr hActCtx = UnsafeNativeMethods.CreateActCtx(ref context);
        if (hActCtx == (IntPtr)(-1))
        {
            throw new Win32Exception(Marshal.GetLastWin32Error());
        }
        try // with valid hActCtx
        {
            IntPtr cookie = IntPtr.Zero;
            if (!UnsafeNativeMethods.ActivateActCtx(hActCtx, out cookie))
            {
                throw new Win32Exception(Marshal.GetLastWin32Error());
            }
            try // with activated context
            {
                action();
            }
            finally
            {
                UnsafeNativeMethods.DeactivateActCtx(0, cookie);
            }
        }
        finally
        {
            UnsafeNativeMethods.ReleaseActCtx(hActCtx);
        }
    }

    [SuppressUnmanagedCodeSecurity]
    internal static class UnsafeNativeMethods
    {
        // Activation Context API Functions
        [DllImport("Kernel32.dll", SetLastError = true, EntryPoint = "CreateActCtxW")]
        internal extern static IntPtr CreateActCtx(ref ACTCTX actctx);

        [DllImport("Kernel32.dll", SetLastError = true)]
        [return: MarshalAs(UnmanagedType.Bool)]
        internal static extern bool ActivateActCtx(IntPtr hActCtx, out IntPtr lpCookie);

        [DllImport("kernel32.dll", SetLastError = true)]
        [return: MarshalAs(UnmanagedType.Bool)]
        internal static extern bool DeactivateActCtx(int dwFlags, IntPtr lpCookie);

        [DllImport("Kernel32.dll", SetLastError = true)]
        internal static extern void ReleaseActCtx(IntPtr hActCtx);

        // Activation context structure
        [StructLayout(LayoutKind.Sequential, Pack = 4, CharSet = CharSet.Unicode)]
        internal struct ACTCTX
        {
            public Int32 cbSize;
            public UInt32 dwFlags;
            public string lpSource;
            public UInt16 wProcessorArchitecture;
            public UInt16 wLangId;
            public string lpAssemblyDirectory;
            public string lpResourceName;
            public string lpApplicationName;
            public IntPtr hModule;
        }

    }
}

The method UsingManifestDo allows you to run any code of your choosing with an Activation Context loaded from a manifest file. Clearly we only need to invoke this when our code is being called from a Unit Test. But how do we structure code elegantly so that it uses the activation context when necessary, but not otherwise? Here’s my solution:

public static class COMFactory
{
   private static Func<Func<object>, object> _creationWrapper = function => function();

   public static T CreateComObject<T>() where T:new()
   {
       var instance = (T)_creationWrapper(() => new T());
       return instance;
   }

   public static object CreateComObject(Guid guid)
   {
       Type type = Type.GetTypeFromCLSID(guid);
       var instance = _creationWrapper(() => Activator.CreateInstance(type));

       return instance;
   }

   public static void UseManifestForCreation(string manifest)
   {
       _creationWrapper = function =>
           {
               object result = null;
               ActivationContext.UsingManifestDo(manifest, () => result = function());
               return result;
           };
   }
}

Whenever I need to create a COM Object in my production code, I do it by calling COMFactory.CreateCOMObject. By default this will create the COM objects directly, relying on the manifest which is embedded in the executable.

But in my Test project, before running any tests I call COMFactory.UseManifestForCreation and pass in the path to the manifest file. This ensures that the manifest gets loaded up before we try to create any COM objects in the tests.

To avoid duplicating the manifest file, I share the same file between my Test project and main executable project. You can do this right clicking your test project, choosing Add->Existing Item… then app.manifest in your main project. Finally, click the down arrow on the Add split button, and choose Add as Link.

If you’ve got any tips to share on using Registration-Free COM, whether in Unit Tests or just in applications, please do leave a comment.

Data virtualization in Silverlight

When I first went freelance (15 months ago – how time has rocketed by!) I promised myself that, with the reigns of my schedule tightly held in my own fist, I would dedicate more time to blogging. Well! You can all see from my blog archives how that turned out. I have discovered that when you have clients willing to pay for as much time as you can give them, there’s a strange temptation to give them as much time as you’ve got!

However.

Just before I went on vacation last week, I found made the time to write a mini-blizzard of blog posts, though not for Functional Fun. Ayende, my client for the last few months, has published them on his company’s blog. In the first batch, I wrote about some exciting new features I’ve coded up for the user interface of RavenDb, Hibernating Rhino’s Document Database server.

But I think you’ll most enjoy a couple of posts I wrote about data virtualization in Silverlight. The problem is a common one. How do you display huge lists of data items in a client application without pulling data unnecessarily from the server? Often applications fall back on paging – show one page of data, and make the user click a Next button when they want to see more. But in terms of usability, this technique feels pretty neolithic. So I figured out a way of using ListBoxes and DataGrids without needing to load all the data up front, but with properly proportioned scrollbars, slurping data from the server just in time as the user scrolls.

Over on Hibernating Rhino’s blog you’ll find the two posts I wrote about this:

• Data Virtualization, Lazy Loading, Stealth Paging–whatever you want to call it: Here’s how to do it in Silverlight
• Data Virtualization in Silverlight: Digging into VirtualCollection

There’s also a bonus post on how I created a VirtualizingWrapPanel for Silverlight – built so that we could display data in a card view rather than a list view.

And best of all, I published a sample application on GitHub with all the code you need to use this technique in your own applications. For those WPF-aficionados amongst you, you should find that the code translates without much difficulty.

Weak Events in .Net, the easy way

I’ve written before about one kind of memory leak the .Net Garbage Collector cannot protect against: those caused by event handlers keeping objects alive beyond their best-before date. Daniel Grunwald has a very comprehensive article on CodeProject explaining the problem in depth, and giving a number of solutions, some of which I’ve used in the past.

Nowadays, my preferred solution is one made possible by the fabulous Reactive Extensions (Rx) framework.

Suppose, as one example, you want to subscribe to a CollectionChanged event, but don’t want your object to be kept alive by that subscription. You can just do:

collection.ObserveCollectionChanged()
          .SubscribeWeakly(this, (target, eventArgs) => target.HandleEvent(eventArgs));

private void HandleEvent(NotifyCollectionChangedEventArgs item)
{
    Console.WriteLine("Event received by Weak subscription");
}

How it works

Like all remedies for un-dying object problems, the active ingredient in this one is the WeakReference class. It works like this

public static class IObservableExtensions
{
    public static IDisposable SubscribeWeakly<T, TTarget>(this IObservable<T> observable, TTarget target, Action<TTarget, T> onNext) where TTarget : class 
    {
        var reference = new WeakReference(target);

        if (onNext.Target != null)
        {
            throw new ArgumentException("onNext must refer to a static method, or else the subscription will still hold a strong reference to target");
        }

        IDisposable subscription = null;
        subscription = observable.Subscribe(item =>
                                                    {
                                                        var currentTarget = reference.Target as TTarget;
                                                        if (currentTarget != null)
                                                        {
                                                            onNext(currentTarget, item);
                                                        }
                                                        else
                                                        {
                                                            subscription.Dispose();
                                                        }
                                                    });

        return subscription;
    }
}

You can see that we hold the intended recipient of the notifications, target, as a WeakReference, so that if the Garbage Collector wants to sweep it up, this subscription won’t stand in its way. Then we subscribe a lambda function of our own to the observable. When we receive a notification from the observable, we check that target is still alive, and then pass along the message. If we discover that target has died, we mourn briefly, then cancel the subscription.

Notice though, that all our clever use of WeakReferences could be subverted if the onNext delegate refers to an instance method on the target. That delegate would then be smuggling in the implicit this pointer as a strong reference to the target. onNext is itself held strongly by the closure that is created for the lambda function, so the net effect would be that the target is kept alive by the subscription.

All of which explains why we do a check to ensure that onNext.Target is null, hence, that onNext is referring to a static method.

To be clear, this doesn’t mean that you can only handle events using static methods. It just means that when you call SubscribeWeakly, the lambda function you supply as onNext must call instance methods via the reference to the target it is given as a parameter (like I showed in the example above) rather than capturing an implicit this reference.

Observing .Net events using Rx

If you’re going to start using this, you’ll need to know how to turn .Net events into IObservables. Fortunately, the Rx framework includes a magic wand in the shape of Observable.FromEventPattern.

Working with events that have been coded-up post-generics, and thus use EventHandler<TEventArgs> delegates, is easiest. Here’s how you would observe the TaskScheduler.UnobservedTaskException event, for example:

Observable.FromEventPattern<UnobservedTaskExceptionEventArgs>(
                handler => TaskScheduler.UnobservedTaskException += handler,
                handler => TaskScheduler.UnobservedTaskException -= handler);

You simply instruct Rx how to attach and detach the event handler it supplies.

Events defined pre-generics all had to roll their own delegate types, and that makes observing them 1-line-of-code more difficult. Here’s the definition of ObserveCollectionChanged which I used earlier:

public static IObservable<EventPattern<NotifyCollectionChangedEventArgs>> ObserveCollectionChanged(this INotifyCollectionChanged collection)
{
    return Observable.FromEventPattern<NotifyCollectionChangedEventHandler, NotifyCollectionChangedEventArgs>(
        handler => (sender, e) => handler(sender, e),
        handler => collection.CollectionChanged += handler,
        handler => collection.CollectionChanged -= handler);
}

What’s happening here, in the first parameter to FromEventPattern, is that we are adapting an event handler delegate in standard form (i.e EventHandler<NotifyCollectionChangedEventArgs>) given to us by Rx into one appropriate for this specific event. In fact, the compiler is doing all the work, inferring the necessary types of the delegates for us. Then, as before, we show Rx how to hook the adapted handler up to the appropriate event, and how to detach when the subscription is cancelled.

Prove It

To prove this all works as it should, I’ve created a little test:

class Program
{
    static void Main(string[] args)
    {
        var collection = new ObservableCollection<object>();

        var strongSubscriber = new StrongSubscriber();
        strongSubscriber.Subscribe(collection);

        var weakSubscriber = new WeakSubscriber();
        weakSubscriber.Subscribe(collection);

        collection.Add(new object());

        strongSubscriber = null;
        weakSubscriber = null;

        GC.Collect();
        Console.WriteLine("Full collection completed");

        collection.Add(new object());

        Console.Read();
    }

    private class StrongSubscriber
    {
        public void Subscribe(ObservableCollection<object> collection)
        {
            collection.CollectionChanged += delegate { Console.WriteLine("Event Received By Strong Subscription"); };
        }
    }

    private class WeakSubscriber
    {
        public void Subscribe(ObservableCollection<object> collection)
        {
            collection.ObserveCollectionChanged().SubscribeWeakly(this, (target, item) => target.HandleEvent(item));
        }

        private void HandleEvent(EventPattern<NotifyCollectionChangedEventArgs> item)
        {
            Console.WriteLine("Event received by Weak subscription");
        }
    }
}

In the blue corner we have StrongSubscriber who subscribes to an event on a collection the standard way. And in the red corner we have WeakSubscriber who listens to events using SubscribeWeakly. We trigger an event to show that both are paying attention, then we attempt to nuke them, setting their references to null and doing a full garbage collection. Finally, we trigger another event, to see who survived the apocalypse. Here’s the result:

As expected, the StrongSubscription clung on to life, whilst the WeakSubscription dutifully died.

Here – take it!

As with all the code on my blog, feel free to take this and use it in your own projects. I’d love to hear how you get on with it.

Solved: Http Error returned for some route urls but not others in Asp.Net MVC or Web API

Here GoogleBing: remember this one for me so I don’t have to waste another two hours re-solving it.

I’m working on a project using the new Asp.Net Web API and hit a really strange problem yesterday. Every time I called the /search endpoint, I’d get back a Http Error 404.20 – Not Found. All the other controllers were working fine. There were no exceptions being thrown and, even stranger, if I set a break point in the SearchController it was never hit.

I spent two hours trying every which-way to get to the bottom of it.

The light-bulb lit up when my eye latched on to this:

(No, that’s not Visual Studio 11. I just faded all the rest to grey so you could see what I saw).

There in the solution was a folder with the same name as the Web API Controller. A quick test confirmed that any route with the same url as a folder in my solution would give the same error.

The Solution

There’s the obvious solution: rename either your folder or your controller, so that there’s no conflict.

But if you’re rather attached to your Controller name, and you’re loath to change your folder structure, there is another solution. When you register your Routes, set

RouteTable.Routes.RouteExistingFiles = true;

This makes Asp.Net Routing try the routes in your RouteTable first, even if there are files and directories that match them.

This by itself was enough for us, because we don’t have any files on disk that we want to serve up directly. But if you do, add

routes.IgnoreRoute("folder/{*path}");

for each folder that you want to serve up, and everything should by hunky-dory.

Fade-Trimming TextBlocks in Silverlight and WPF

Ayende just published a guest-post I wrote on his company blog:

Using ellipsis is so last year. When did you last see the cool HTML 5 kids writing … when they couldn’t fit all their text in a column? They’ve got this fancy new feature, powered by CSS3, where text that doesn’t quite fit simply fades out as it reaches the edge of the text block. Like this (with thanks to the QuickUI Catalog)

Subtle! It looks much prettier, and it means you can fit in an extra three characters of real text instead of the …s.

Silverlight’s future might be uncertain, but I want to show you that it isn’t ready to roll over and die just yet. We can have those fancy fade-trimming effects too!

As part of the work I’m doing on the UI for RavenFS, I implemented Fade-trimming to make our DataGrids look smarter, and I thought it would be nice to share. So I’ve packaged up all you need into an easy-to-use class called FadeTrimming.cs which works in Silverlight and WPF, and given you two demo projects to show you how to get started.

Read the whole article here

Crunch, the RESTful Accountants (And Introducing the Crunch API Explorer)

One of the things that scared me a little when I thought about starting my own business was the thought of the bookkeeping. I know, I know – I have a Maths degree, so I ought to be able to cope with adding up a few numbers then working out the 20% I owe to the tax man. And the 20% I owe to the VAT lady. And how to take just the right amount of salary to avoid giving anything to the PAYE person. But my arithmetic has always been terrible, and the mathematicians who lectured me, strange to say, rarely used actual numbers. It was all xs and ys and βs and πs. So that didn’t help.

A big factor in persuading me to make the leap into running a company was finding an excellent online accounting service to handle all those troublesome numbers for me. Crunch are a smart bunch of accountants based down in Brighton who have teamed up with an equally smart bunch of developers to create bookkeeping software that is actually quite fun to use (I know – I couldn’t quite believe that when I read it on another freelancer’s blog, but it’s true!). You enter all the numbers in the website as you rake in the profits, or fork out to your suppliers, and at the end of the year, Crunch will put together your accounts and send them off to H.M.R.C. They’ll even submit your VAT returns for you. And handle your payroll, if you should happen to have any minions employees. All for £59.50 a month. It’s great. Sign up here and we’ll both get a £25 Amazon voucher!

And as if all that wasn’t awesome enough, earlier this month they launched an API. It its RESTful, speaking XML, with OAuth authentication. That’s right – a REST API from an accountancy company! I should caution that the first release is limited to dealing with expenses and suppliers, but the dev team plan to add areas according to the priorities indicated by us users.

The Crunch API Explorer

Well, you know me. I couldn’t leave a shiny new toy like that lying on the shelf. So I had a play, and knocked together something that I think you’ll like.

Allow me to introduce the Crunch API Explorer:

It’s a little tool to help you poke and prod the Crunch API. You enter a URL, set the appropriate Http Method, hit Go!, and it will show you the XML that Crunch returns.

You can download it here (it installs using ClickOnce, so it will auto-update when I add new features. If you don’t already have .Net 4.0 on your machine, you should be prompted to install it). To connect to Crunch using the Crunch API Explorer, you’ll need your own API key which you can get by contacting the nice folks at api-dev@crunch.co.uk. Then you can make REST calls to your hearts delight. All the documentation you need about the resource URLs and the structure of the XML for submitting updates can be found here on the Crunch website.

Here are a few of my favourite features:

• XML Syntax colouring when you’re editing requests (this courtesy of AvalonEdit, the open source WPF text editor component that is part of SharpDevelop)
• Makes you confirm any update/delete requests made to the live server (but not to the test server)
• All the source is available on GitHub, so if ever you wanted an example of how to connect to an OAuth API with DotNetOAuth, well – now you have one (see CrunchFacade.cs). It’s all in C#, with some WPF, and of course a topping of MVVM.

There are a couple of things I wanted to add, but didn’t get time for – maybe next week:

• Remembering frequently used resource URIs, and maybe saving template Xml request documents
• Ability to choose a file and insert it into the XML request documents in Base64 format for the APIs that support file upload
• Saving Base64 encoded data in the responses to files

Anything else?

I’d love to hear from anyone who finds this useful. Feature requests are welcome (pull requests even more so). And if you fancy forking this and adapting it to explore other APIs, be my guest.