Friday 17 December 2010

Say hello to Simon Squared, my 3.5 day old WP7 Game

LogoScreenSo I finished my “Build a Windows Phone 7 Game in 3 days” challenge, and in just 3.5 days! Only to find, when I went to the competition website a few hours ago, that they’ve moved the goal posts: the deadline is no longer this coming Sunday, but February 28th 2011.

So, if I can convince myself that it doesn’t go against the spirit of my own challenge, I can now spend a bit more time polishing the game, and adding things like sound effects, tombstoning, leaderboards, etc.

But what does it actually do?

I’ve called it Simon Squared, in homage to Simon, an electronic game popular in the 1980s. I’m too young to remember it, but it was a game that flashed coloured lights in sequence, and you had to remember the sequence and play it back. Simon Squared shows you a tiled shape, and then explodes it before your eyes, twisting the pieces one by one. You have to remember the moves it has made, then put the pieces back together.

Since this is Windows Phone 7, you do this by touching the pieces, dragging left and right, up or down to flip them horizontally or vertically, or dragging diagonally to rotate them. Once they’re in the correct orientation, you double tab them to send them sliding back into place in the puzzle.

Here’s a short video to demonstrate1:

What do you think? Does it look like 3 1/2 days work?


1. I was hoping to have a longer, more polished video, but Windows Live Movie Maker kept blue-screening my computer when I tried to render it.

Day 3.5 of my “Build a Windows Phone 7 game in 3 days” Challenge

Red Gate’s Windows Phone 7 App competition ends in three days time. I’ve set myself the challenge of building an XNA game in 3 unplanned days I have in my schedule. Go read about Day 1, Day 2 and Day 3. Sundown of Day 3 found the game to hard for me to play. This was the result of an extra half day I found scrunched up in my coat pocket:

09:30 Trying to find ways to make the game easier!

09:56 Figured out how to draw translucent, alpha-blended models: set BasicEffect.Alpha to something less than 1, and remember to set GraphicsDevice.BlendState = BlendState.AlphaBlend before rendering. Oh – and objects have to be drawn in reverse order – those at the back first. Now the original puzzle is shown along with exploded pieces – I think I am able to play it now!

10:06 Added a little bump to the front of the tile model, so that you can tell when shapes have been flipped on their backs:

image

10:55 Implemented a more mathematical way of checking if a shape is back in its home orientation: Take the Vector (1,1) and apply the same sequence of X,Y, and Z rotations to it as to the shape, and check if it still equals (1,1).

11:00 Trying to figure out an algorithm for increasing puzzle hardness gradually.

11:34 Started using SpriteBatch to draw text on top of my game, but 3D objects started rendering weirdly. As ever though, Sean Hargreaves had the answer: SpriteBatch changes several of the GraphicsDevice properties which need to be reset the next time round before drawing the 3D stuff.

12:18 Got a scoring system working, with the score being drawn up on screen

13:19 Added a help button, and a help screen on application startup. Now all I need is a logo, and I think I’m done.

Day 3 of my “Build a Windows Phone 7 game in 3 days” Challenge

The deadline for Red Gate’s Windows Phone 7 App competition is three days away. I’ve set myself the challenge of building an XNA game in 3 unplanned days I have in my schedule. Go read about Day 1 and Day 2. This was Day 3:

8:50 Further work on the Csv ContentImporter to make it do more of the work in processing the shape data, saving some time in the game.

9:20 Having sorted out the skin of the game, and given it some rudimentary muscles, I now have to introduce mind, or at least, a Brain Stem: I need game logic.

11:12 Made some changes to my content importer, and it no longer works. Now to learn about debugging the content pipeline! Turns out to be pretty easy, thanks to this. Problem is fixed moments later.

11:15 Actually, it’s not. Pressure is starting to get to me. Finding myself guessing at fixes and hitting F5 to see if it works, then repeating the cycle when it doesn’t. Calm down!

13:00 Got first bit of game logic working – deciding whether a shape is orientated correctly for snapping back to the home position.

13:54 Decided to try using radial layout to arrange the exploded shapes on screen –. Now how do I do that? Found an F.A.Q on polar coordinates, and a few minutes later:

image

14:40 Roped Vinod into creating some levels for the games. He’s taken my Excel level editor spreadsheet to the next level by adding VBA code to generate puzzles automatically.

17:30 Got something looking gamish. Puzzle now knows when it has been completed, and animates off screen. New puzzle animates on. Only problem is, it’s too hard for me to play! Can’t fix that now though – off to the company Christmas dinner.

22:30 First feedback on the game from a friend who volunteered for testing: “The idea is cool, …, very cool!”

I wonder if I can borrow a bit of extra time?

Wednesday 15 December 2010

Day 2 of my “3 days to Build a Windows Phone 7 Game” challenge

The deadline for Red Gate’s Windows Phone 7 App competition is four days away. I’ve set myself the challenge of building an XNA game in 3 unplanned days I have in my schedule. Go read about Day 1. This was Day 2:

10:05 – Caught the lurgy. Not feeling well enough to concentrate on real work, and family poorly too. But I might be able to get a bit of the game done.

10:22 - starting work on better gesture processing. Decided on horizontal drag to flip shape in Y axis, vertical to flip in X axis, and diagonal drag to rotate left or right around the Z axis. Double tap will be used to snap the shape into its final position.

10:37 - my little boy wants to use the dining room table to do some colouring, so I head upstairs to sit on the bed. Optical mouse works surprisingly well resting on the duvet.

11:30 - Completed work on gesture recognition. To get overall drag movement, need to store the first FreeDrag sample, and the last, then process them both when a DragCompleted sample is read.

11:35 - Now to work on Hit Testing my shapes when the user touches them. Apparently the term to google for is "3D Picking". XNA helps out here with the ViewPort.Unproject method. There’s also a sample showing how it's done.

14:40 - Hit testing working nicely now. I was impressed with how quickly it came together following the sample (and borrowing some of its code!).

15:13 - Found out how to change the colour of the material: set the DiffuseColor property of BasicEffect on each mesh in the model before rendering. Rather weirdly, DiffuseColor is a Vector3, but if you want to work with something more normal you can do

effect.DiffuseColor = new Color(255, 0, 0).ToVector3();

15:31 - Got shapes rendering with tiles of different colours:

clip_image001

17:10 Built a “level editor” for defining puzzles. Uses Excel to create a CSV which is then imported using a custom XNA ContentImporter. My content importer converts the lines of CSV data in Level objects, Level being a class I defined. The XNA Content Pipeline then takes care of serializing this into binary form and packaging it with my game. Within the game I can just do

var levels = Content.Load<Level[]>("Levels\\Levels");

and XNA will automatically deserialize it. I have to say, I'm very impressed with the XNA content pipeline, and how easy it is to extend it.

image

Saturday 11 December 2010

Build a Windows Phone Game in 3 days – Day 1

After an unpromising start, Day 1 of my 3 day Build a Windows Phone 7 game challenge ended very satisfactorily. Here's how it unfolded:

9:16 - installed WP7 tools, downloaded XNA book, setup a BitBucket Mercurial repository, and created a Solution. We're off!

9:32 - trying to render my first 3D shape - found a great roadmap to XNA tutorials.

10:07 - Got a 3D spaceship animating on screen following this tutorial

11:12 - thrashing around a bit trying to find a modelling tool to build my own 3D shape. Tried TrueSpace and Autodesk's SoftImage Mod Tool. They all have such arcane and obtuse interfaces - might have spotted a gap in the market here! Found a tutorial on CodeProject about using Blender to create XNA models.

12:10 - at last, my own model bouncing around on screen. Created it using Wings3D - which isn't exactly an intuitive tool, but (to pay it a huge compliment) has the least difficult to use interface of any of the tools I've tried this morning. The results: WP7GamePic1Move over Electronic Arts!

14:16 - started researching how to extract the Mesh from my tile Model, so that I can combine them into Models representing shapes consisting of multiple tiles. Instead found a great insight from Sean Hargreaves: Models are archetypes: you only need one of each kind, and you can then draw it at multiple places with different transformations (I might have called them 3D Stencils, but I think "stencil" has a different meaning in the 3D community). With this guidance, created a Tile class with a reference to the Tile model, plus an Offset vector which can be used to generate a Translate transform to position the Model correctly when it is rendered on behalf of that Tile.

15:16– figured out how to transform shapes consisting of multiple tiles so that they move as a unit. It all boils down to Matrix multiplication. To position a particular Tile, first multiply together the matrices representing its own translations within the shape, then multiply the result by the matrix representing the transforms of the shape as a whole.  Or as one post in the forums very usefully put it:

Matrix Order = Scale x Local Translation x Local Rotation x World Translation x World Rotation

Isn’t Maths magic? Now look what I’ve got:

WP7GamePic2

16:09 – Thanks to the Input.Touch namespace I now have the shape responding to gestures. It will flip and spin at the swipe of a finger! Only trouble is, when the gesture is a Drag, the shape flips, then keeps on flipping. Turns out the problem is that in each call to my Update method I’m only reading one gesture from TouchPanel.ReadGesture, and the other multiple gestures that are generated by that single swipe get queued up -  only to be interpreted by my Update method in subsequent calls as further gestures. From the examples in Charles Petzold’s book I learn that in each call to the Update method I should ReadGesture until IsGestureAvailable returns false

17:05 – feeling rather chuffed. Just implemented a simple Storyboard class that allows me to schedule a series of animations, then play them back at the appropriate point as time progresses. I’ve now got multiple Shapes doing a little dance on screen.

Friday 10 December 2010

Can I build a Windows Phone 7 game in 3 days?

Red Gate Software launched a competition three weeks ago: three prizes of $10,000 to be won for the best Windows Phone 7 application, as judged by panels of developers. Now I’ve always had a hankering to write a game, particularly one involving 3D graphics. Microsoft made me hanker some more when they released the XNA framework a few years back. And, would you believe it, you can use the XNA framework to write 3D games for the Windows Phone 7.

Now it just so happens that I have 3 unplanned days in my schedule between tomorrow morning1 and the competition deadline on December 19th. And my colleague Vinod has had an idea for a memory game with a twist.

Taking stock, what do we have?

So, what do you think? Can I learn the XNA framework, and produce an award winning game in just three days? There’s only one way to find out.

On my marks … Get SetGo!


1. Yes, I am writing this at 1 o’clock in the morning. I couldn’t sleep for excitement. Sad, am I not?

Tuesday 2 November 2010

C# 5.0 and the sample that made me go “wow!”

In the Visual Studio Async CTP there is one sample project called 101 Asyncs. This is a browser displaying 101 code-snippets involving the new C# 5.0 async functionality. As I was flicking through I found the following sample, which I think speaks volumes for the elegance and simplicity of the feature:

public async void AsyncSwitchToCPU()
{
    Console.WriteLine("On the UI thread.");

    // Switch to a thread pool thread:
    await new SynchronizationContext().SwitchTo();  
                
    Console.WriteLine("Starting CPU-intensive work on background thread...");
    int result = DoCpuIntensiveWork();
    Console.WriteLine("Done with CPU-intensive work!");

    // Switch back to UI thread
    await Application.Current.Dispatcher.SwitchTo();                

    Console.WriteLine("Back on the UI thread.  Result is {0}.", result);
}

public int DoCpuIntensiveWork()
{
    // Simulate some CPU-bound work on the background thread:
    Thread.Sleep(5000);
    return 123;
}

Using Ready-made Virtual PCs to try out the Visual Studio Async CTP

The Visual Studio Async CTP containing the preview of C# 5.0 which was all the talk at Microsoft PDC 2010 installs on top of Visual Studio 2010. If, like me, you’re wary of trashing your main development machine, you might like to try out the ready made Visual Studio Virtual Machines that Microsoft provide. They are a pretty hefty download (over 9 GB), but I set my download manager to work on the task just after sitting down at my desk this morning, and it was done by mid-afternoon.

Having a Windows Server 2008 Virtual Machine ready built with Visual Studio 2010 (and 2008), Team Foundation Server 2010, SQL Server 2008 and Microsoft Office is a huge time-saver. Though they are all time-limited versions, you can apparently use your MSDN product keys to activate them into perpetual mode.

They come in three flavours, one for Hyper-V, one for Virtual PC 2007, and one for Windows 7 Virtual PC. If you hop over to Brian Keller’s blog he has a some lists of URLs that you can paste into your download manager to make the download job pretty painless, and then some instructions to get you started. One thing to note, once you’ve booted the machine, is that all the user accounts have password “P2ssw0rd”.

Having booted into the Virtual PC, you will need to install the Silverlight 4 SDK and then, of course, the Visual Studio Async CTP itself. Both are pretty small downloads and quite quick to install.

Have fun!

Monday 1 November 2010

WPF to support hosting of Silverlight controls in vNext

To the great consternation of the blogosphere, Joe Stegman’s planned session on Silverlight quietly disappeared from the Microsoft PDC schedule, and no new announcements were made on that front. Instead we got a pre-recorded session from Rob Relyea entitled WPF Today and Tomorrow.

Microsoft adoption

Early on in his talk Rob reminded us that Microsoft is making the ultimate commitment to WPF in that many of their latest products are built on it, including the Expression Studio suite, Visual Studio, and most recently Web Matrix [see the demo in Rob’s talk]. He also hinted that there are other products on the way which cannot be announced yet.

As part of these efforts across Microsoft a number of controls have been developed which Rob feels would be widely applicable(one of which is WPF equivalent of the Silverlight ChildWindow) and he intends to work with the respective teams to release these components to the web.

Silverlight Integration

One welcome announcement Rob made was that the next version of WPF will support hosting of Silverlight controls [watch his demo]. Today, of course, this is somewhat possible by hosting a Web Browser inside WPF, then navigating it to some Silverlight content. What Rob announced was something deeper than that, in two ways. The first is that they want to enable good communication between the WPF host and the Silverlight components, including sharing data.

The second improvement was part of a bigger announcement: they are planning to solve the airspace issues with hosting embedded Hwnd content [watch]. This was something I didn’t expect – in fact, I thought the airspace issue was a problem here to stay.

To briefly recap what the airspace issue is: in current versions of WPF, whenever you need to embed Win32-based content inside WPF, the embedded content always ends up sitting on top of the WPF stuff. This is because Win32 Hwnds don’t layer well with WPF elements – they can’t share airspace. You also can’t rotate or scale the Hwnds or adjust their opacity like you can with WPF elements.

Rob announced that Microsoft will be making a large investment, and will remove the airspace restrictions. And I can see why it will be a large investment. From what Rob said [watch] it seems that they way they plan to tackle this is by rendering Hwnds off-screen so that their visuals can be composited into the appropriate place in the visual tree, then any keyboard our mouse input intended for the control will have to be redirected from the WPF surface back to the Hwnd. A tough challenge indeed – but with great rewards for anybody looking to port legacy applications.

Amongst other things that Microsoft are considering for WPF vNext, Rob mentioned the following:

  • Proper support for Grouping in ItemsControls when doing UI Virtualization [watch]
  • Improved support for data-binding to collections and properties that get updated in a background thread [watch]. For collections, this will involve adding APIs to allow the data-binding engine to interact with the locking scheme used in the collection.

C# 5.0 – Other News

Compiler as a Service

In the last 10 minutes of his talk on the future of C# 5.0, Anders Hejlsberg gave an update on a project that he first mentioned back at PDC 2008 – Compiler as a Service [watch]. Anders confirmed that what was a prototype back in 2008 is now being worked on in earnest. It has a codename – “Roslyn” (he didn’t actually say this, but you’ll see it appear as a namespace in the code samples, and in one of the Visual Studio tabs), but no ship dates as yet.

As Anders put it, during the compilation process, the compiler builds up a huge amount of knowledge about what your code actually means – and then throws it all away. The idea of Compiler as a Service is give you a Language Object Model encapsulating exactly the same knowledge about your code as the compiler has - right down to the finest semantic details like what an identifier means in a particular context, or precisely how a method call will bind at run time. image

It seems that this Language Object Model will be available in a number of different ways, including to Visual Studio Extensions, which will be able to use it for things like performing Refactorings, or formatting code intelligently.

Anders showed one example of a Visual Studio Extension that could add outline regions to if/else blocks [watch]. Sadly, the extension failed to run in Anders demo, but it still conveyed the simplicity and power of the Language Object Model.imageIn another example he showed how the Language Object Models for VB.Net and C# could be used together to create a Visual Studio extension allowing you to copy C# code and paste it as VB.Net [watch]. “I’m not saying this is super-trivial”, remarked Anders, “I mean, there is a couple of thousand lines of code here. But it’s not a complete compiler at all – it’s relatively modest.”

There is more to come

In answer to a question at the end of the session [watch], Anders confirmed that there will be other features added to C# 5.0 that they’re not ready to talk about yet. He did hint that the kind of areas they are exploring were Immutability, Isolated State and Pure Functions – all features associated with ensuring that programs continue to work correctly when parallelism is introduced. Check out the Programming Languages Panel session if you want to hear Anders talk in a little more detail about this.

Friday 29 October 2010

C# 5.0: The Asynchronous Revolution

imageIn his usual quiet and unpretentious way, Anders Hejlsberg stepped up on stage at Microsoft PDC yesterday – and proceeded to launch a revolution. Two new keywords added to the C# and VB.Net languages was all it took, but the impact of those changes will be felt by programmers for years to come. And not just programmers: unlike other changes made to the .Net languages across the versions, these new additions will produce tangible benefits for users.

The Status Quo

Anders started his presentation by noting a current trend in applications [watch]: they are becoming increasingly connected. We’ve had client-server applications for decades, but we’re now seeing more and more applications that have to call out to a number of different services and co-ordinate their responses. This brings a number of problems. First there’s the latency - the application having to wait as the requests and responses pass back and forth over the network. Then, if we’re not careful, the application appears to hang, unable to repaint the user-interface because it’s too busy waiting for the server to respond. Meanwhile, on the other side of the network, all the server’s threads are tied up, waiting for responses to come back from other servers, so it stops responding to incoming requests – and scalability plummets.

Now we all know what the answer is: asynchronous programming. Rather than tying up our main thread waiting for a response to come back, launch the request on another thread and set up some kind of event handler so that you can be notified when the response comes back, and then finish the work.

We have the answer, but it’s not a pretty one, as anybody who has programmed a serious client-server application in Silverlight will tell you. In Silverlight there is no way of calling the server synchronously: Silverlight forces you to call asynchronously so that the browser hosting your application can remain responsive. In fact, there’s no way of doing something as simple as showing a dialog synchronously: if the way your application continues is dependant on the users’ response, you have to move the continuation logic into an event handler attached to the dialog so that it can be executed once the user clicks their choice of button.

Inside-out Thinking

Programming like this requires you to turn your thinking inside out, because you have to prepare your response (in an event-handler or whatever) before you even make the request. And if in your response you’re going to make another asynchronous call, you have to prepare your response for that even before your first response. Anders gave a small example of that [watch]. Here’s a small method that queries the NetFlix web service. Nice straight-forward code:

image

Make that asynchronous and see what happens:

image

See how it’s turned inside out: in line 4 we prepare our what-happens-next by attaching a lambda to the DownloadStringCompleted event, and the call to download the string is the very last thing we do. Then notice that we can no longer return a value from the method: it isn’t available when the method returns. Instead we have to get the user to pass in an Action that we can call back when the result is ready. So we’re forced into changing all the places where we call the QueryMovies method too: the effects of this change ripple out through the code. As Anders said, “it becomes lambdas all the way down”.

No wonder we only do this stuff when absolutely necessary – and our customers pay a price, getting applications that become unresponsive when busy, and servers that are not as efficient as they could be. All because, for mere mortals at least, asynchronous programming is just too hard.

Not any longer.

Asynchronous Programming for the rest of us

Up on stage, Anders introduced his two new keywords, async and await [watch]. Here’s how the asynchronous version of QueryMovies looks in C# 5.0:

QueryMoveisAsync

It’s identical to our nice straight-forward but sadly synchronous version, with the exception of four small changes (highlighed in orange). The first is the addition of the async keyword in front of the method definition. This tells the compiler that it’s going to have to work some magic on this method and make it return its result asynchronously. This produces the second change: the method now returns a Task<Movie[]> instead of just Movie[] – but notice how the return statement is unaltered – the compiler is taking care of turning the return value into a Task.

In case you’re wondering, Task is a class that was introduced in .Net 4.0 as part of the Task Parallel Library. It simply encapsulates a value that is going to become available at some point in the future (hence, why in some languages the equivalent concept is called a Future). Importantly, Task provides the ContinueWith that allows you to schedule an event handler to be called when that value is available.

The most significant change is the introduction of the await keyword in line 4, and the change to call DownloadStringTaskAsync. DownloadStringTaskAsync is just a version of DownloadString that, instead of returning string, returns Task<string>. Again, this Task<string> is something that will deliver a string sometime in the future. Adding the await keyword in front of the method call allows us to await that future moment, at which point the string will be popped out of its Task wrapper and the method will continue as normal.

But get this: it is not waiting by blocking the current thread. Under the covers the compiler takes everything in the method that comes after the await keyword, everything that depends on the value we’re waiting for, and packages it up into a lambda function – an event handler. This lambda function it attaches to the Task<string> returned by DownloadStringTaskAsync so that it can be run when the string becomes available.

There are very few restrictions on what the bodies of async methods can look like. All the usual control flow operations, if statements, foreach, while, etc. work. Best of all, exception handling works without needing any changes. The compiler takes care of all the twistedness of the asynchronicity, leaving you to think in straight lines.

All this means that, once C# 5.0 is here, there will be no excuses for applications that suffer white-outs while they do some work. And it means some cheap improvements to scalability on the server side. Anders showed an example of a server application that processes some RSS feeds over which he waved the async wand [watch]. The time needed to process the page dropped from about 1.25 seconds to 0.25 seconds – 5 times faster. Here’s a snippet of the code so you can see again how few changes are needed:

RssAsyncExample

This is big news. There are other languages that can already do this kind of thing, F# being one of them. But I don’t know of any other mainstream language that does it, or does it as neatly as this. Anders and team have created an incredibly powerful feature, but exposed it a way that everybody will be able to use.

You can play with this now. Anders announced a Visual Studio Async CTP which contains new versions of the C# and VB.Net compilers, and the necessary supporting APIs in the framework (including Async versions of lots of the existing APIs). It installs on top of Visual Studio 2010, and I gather that it enables Asynchronous methods in both .Net 4.0 and Silverlight.

Further Reading

Thursday 21 October 2010

Microsoft PDC 2010 Speculations

We’re now less than a week away from Microsoft PDC 2010. The session list has just been published so I think it’s time to indulge in some speculation as to this years announcements.

C# 5

For me the highlight of recent PDCs has been discovering what Anders and his sous-chefs have been cooking in the C# kitchen. Of all the changes likely to be announced, it is changes to the C# language that are likely to make the most day-to-day difference to my work. In 2005 it was LINQ, in 2008 it was dynamic programming. What will it be in 2010? Well, Anders has been given a slot, and we already have some clues as to what he might say. At his presentation on the future of C# and VB.Net at PDC 2009, Luca Bolognese trailed some of the likely features, including:

  • Compiler as a Service – the C# compiler completely rewritten in C#, and available to use programmatically, allowing all kinds of interesting meta-programming, including the ability to write refactorings.
  • Better support for immutability built into the language
  • Resumable methods.

It was this last one that interested me most. Luca showed an example involving asynchronous programming. Today we do this by calling a Begin* method, and passing in a callback that will be executed when the asynchronous invocation has finished (and we never, ever, forget to call the corresponding End* method!). This can get quite difficult to wrap your head around if you have a whole chain of methods that need to be invoked asynchronously – say in the callback of the second one you kick off another asynchronous method, passing in a callback that runs a third – and so on.

This style of programming actually has a name – Continuation Passing Style (or CPS) – and, from recent developments on Eric Lippert’s blog it seems they’re planning to add support for CPS to the C# language to make it easier to write this kind of code without blowing a fuse in your brain (in typical Eric Lippert style, he is neither confirming or denying what hypothetical features might or might not hypothetically be added to a hypothetical future version of C#, hypothetically speaking).

No doubt Anders will reveal all.

LINQ

Since it was released in 2007, LINQ has colonised .Net code bases at an incredible pace. Parallel LINQ was another great innovation in .Net 4. And Microsoft aren’t done yet. The Reactive Framework (aka Linq-to-Events) is coming on nicely. And this year Bart de Smet has a session tantalisingly entitled “Linq, Take Two: Realizing the LINQ to Everything Dream”. Linq-to-Everything sounds pretty ambitious, so I’m looking forward to hearing what he has to say.

Silverlight

It looks like we can expect some big announcements for Silverlight this year – check the session list for proof: there’s a note against Joe Stegman’s session on Silverlight saying that the abstract is embargoed until after the PDC keynote.

What would I like to see? Well, one of the biggest pain-points I’ve found in my Silverlight programming is that Continuation-Passing-Style programming is thrust upon you whenever you want to show a dialog, or make a call to the server: because Silverlight runs in a browser, you aren’t allowed to block the UI thread, so every potential blocking operation has to be done asynchronously. It would be really nice if C# 5 does simplify CPS, and if that also applied to Silverlight.

Azure

I’ve already made my PDC prediction for Windows Azure: I’m guessing that Microsoft will announce free, enthusiast-scale hosting for Web Applications in Azure. This is looking even more likely now that Amazon have announced a free entry-level version of their EC2 cloud hosting platform.

Thursday 28th October will reveal all. Stay tuned – once again my company has graciously given me time to follow along with the PDC sessions (and this year they’re being transmitted live, as well as being available for download within 24 hours). As on previous occasions, I’m hoping to find time to share with you all what I learn.

Wednesday 20 October 2010

Quick Fix: Silverlight Application showing as a blank screen

I just diagnosed and fixed a problem with my Silverlight application that I’ve hit often enough to make me think it’s worth sharing my approach.

The problem was a simple one. When I deployed my application, and browsed to a page that should have displayed a Silverlight control, all I saw was a blank page.

So I fired up the diagnoser of all problems Http, the excellent Fiddler, and reloaded my page. This is what I saw:

FiddlerMissingFile

In glaring red, Fiddler shows that my Silverlight control made a request to download one of its dependencies (which are all packaged in zip files and stored in the ClientBin folder), and was told it was not available – a HTTP 404 error.

So the fix was obvious: just add the missing file to my installer. Sorted!

Thursday 7 October 2010

Getting started with Protocol Buffers – and introducing Protocol Buffers Workbench

So I’ve fallen out of love with Xml, and decided that Google Protocol Buffers is a better serialization format for a key (and voluminous) data structure in our application (as has Twitter, incidentally). For various reasons, I decided to write my own Reader and Writer rather than go with the existing implementations. The first thing I needed to do then, was to understand how the Protocol Buffers encoding works.

Google have published some very good documentation, both for the message definition language, and the encoding, but nothing beats seeing it in action. For that there’s a Protocol Buffers compiler, protoc.exe which, amongst other things, will handle encoding and decoding messages (get it from the Protocol Buffers website - choose “Protocol Buffers compiler – Windows Binary”). The only thing is, protoc.exe is a command-line tool, and not very friendly if you’re just wanting to fiddle around. So I’ve summoned all my WPF powers, and created a graphical wrapper around it. Allow me to introduce Protocol Buffers Workbench:

image

In brief

Using the Workbench is easy:

  1. Install it using the ClickOnce installer.
  2. Put your message definition in the first column, and, from the drop-down pick the message that will be at the root of your document
  3. Enter your message in the second column, then hit the encode button (“>”). The encoded message is shown in the third column.

Or, if you have an encoded message, you can paste it into the third column and hit decode (“<”) to have the message translated into text format. The Paste button above the third column is particularly useful if you want to decode protocol buffers messages stored in binary columns in Sql Server. If you view such columns in Sql Management Studio they are displayed like “0x0A1B2C4D…”: the Paste button will recognise text in the clipboard in that format and paste it as binary data.

If you’re wondering about the “C#” toggle beneath the binary editor: that will encode the binary data as a C# byte array – particularly useful if you’re wanting to seed your unit tests with authentic protocol buffer bytes!

And that’s it. Unless you want to know more about defining message types, and writing messages in text message format – in which case, read on.

Message Types

The first thing to understand are the message types. Message types are a bit like an Xml Schema. But whereas you can understand Xml just fine without a Schema, having the message types to hand is essential when parsing a Protocol Buffers message. That's because, in a bid to squeeze the output as small as possible, all the wordy field identifiers and even field type information is stripped out. All that's left for each piece of data is a field number, and a few bits telling you how big it is.

Message types look like this:

enum Binding { 
   PAPERBACK = 1;
   HARDBACK = 2;
} 
 
message Author { 
   required string name = 1; 
   optional string location = 2; 
} 
 
message Book { 
   required string title = 1; 
   required Binding binding = 2; 
   repeated Author author = 3; 
   optional int32 number_of_pages = 4; 
   optional double weight = 5; 
}

So for each field in a message, you define if it must be present ("required"), it may be present ("optional"), or if it can appear any number of times, including never ("repeated"). You then give its type - and you have a range to choose from, including enums and your own custom message types. Finally, you assign it a field number: this is the critical part, as this is all that will distinguish between the different pieces of data in the resulting messages.

Messages in Text Message Format

It took me a bit of puzzling to work out how to write messages in text format. For some reason, though Google have defined a text format for Protocol Buffer Messages, they don't appear to have documented it any where.

Here's an example, using the message types defined above:

title: "The Holy Bible"
binding: HARDBACK
author: {
  name: "Moses"
  location: "Sinai Peninsula"
}
author: {
   name: "Paul"
   location: "Tarsus"
}
author: {
   name: "Luke"
}
number_of_pages: 1723
weight: 0.225

As you can see, it has some similarities to JSON, but not many. Field names aren’t enclosed in strings, for example, and they aren’t terminated by commas.

Tuesday 21 September 2010

Move over Xml – Make Way for Protocol Buffers

In loveIt was in the summer of 2001 that I fell in love with Xml. I was writing an Excel Add-in, and needed to store some configuration data to disk. VBA grudgingly offered me Open, Print, Input, and the like to do the job: “Nested fields, huh? Do it yerself!”. Then along came Xml with her DOMDocument class, elements and attributes. “So you’d like nested fields? Sure – just put one element inside another”.

I was smitten.I would run my program, then call colleagues over to look at the output in notepad. “Look, it’s human-readable!”, I’d say, and so obvious was my infatuation that my kind-hearted co-workers never asked, “Yes – but what kind of human would bother?”

Fast forward 9 years, and I’m over that crush. Xml and I are now just good friends. I still see her every couple of weeks, but now I have eyes for other message formats. Then I saw only pros. Now I can concede the cons.

Way back in 2004 Dare Obasanjo, on the Xml team at Microsoft, wrote the Xml Litmus test to help developers check that they’re using Xml appropriately. His criteria:

  1. There is a need to interoperate across multiple software platforms.
  2. One or more of the off-the-shelf tools for dealing with XML can be leveraged when producing or consuming the data.
  3. Parsing performance is not critical.
  4. The content is not primarily binary content, such as a music or image file.
  5. The content does not contain control characters that aren't carriage return, line feed, or tab because they are illegal in XML.

Dip this litmus strip in your scenario, and see how may of those five points turn green. If you score less than 4 out of 5, then Dare suggests you look elsewhere for your serialization format.

In the application I’m currently working on, I fell into the trap of using Xml just because it’s there. I have a data structure that I need to store in a single column in the database, and I picked Xml as the format. Problem is, most of what our application does revolves around this data structure, and it ends up writing, storing and parsing an awful lot of it.

As our databases ballooned in size, it has become clear to me that Dare needs to add another point to his list:

  1. Document size is not critical

There’s no getting away from it: Xml is verbose. And since my application is the only one that’s going to be using this data, and I don’t need to query it with XPath or validate it with Xml Schema or transform it with XSLT, and I do need to get at it fast, I’ve decided verbosity isn’t a price I’m willing to pay any longer. Xml, with regret, you’re fired! (at least for this part of the application).

Xml Alternatives

So what does one use when one does not want to use Xml? Being with Xml for so long has made me wary of proprietary binary formats, so I don’t want to go down the route of inventing my own.  I’m loathe to bring into the world further chunks of bytes that can only be interpreted with source code to hand.

So I’ve been investigating the state of the art.

An obvious first candidate was Binary Xml. But it has a major black mark against it in that that nobody can agree what Binary Xml should look like. Despite years of committee meetings, there is still no officially sanctioned or widely adopted Binary Xml specification. Lots of organisations have had a shot at it, including Microsoft who in invented their own with the specific needs of WCF in mind. Nicholas Allen, a WCF architect, has the details for that on his blog. 

A Bison!      JSON is the current darling of the Web 2.0 community, and BSON is its binary equivalent. James Newton-King has created a .Net implementation. I didn’t choose BSON for one simple reason: like JSON and Xml, it is self-describing with field names embedded in the message. This is a good thing if you want interoperability, but costs on compactness.

The last two widely adopted formats I found are Thrift and Protocol Buffers. These are both quite similar in their aims and accomplishments (not surprisingly, I understand, because Thrift was  developed by engineers at Facebook who liked Protocol Buffers when they’d worked with it while at Google and so reinvented it). Having established that much, and determined that Protocol Buffers is twice as popular as Thrift (currently 153 questions for Protocol Buffers vs 75 for Thrift on StackOverflow, so don’t argue!) I decided to dig into Protocol Buffers.

I like it.

There’s a little language you use for describing the structure of your messages – and of course, nested messages are supported. Then there’s a compiler that turns your message definitions into code in your language of choice – code for serializing and deserializing binary messages into objects matching your definitions.

There’s very clear documentation of the language and the encoding on the Protocol Buffers website. And if you want to try it out in .Net, there’s not one, but two rival implementations. In the red corner we have Protobuf.net by Marc Gravell, and in the blue corner DotNet-ProtoBufs, written by Jon Skeet (in a couple of idle minutes he had when Stack Overflow was down, no doubt).

I’ve spent the last week or so implementing enough of the Protocol Buffers spec for my needs, and I’ve got some very pleasing results. Storing my data in a Protocol Buffers format uses 45% of the space that SQL Server’s Xml data type requires, and it loads and deserializes in less than half the time.

Stick around. I’ve got a few Protocol Buffers tips to share.

Wednesday 25 August 2010

if (DateTime.Now.Month == 8 && DateTime.Now.Day == 25) { age++; }

It’s OK! I’m not old yet – 1 year to go till the big Three O.

I have to share with you the birthday card my wife drew for me. It depicts an essential tool of my tradeSeagull Surfer 2010 (aka 2008 R2!)

Seagull Surfer

Wednesday 4 August 2010

Practical Linq #6: Anonymous types as Parameter Dictionaries (with applications to Ado.Net and string formatting)

I recently had cause to venture out from the cosy comforts of Linq-to-Sql into the scary wilds of raw ADO.Net. But I’m pleased to tell you I found a few tools in my C# 3 backpack that made the experience more tolerable than I feared.

Take a look at this:

public bool Exists(string tableName)
{
    return ExecuteScalar<int>(
        "SELECT COUNT(*) FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = @tableName",
        new { tableName }) > 0;
}

The details of the query are irrelevant. What’s interesting is the way I’m using an anonymous type to pass in its parameters. I’m sad to say I’m by no means the first to think of the anonymous-type-as-parameter-dictionary idea – ASP.Net MVC was probably the first place I saw it: but this application is particularly neat in the way the anonymous type appears to capture the local variable on behalf of the query.

So how does it work?

Simple. It uses Reflection to grab the names and values of each property on the anonymous type, and then adds parameters to a SqlCommand accordingly.

Hey. Don’t run away: it only uses Reflection the first time round. Then it uses Expression-tree magic to compile delegates for each of the property accesses so that subsequent calls cost barely anything.

Intrigued?

All aboard the Reflection Express

The crux of the matter is this method here:

private static Func<object, object> CreatePropertyGetter(PropertyInfo propertyInfo)
{
    var instanceParameter = Expression.Parameter(typeof(object), "instance");

    var expression = Expression.Lambda<Func<object, object>>(
        Expression.ConvertChecked(
            Expression.MakeMemberAccess(
                Expression.ConvertChecked(instanceParameter, propertyInfo.DeclaringType),
                propertyInfo),
            typeof(object)),
        instanceParameter);

    var compiledExpression = expression.Compile();

    return compiledExpression;
}

Given a PropertyInfo representing the property we’re interested in, this method creates a delegate that retrieves the value of the property from an instance of the type. If, for example, MyType has AProperty, then all that Expression malarkey amounts to this:

Func<object, object> compiledExpression = instance => (object)(((MyType)instance).AProperty)

(If you’ve not encountered an Expression tree before, think of it as CodeDOM on a diet. Expression trees are data structures representing a fragment of a program. The LambdaExpression is the most interesting specimen, because, as you see, it allows you to compile the expression into a delegate. And whereas CodeDOM fires up a full-blown compiler and creates an Assembly (with all the overhead that entails), LambdaExpressions are compiled into DynamicMethods using Reflection.Emit. All in all, Expression trees allow you to create little programs on the fly in a very light-weight and performant fashion.)

With that in place, we can now take a Type and create a list of assessors for all its public properties:

var type = instance.GetType();

var propertyDetails = (from propertyInfo in type.GetProperties()
                   where propertyInfo.CanRead
                         && propertyInfo.GetGetMethod().GetParameters().Length == 0
                   select new PropertyDetails
                              {
                                  Name = propertyInfo.Name,
                                  GetValue = CreatePropertyGetter(propertyInfo)
                              })
.ToList();

(Before I put lines 4 and 5 in there I was getting rather funky VerificationExceptions when I tried this on certain objects: .Net slapped my wrist and told me the “Operation could destabilize the runtime”. I apologized and sat humbly on the naughty step for a minute until I remembered that Indexers are properties too, and they take parameters, which the delegates made by CreatePropertyGetter don’t allow for.)

Since Types in .Net don’t grow new properties during the execution of a program, we only need to do this reflection once, and we can then cache the result. That’s exactly what I’ve done in the version of the code shown at the bottom of the post, where I’ve packaged all of this into a handy little extension method, GetPropertyValues.

Finally it’s just a case of running over the object, using each of the getters in turn:

IEnumerable<KeyValuePair<string, object>> EnumeratePropertyValues(object instance, IList<PropertyDetails> propertyDetails)
{
    foreach (var property in propertyDetails)
    {
        yield return new KeyValuePair<string, object>(
                                     property.Name, 
                                     property.GetValue(instance));
    }
}

Applications

Returning to my introductory Ado.Net example, here’s how you make use of this:

public TResult ExecuteScalar<TResult>(string commandText, object parametersObject)
{
    using (var connection = new SqlConnection(ConnectionString))
    {
        connection.Open();
        var command = new SqlCommand(commandText, connection);

        foreach (var propertyValue in parametersObject.GetPropertyValues())
        {
            command.Parameters.AddWithValue("@" + propertyValue.Key, propertyValue.Value);
        }

        return (TResult)command.ExecuteScalar();
    }
}

Another example. How often do you get in a muddle with string formatting, trying to keep your values matched up with the {0}s, {1}s and {2}s in the format string? Wouldn’t this be simpler:

var name = "Bob Smith";
var age = 37;
var town = "Bognor Regis";

var result = "{name} is {age} years old and lives in {town}. He likes reading {book}."
             .FormatWithParameters(new { age, name, book = “Winnie-the-Pooh”, town });

As you see FormatWithParameters doesn’t care what order the parameters arrive in.

With GetPropertyValues to hand, it’s this easy:

public static string FormatWithParameters(this string value, object parametersObject)
{
    var formattedString = value;

    foreach (var propertyValue in parametersObject.GetPropertyValues())
    {
        formattedString = formattedString.Replace(
            "{" + propertyValue.Key + "}",
            propertyValue.Value.ToString());
    }

    return formattedString;
}

Ready to Copy-and-Paste

Here’s the full code to the GetPropertyValues extension method:

public static class ObjectExtensions
{
    private static readonly Dictionary<Type, IList<PropertyDetails>> PropertyDetailsCache = new Dictionary<Type, IList<PropertyDetails>>();

    public static IEnumerable<KeyValuePair<string, object>> GetPropertyValues(this object instance)
    {
        if (instance == null)
        {
            throw new ArgumentNullException("instance");
        }

        var propertyDetails = GetPropertyDetails(instance.GetType());
        return EnumeratePropertyValues(instance, propertyDetails);
    }

    private static IEnumerable<KeyValuePair<string, object>> EnumeratePropertyValues(object instance, IList<PropertyDetails> propertyDetails)
    {
        foreach (var property in propertyDetails)
        {
            yield return new KeyValuePair<string, object>(property.Name, property.GetValue(instance));
        }
    }

    private static IList<PropertyDetails> GetPropertyDetails(Type type)
    {
        lock (PropertyDetailsCache)
        {
            if (!PropertyDetailsCache.ContainsKey(type))
            {
                var details = CreatePropertyDetailsList(type);
                PropertyDetailsCache.Add(type, details);
            }

            return PropertyDetailsCache[type];
        }
    }

    private static List<PropertyDetails> CreatePropertyDetailsList(Type type)
    {
        var propertyDetails = (from propertyInfo in type.GetProperties()
                               where propertyInfo.CanRead
                                     && propertyInfo.GetGetMethod().GetParameters().Length == 0
                               select new PropertyDetails
                                          {
                                              Name = propertyInfo.Name,
                                              GetValue = CreatePropertyGetter(propertyInfo)
                                          })
            .ToList();

        return propertyDetails;
    }

    private static Func<object, object> CreatePropertyGetter(PropertyInfo propertyInfo)
    {
        var instanceParameter = Expression.Parameter(typeof(object), "instance");

        var expression = Expression.Lambda<Func<object, object>>(
            Expression.ConvertChecked(
                Expression.MakeMemberAccess(
                    Expression.ConvertChecked(instanceParameter, propertyInfo.DeclaringType),
                    propertyInfo),
                typeof(object)),
            instanceParameter);

        var compiledExpression = expression.Compile();

        return compiledExpression;
    }

    private class PropertyDetails
    {
        public string Name { get; set; }
        public Func<object, object> GetValue { get; set; }
    }
}

Friday 23 July 2010

Need Virtual Server hosting in the UK or USA? Try ElasticHosts

If you’re looking to host Virtual Servers in the cloud with good response times from the Europe or the USA, you should take a look at ElasticHosts. Here’s why:

  • By my calculations they are just about the cheapest of the options available in the UK. I compared them against RackSpace, CloudHosts (part of UKFast), Amazon and FlexiScale. Only FlexiScale came close.
  • Incredible flexibility in sizing your server. Forget about Small, Large and Extra Large: with ElasticHosts you get a slider control for CPU, Memory and Hardisk allowing you to resize your machine as your needs vary.
  • You can either choose pay-as-you-go pricing, with no minimum commitment; or if demand is predictable you can take out monthly subscriptions and save money. You can even mix and match: if say, you typically use 5GB bandwidth in a month, but sometimes burst up to 7GB, you can subscribe for 5GB a month, and use pay-as-you-go pricing for the extra.
  • They have a REST API giving complete programmatic control over your pool of servers, allowing you to do everything from stopping and starting servers, through provisioning completely new servers on the fly, to copying drives from the cloud to your machine.
  • They offer a free, no commitment, five day trial to get you hooked started.

I’ve saved the best till last: ElasticHosts Customer Service is amazing. Fronted by Elastigirl herself (aka Anna Griffiths, her cover as Helen Parr now being thoroughly blown) they’ve been quick to respond whether I’ve called or emailed.

The first time I got in touch, I was rather cheeky: the server they offer for the free trial is clocked at 2GHz with 1GB RAM. This rather struggled as I loaded it with Windows Server 2008, so I called asking if there was anything they could do to make the trial server more representative of the system we might be looking to subscribe to.

“Sure”, said Anna, “What would you like?”

I was stunned! And continued to be so as she upped the memory, hard disk capacity, bandwidth availability and gave me the use of a static IP address.

If I had one suggestion to make, it’s that they set up a support forum to supplement their email and telephone support. I’m sure they’ve already answered several of the questions I asked many times over, and I always prefer to bother Google before I bother a support person. Added to that, it would be nice to have a place to share some of the titbits that I discovered – like how to call the ElasticHosts REST API from .Net (you have to force WebRequest to include the Basic Authentication header).

So give ElasticHosts a try. I’m sure you’ll be pleasantly surprised.

Wednesday 21 July 2010

Debugging Windows batch files when used as scheduled tasks

Here’s a little gem that I pieced together thanks to Google this morning: how to trouble-shoot a Windows bat file when using it as a scheduled task.

We’re using SQL Express, and we want to make sure all our databases are safely backed up on a regular schedule. One thing that Microsoft cut out of SQL Server when pruning it to create the free version is SQL Agent, the tool that enables you to run scheduled tasks against the database.

No big deal: following Greg Robidoux’s advice I created a stored procedure to backup a database, and then wrote a batch file that used SQLCMD to execute it for each database on the server. Add to the batch file a call to RoboCopy to transfer the backups to our NAS drive, then set up a scheduled task against the batch file, and I’m done, I thought.

If only!

The first problem was how to get the task to run under the Local System account – I didn’t want to use a standard account, because then I have the hassle of password management (I’m using Windows Server 2003 here – if I was on Windows Server 2008 R2 I could use Managed Service Accounts and have the server take care of the passwords). Going through the Add Scheduled Task UI doesn’t give you the option of using the Local System account.

For that, I discovered, you need to use the at command:

at 20:00 /every:m,t,w,th,f,sa,su "D:\Backups\DoBackups.bat"

does the job, scheduling my batch file for 8:00pm every day of the week.

OK. Scheduled task appears in the list. Run it to check that it works. Uh oh!

The task clearly failed to complete, but how was I to find out why? Clicking Advanced > View Log in the scheduled tasks window brings up the log file – the completely useless log file that tells you that your task started and then stopped straight away “with an error code of (2)”. Right – could you be more … specific?

So I pushed further into the murky world of bat file programming.

Joshua Curtis saved the day. His post on Redirecting Output to a File in Windows Batch Scripts gave me exactly what I needed.

First, I refactored my batch script into DoBackupsCore.bat. Then, in DoBackups.bat I wrote this:

echo [%date% - %time%] Log start > D:\Backups\log.txt
CALL "D:\Backups\DoBackupsCore.bat" >> D:\Backups\log.txt 2>&1

On the first line the > command redirects output to the log file, erasing anything already in it. In the 2nd line, the >> command redirects the output of my actual backup script to the log file, but appends to what’s already there. The really clever part of this magic spell is the last 4 characters: "2>&1”. I refer you to Joshua for the details, but this basically makes sure that errors are logged to the file, as well as successful outcomes

So I got what I needed: lovely, wordy error messages enabling me to fix my script and go home with that lovely backed-up feeling.

Friday 16 July 2010

My PDC 2010 Prediction: Free Enthusiast App Hosting on Windows Azure

So Microsoft have announced that there will, after all, be a Platform Developers Conference this year. It will however, be an austerity PDC, chopped down to two days, held on Microsoft’s campus rather than in a big conference city, and with a much narrower focus. This is billed to be a Cloud Conference. And I think I know what Mr Steve “Developers, Developers, Developers” Balmer is going to be announcing.

Think about what Microsoft have revealed over the last couple of weeks:

  • WebMatrix, a new streamlined IDE aimed at enthusiasts wanting to produce web applications. And skipping hand-in-hand with WebMatrix comes Razor, a new syntax for creating web views.
  • SQL Server Compact Edition 4, a new version of the embeddable, in-process edition of SQL Server that has been enabled to run inside ASP.Net
  • IIS Express, a version of IIS 7.5 with full support for the new IIS Integrated pipeline, SSL, etc. but capable of running in a locked down environment with no admin privileges.

Then there are the rumours about KittyHawk, a tool aimed a tech-savvy business users wanting to produce Silverlight applications without any coding.

Add all this up, and what do you get? Hundreds of enthusiasts with shiny new web applications, eager to share them with the world – but not wanting to go through the pain of ftp-ing their creation to a web host.

Hence my prediction: At PDC 2010 Microsoft will announce free hosting for enthusiast-scale web applications within Windows Azure. And they’ll throw in tools to publish web applications from WebMatrix and Visual Studio Express to Azure at the click of a button.

Take a look at the top two feature requests on mygreatwindowsazureidea.com (the official feature request list for Windows Azure): Make it less expensive to run my very small service on Windows Azure and Continue Azure offering free for Developers. Microsoft seem pretty serious about addressing issues on such lists just recently.

Google have a head-start in this arena: they’ve been offering this kind of free hosting in their AppEngine for years. But with their Visual Studio tooling, I think Microsoft could clean up.

Wednesday 14 July 2010

C# 4 broke my code! The Pitfalls of COM Interop and Extension methods

A couple of weeks back I got the go-ahead from my boss to upgrade to Visual Studio 2010. What with Microsoft’s legendary commitment to backwards compatibility, and the extended beta period for the 2010 release, I wasn’t expecting any problems. The Upgrade Wizard worked its magic without hiccup, the code compiled, everything seemed fine.

Banana Skin - Image: Chris Sharp / FreeDigitalPhotos.netThen I ran our suite of unit tests. 6145 passed. 15 failed.

When I perused the logs of the failing tests they all had this exception in common:

COMException: Type mismatch. (Exception from HRESULT: 0x80020005 (DISP_E_TYPEMISMATCH))

How could that happen? Nothing should have changed: though I was compiling in VS 2010, I was still targeting .Net 3.5. The runtime behaviour should be identical. Yet it looked like COM Interop itself was broken. Surely not?

A footprint in the flowerbed

I dug deeper. In every case, I was calling an extension method on a COM object – an extension method provided by Microsoft’s VSTO Power Tools, no less.

These Power Tools are painkillers that Microsoft made available for courageous souls programming against the Office Object Model using C#. Up until C# 4, calling a method with optional parameters required copious use of Type.Missing – one use for every optional parameter you didn’t wish to specify. Here’s an example (you might want to shield your eyes):

var workbook = application.Workbooks.Add(Missing.Value);
workbook.SaveAs(
    "Test.xls", 
    Missing.Value, 
    Missing.Value, 
    Missing.Value, 
    Missing.Value, 
    Missing.Value, 
    XlSaveAsAccessMode.xlNoChange, 
    Missing.Value, 
    Missing.Value, 
    Missing.Value, 
    Missing.Value, 
    Missing.Value);

The Power Tools library provides extension methods that hide the optional parameters:

using Microsoft.Office.Interop.Excel.Extensions;

var workbook = application.Workbooks.Add();
workbook.SaveAs(
    new WorkbookSaveAsArgs { FileName = "Test.xls"} );

Can you see what the problem is yet? I couldn’t. So I thought I’d try a work-around.

A big part of C# 4 is playing catch-up with VB.Net improving support for COM Interop. The headline feature is that fancy new dynamic keyword that makes it possible to do late-bound COM calls, but the one that’s relevant here is the support for optional and named parameters. The nice thing is that, whereas using the dynamic keyword requires you to target .Net 4.0, optional and named parameters are a compiler feature, so you can make use of them even if you are targeting .Net 3.5.

That meant that I didn’t need Power Tools any more. In C# 4 I can just rewrite my code like this:

var workbook = application.Workbooks.Add();
workbook.SaveAs( Filename: "Test.xls" );

And that worked. COM Interop clearly wasn’t broken. But why was the code involving extension methods failing?

Looking again over the stack traces of the exceptions in my failed tests I noticed something very interesting. The extension methods didn’t appear. The call was going directly from my method into the COM Interop layer. I was running a Debug build, so I knew that the extension method wasn’t being optimised out.

The big reveal

Then light dawned.

Take a look at the signature of the SaveAs method as revealed by Reflector:

void SaveAs(
   [In, Optional, MarshalAs(UnmanagedType.Struct)] object Filename,
    /* 11 other parameters snipped */)

Notice in particular how Filename is typed as object rather than string. This always happens to optional parameters in COM Interop to allow the Type.Missing value to be passed through if you opt out of supplying a real value.

Now when C# 3.0 was compiling my code and deciding what I meant by

workbook.SaveAs(new WorkbookSaveAsArgs { ... } );

it had to choose between a call to the SaveAs instance method with 12 parameters or a call to the SaveAs extension method with a single parameter. The extension method wins hands down.

But the landscape changes in C# 4.0. Now the compiler knows about optional parameters. It’s as if a whole bunch of overloads have been added to the SaveAs method on Workbook with 0 through to 12 parameters. Which method does the compiler pick this time? Remember that it will always go for an instance method over an extension method. Since new WorkbookSaveAsArgs { … } is a perfectly good object the compiler chooses the SaveAs instance method, completely ignoring the extension method.

All seems well until we hit Run and Excel is handed the WorkbookSaveAsArgs instance by COM Interop. Not knowing how to handle it, it spits it back out in the form of a Type Mismatch exception.

Mystery solved.

The moral

So watch out if your code uses extension methods on COM objects and you’re planning on upgrading to VS 2010. Don’t assume that your code works the same as before just because it compiled OK. Unit tests rule!

Update: Microsoft have confirmed that this is indeed a breaking change that they didn’t spot before shipping.

Friday 25 June 2010

WPF MVVM Commands - new reduced-boilerplate recipe

Morgan Skinner, an Application Development Consultant with Microsoft UK, had a post on his blog yesterday complaining at the quantity of boilerplate code needed to create commands in ViewModels for WPF and Silverlight. I share his irk about that.

He shows a clever technique involving ICustomTypeDescriptor that avoids creating properties (and hence backing fields) for each of the commands. But I think he goes too far. Without the properties on the ViewModel, its tests are going to have a hard time knowing what commands it exposes; and accessing them through code isn’t going to be as straightforward as a property access.

So I propose an alternative technique. It boils down to this:

public class MyViewModel : ViewModelBase
{
    public ICommand Demonstrate
    {
        get
        {
            return Commands.GetOrCreateCommand(
                () => this.Demonstrate,
                DoDemonstration,
                IsFeelingDemonstrative);
        }
    }

    private void DoDemonstration()
    {
        MessageBox.Show("Ta da!");
    }

    private bool IsFeelingDemonstrative()
    {
        return DateTime.Today.DayOfWeek == DayOfWeek.Friday;
    }
}

As you can see, ViewModelBase is providing a Commands property to manage all the commands. This neatly avoids having to create a field for each one. Commands is an instance of CommandHolder which is equally simple:

public class ViewModelBase : INotifyPropertyChanged
{
    CommandHolder _commandHolder = new CommandHolder();

    // snipped Property Change stuff

    protected CommandHolder Commands
    {
        get
        {
            return (_commandHolder ?? (_commandHolder = new CommandHolder()));
        }
    }
}
public class CommandHolder
{
    Dictionary<string, ICommand> _commands = new Dictionary<string, ICommand>();

    public ICommand GetOrCreateCommand<T>(
        Expression<Func<T>> commandNameExpression, 
        Action executeCommandAction, 
        Func<bool> canExecutePredicate)
    {
        return GetOrCreateCommand(
            commandNameExpression,
            parameter => executeCommandAction(),
            parameter => canExecutePredicate());    
    }

    public ICommand GetOrCreateCommand<T>(
        Expression<Func<T>> commandNameExpression, 
        Action<object> executeCommandAction, 
        Func<object, bool> canExecutePredicate)
    {
        var commandName = SymbolHelpers.GetPropertyName(commandNameExpression);

        if (!_commands.ContainsKey(commandName))
        {
            var command = new DelegateCommand(executeCommandAction, canExecutePredicate);
            _commands.Add(commandName, command);
        }

        return _commands[commandName];
    }
}

What do you think? Are my concerns about testa-reada-bility warranted? Does my technique eliminate enough of the boilerplate?

I’ve put a full sample on MSDN code gallery.

Monday 14 June 2010

Project Euler 59: Cracking an XOR code

Code breaking is about as glamorous as it gets for a Geek. So when it comes to Project Euler Problem 59, think 24, with Jack Bauer unscathed amidst a hail of bullets, memory stick in one hand, terrorist’s neck throttled in the other, threatening terrible retribution to the bad guy’s family if the secret goes to the grave with him. Terrorist rasps out,

“There’s a file … cipher1.txt … ASCII format … urrrh … XOR … 3 … letter … password … lower case … all English”

What would Chloe O’Brian make of that when she received the file over a secure socket from Jack’s Smart Phone? Being the bright girl she is, I’ve no doubt that she’d fire up MonoDevelop (only the dumb FBI would use Visual Studio and Windoze) and write the following:

var encryptedBytes = File.ReadAllText(dataPath)
                        .Split(',')
                        .Select(byte.Parse)
                        .ToArray();

That would get her the encrypted data from the file as a sequence of bytes. Now to generate all possible passwords:

const byte FirstChar = (byte)'a';
const byte LastChar = (byte)'z';

var allPossiblePasswords = from a in FirstChar.To(LastChar)
                           from b in FirstChar.To(LastChar)
                           from c in FirstChar.To(LastChar)
                           select new[] { a, b, c };

XOR encryption requires that the password be repeated cyclically, abcabcabc.... So given a sequence containing the characters of password, how about a Repeat extension method that repeats the sequence ad infinitum? (if you look at the following code and hear the Infinite Loop alarms go off, remember that iterators are lazily evaluated):

public static IEnumerable<T> Repeat<T>(this IEnumerable<T> sequence)
{
    while (true)
    {
        foreach (var item in sequence)
        {
            yield return item;
        }
    }
}

While she’s at it, she’ll need to Xor two sequences together:

public static IEnumerable<byte> Xor(this IEnumerable<byte> sequenceA, IEnumerable<byte> sequenceB)
{
    return sequenceA.Zip(sequenceB, (left, right) => (byte)(left ^ right));
}

This uses the Zip method, new in .Net 4.0, which merges two sequences together, allowing elements from left and right sequences to be processed in pairs (if you’re surprised, as I was, that the (byte) cast is necessary, see this question on Stack Overflow).

Now she can try out all the passwords, seeing what they give her if she uses them to decrypt the text. Note that Chloe has already adopted .Net 4.0 and the PLINQ extensions, so she’s using AsParallel() to max out her 128 Core workstation. (ConvertAsciiBytesToString is another extension method that simply delegates to ASCIIEncoding.GetString).

var possibleDecryptions = from trialPassword in allPossiblePasswords.AsParallel()
                          let repeatedPassword = trialPassword.Repeat()
                          select encryptedBytes
                                          .Xor(repeatedPassword)
                                          .ConvertAsciiBytesToString();

But she’s hardly going to spend the next 24 hours trawling through the possible decryptions to pick out the one in recognisable English from the 17575 in gobledygook. So she needs some means of assessing them automatically. This is where Chloe is forever in my debt. She read last weeks episode of Functional Fun, so she knows about my LanguageAnalyser that uses frequency analysis to determine how likely a sample of text is to be written in a particular language.

Rather than using the analyser to determine which of several languages the sample is written in, Chloe can use her prior knowledge that the coded message is written in English, and determine which of the possible decryptions has letter frequencies that most closely match those in English.

var languageAnalyser = new LanguageAnalyser();

var decryptionsWithDistanceToEnglish = from decryptedString in possibleDecryptions
                                       select new
                                                  {
                                                      DecryptedString = decryptedString,
                                                      Distance = languageAnalyser.DetermineCloseness(decryptedString, LanguageProfiles.English)
                                                  };

var mostLikelyDecryption = decryptionsWithDistanceToEnglish.MinItem(p => p.Distance);

And that's it. Thanks to Linguistic Geometry and LINQ, Chloe can report the decrypted (and surprisingly biblical!) message to Jack, so that he can get on with saving the world.

Ahem. Before we got carried away we were trying to solve Project Euler Problem 59, and that asks for the sum of the Ascii values of the decrypted message. So we need one more step:

var sumOfAsciiValues = mostLikelyDecryption.DecryptedString
                        .ConvertStringToAsciiBytes()
                        .Aggregate(0, (runningTotal, next) => runningTotal += next);

The full code is shown below, and is also available on MSDN Code Gallery.

namespace ProjectEuler
{
    [EulerProblem(59, Title="Using a brute force attack, can you decrypt the cipher using XOR encryption?")]
    public class Problem59
    {
        public long Solve()
        {
            const byte FirstChar = 97;
            const byte LastChar = 122;

            var dataPath = @"ProblemData\Problem59.txt";

            var encryptedBytes = File.ReadAllText(dataPath)
                                    .Split(',')
                                    .Select(byte.Parse)
                                    .ToArray();

            var allPossiblePasswords = from a in FirstChar.To(LastChar)
                                       from b in FirstChar.To(LastChar)
                                       from c in FirstChar.To(LastChar)
                                       select new[] { a, b, c };

            var possibleDecryptions = from trialPassword in allPossiblePasswords.AsParallel()
                                      let repeatedPassword = trialPassword.Repeat()
                                      select encryptedBytes
                                          .Xor(repeatedPassword)
                                          .ConvertAsciiBytesToString();

            var languageAnalyser = new LanguageAnalyser();

            var decryptionsWithDistanceToEnglish = from decryptedString in possibleDecryptions
                                                   select new
                                                              {
                                                                  DecryptedString = decryptedString,
                                                                  Distance = languageAnalyser.DetermineCloseness(decryptedString, LanguageProfiles.English)
                                                              };

            var mostLikelyDecryption = decryptionsWithDistanceToEnglish.MinItem(p => p.Distance);

            var sumOfAsciiValues = mostLikelyDecryption.DecryptedString
                .ConvertStringToAsciiBytes()
                .Aggregate(0, (runningTotal, next) => runningTotal += next);

            return sumOfAsciiValues;
        }
    }
}