Crunch, the RESTful Accountants (And Introducing the Crunch API Explorer)

One of the things that scared me a little when I thought about starting my own business was the thought of the bookkeeping. I know, I know – I have a Maths degree, so I ought to be able to cope with adding up a few numbers then working out the 20% I owe to the tax man. And the 20% I owe to the VAT lady. And how to take just the right amount of salary to avoid giving anything to the PAYE person. But my arithmetic has always been terrible, and the mathematicians who lectured me, strange to say, rarely used actual numbers. It was all xs and ys and βs and πs. So that didn’t help.

A big factor in persuading me to make the leap into running a company was finding an excellent online accounting service to handle all those troublesome numbers for me. Crunch are a smart bunch of accountants based down in Brighton who have teamed up with an equally smart bunch of developers to create bookkeeping software that is actually quite fun to use (I know – I couldn’t quite believe that when I read it on another freelancer’s blog, but it’s true!). You enter all the numbers in the website as you rake in the profits, or fork out to your suppliers, and at the end of the year, Crunch will put together your accounts and send them off to H.M.R.C. They’ll even submit your VAT returns for you. And handle your payroll, if you should happen to have any minions employees. All for £59.50 a month. It’s great. Sign up here and we’ll both get a £25 Amazon voucher!

And as if all that wasn’t awesome enough, earlier this month they launched an API. It its RESTful, speaking XML, with OAuth authentication. That’s right – a REST API from an accountancy company! I should caution that the first release is limited to dealing with expenses and suppliers, but the dev team plan to add areas according to the priorities indicated by us users.

The Crunch API Explorer

Well, you know me. I couldn’t leave a shiny new toy like that lying on the shelf. So I had a play, and knocked together something that I think you’ll like.

Allow me to introduce the Crunch API Explorer:

It’s a little tool to help you poke and prod the Crunch API. You enter a URL, set the appropriate Http Method, hit Go!, and it will show you the XML that Crunch returns.

You can download it here (it installs using ClickOnce, so it will auto-update when I add new features. If you don’t already have .Net 4.0 on your machine, you should be prompted to install it). To connect to Crunch using the Crunch API Explorer, you’ll need your own API key which you can get by contacting the nice folks at api-dev@crunch.co.uk. Then you can make REST calls to your hearts delight. All the documentation you need about the resource URLs and the structure of the XML for submitting updates can be found here on the Crunch website.

Here are a few of my favourite features:

• XML Syntax colouring when you’re editing requests (this courtesy of AvalonEdit, the open source WPF text editor component that is part of SharpDevelop)
• Makes you confirm any update/delete requests made to the live server (but not to the test server)
• All the source is available on GitHub, so if ever you wanted an example of how to connect to an OAuth API with DotNetOAuth, well – now you have one (see CrunchFacade.cs). It’s all in C#, with some WPF, and of course a topping of MVVM.

There are a couple of things I wanted to add, but didn’t get time for – maybe next week:

• Remembering frequently used resource URIs, and maybe saving template Xml request documents
• Ability to choose a file and insert it into the XML request documents in Base64 format for the APIs that support file upload
• Saving Base64 encoded data in the responses to files

Anything else?

I’d love to hear from anyone who finds this useful. Feature requests are welcome (pull requests even more so). And if you fancy forking this and adapting it to explore other APIs, be my guest.

XmlSerializerFormat is NOT supported by WCF Rest 4.0 in Medium Trust

Here’s one for you googlers (or bingites): Using the XmlSerializerFormat attribute on methods that you expose over WCF Rest (i.e. services using the webHttpBinding) is not supported if your service is running in Medium Trust (as it most likely is if you’re using Shared Hosting, like Rackspace Cloud Sites).

This is contrary to what you might expect from reading documents like this, which say that XmlSerializer is supported in partial trust. I guess they forgot to test the webHttpBinding scenario. I should be clear: webHttpBinding itself works fine in Medium Trust: it’s using XmlSerializerFormat with webHttpBinding that will give you problems.

That’s the short story.

Here’s the amplified version.

How I learned: the hard way

I’m building a web service that my friends running MacOSX want to be able to use. Out in the Apple orchards they’re not so keen on SOAP as we are who sit behind Windows. So I turned to WCF Web Http, now part of .Net 4.0, which lets you take control of your urls and keep your messages to the bare necessities.

With WCF REST you have the choice of XML or JSON. I chose XML, and by default WCF will use the DataContractSerializer. That produces reasonably clean XML, but it is picky about ordering of elements within messages, it annotates elements with type attributes that are often unnecessary, and it has a rather clunky syntax for serializing arrays.

Then I read about the XmlSerializerFormat attribute. If you apply that to your Operation Contracts, it switches to using the XmlSerializer, and then, by decorating your data contracts with the appropriate attributes, you can take complete control of the xml which gets written to the wire.

All well and good – while the service was running on my machine. But when I pushed the bits to our web host, who run all websites under Medium Trust, it was a different story. All the calls to the service threw a ProtocolException with the helpful message: “(400) Bad Request”.

Turning on includeExceptionDetailInFaults, then spying on the interaction using Fiddler shed more light on the situation. I was getting back a stack trace with this at the top:

at System.ServiceModel.Dispatcher.UnwrappedTypesXmlSerializerManager.BuildSerializers()
   at System.ServiceModel.Dispatcher.UnwrappedTypesXmlSerializerManager.GetOperationSerializers(Object key)
   at System.ServiceModel.Dispatcher.SingleBodyParameterXmlSerializerMessageFormatter.EnsureSerializers()
   at System.ServiceModel.Dispatcher.SingleBodyParameterXmlSerializerMessageFormatter.GetInputSerializers()
   at System.ServiceModel.Dispatcher.SingleBodyParameterMessageFormatter.ReadObject(Message message)
   at System.ServiceModel.Dispatcher.SingleBodyParameterMessageFormatter.DeserializeRequest(Message message, Object[] parameters)

A bit of hunting around using Reflector revealed that the BuildSerializers method always calls XmlSerializer.FromMappings, and if you check that out on MSDN, you’ll see that it requires Full Trust.

So there’s no getting away from it. You can’t use XmlSerializer together with WebHttpBinding under Medium Trust. I did try the usual work-around for getting XmlSerializer to work in Medium Trust – pre-generating the serialization assemblies at build time: but that was before Reflector showed me the clinching evidence that my efforts were futile.

So what are your options? Either stick with DataContractSerializer, or switch to JSON¹. Or, if time is on your side, wait for the next version of WCF which has a new extensible Web API specially designed to let you take complete control of your message formats and such like.

---

1. Decorate your methods like this to switch to JSON format:

[ServiceContract()]
public interface IService
{
    [OperationContract]
    [WebInvoke(UriTemplate = "/MyOperation", 
               Method = "POST",
               RequestFormat = WebMessageFormat.Json,
               ResponseFormat = WebMessageFormat.Json)]
    Response MyOperation(Dto dto);
}

Let’s synchronize our watches: Determining Clock Skew with Christian’s Algorithm and the Reactive Framework

In every playground race, there’s always one child who sets off somewhere between “On your marks” and “Get set”, gaining an unfair advantage over the honest ones who wait for “Go!”. I faced a similar problem in the design of my Windows Phone 7 multi-player game, Simon Squared, though it was unsynchronized clocks and network latency that were to blame rather than cheats.

The multi-player mode in the game relies on puzzles being shown to all players at the same time, and players race to complete them. Since some puzzles can be solved in a matter of seconds, players who get to see them even slightly sooner than their competitors stand a good chance of getting to the solution first. So I needed a plan for avoiding the digital equivalent of playground squabbles.

Any such plan has to take into account network latency: an unpredictable amount of time can elapse between the server shouting “Go!”, and the phones hearing the message. So I got my server to send out a message announcing that it would shout “Go!” at time T, where T is 4 seconds on from the time on the server’s clock (and broadcast in UTC obviously). This gives even the most far-flung phone plenty of time to receive the message, check its own clock, and then start a count-down.

Bet you’ve spotted the flaw in that straight-away! What if the server’s clock and the phones’ clocks are out of sync?

An Overview of Christian’s Algorithm

That’s where Christian’s algorithm comes in. It gives us a simple method of broadcasting the time at the server to a client whilst allowing for network latency. It works like this:

1. Client sends a message to the server: “What’s the time?” [adding ‘Mr. Wolf’ is optional]. Crucially, it notes the time that it sent the message (call it T_sent)
2. Server responds as quick as it can, giving the time according to its own clock, T_server.
3. When Client gets the message, it notes the time of receipt (call it T_received). Then it does some maths: the round-trip time, RTT, is T_received – T_sent . So assuming that the server responded instantly, and that the network latency was the same in both directions, that means that the server actually sent the message RTT/2 seconds ago. Thus, at the instant Client receives the message, the time at the server is T_server + RTT/2. Then the Client can compare with its own clock and determine the difference – the clock skew.

Since network latency can vary with each request we can get an improved result by trying the whole exercise several times then cherry picking the result which had the shortest round trip time, since that minimises the error in our estimate of the network latency.

So what does that look like in code?

The Server Side – with WCF REST

Well first we need the server to be able to serve up the time:

[AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
[ServiceContract]
public class TimeService
{
    [WebGet(UriTemplate = "/?timeAtClient={clientReportedTime}")]
    public TimeCheck GetTime(string clientReportedTime)
    {
        return new TimeCheck()
                   {
                       ClientReportedTime = DateTimeOffset.Parse(clientReportedTime, CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal),
                       TimeAtServer = DateTimeOffset.UtcNow,
                   };
    }
}

This is making use of WCF 4’s REST capabilities. Notice the WebGet attribute on the method, with it’s UriTemplate. If I add the appropriate route  to my Global.ascx file,

routes.Add(new ServiceRoute("Time", new WebServiceHostFactory(), typeof(FeedbackService)));

that will ensure that HTTP GET requests like http://localhost/MyApp/Time/?timeAtClient=2011-03-17T20:19:30.2196972Z get routed to the CheckTime method with the appropriate part of the query string passed into the clientReportedTime parameter.

So that's the server side done.

RestSharp and Rx for Asynchronous Web Requests

Then the client needs to be able to call the server. I’ve been making use of the excellent RestSharp library here:

_restClient = new RestClient(ServerUrl);

...

private IObservable<TimeCheck> GetServerTime()
{
    var request = new RestRequest("Time?timeAtClient={timeNow}");
    request.AddUrlSegment("timeNow", DateTime.UtcNow.ToString("O"));

    return ExecuteRequest<TimeCheck>(request);
}

private IObservable<TResult> ExecuteRequest<TResult>(RestRequest restRequest) where TResult : new()
{
    var subject = new AsyncSubject<TResult>();

    _restClient.ExecuteAsync<TResult>(restRequest, response => HandleResponse(response, subject));

    return subject;
}

private void HandleResponse<T>(RestResponse<T> response, AsyncSubject<T> subject)
{
    subject.OnNext(response.Data);
    subject.OnCompleted();
}

This is where things start hotting up. The first part of GetServerTime is obvious enough: in lines 7-8 I’m constructing a RestRequest consisting of a url and the appropriate query string parameter. Then I Execute it. Now since we’re talking Windows Phone here, I can’t make synchronous web requests – the phone UI has to remain responsive. So I can’t wait and then return the response from the method.

Instead I return an IObservable, an interface you might not be familiar with. This creature is the brain-child of the clever folks on the Reactive Framework team. They like to call IObservable the mathematical dual of IEnumerable. And what they mean is this: IEnumerable is used when you want to pull elements one by one out of a collection. IObservable is used when you want elements to be pushed one by one to you. You can think of an IObservable as being a stream of events. And just like you can build queries on top of IEnumerable, using Where and Select, so you can build queries on top of IObservable.

Here I’m using an IObservable as the channel to push back the result of the Web request when it becomes available – you can see that happening in the HandleResponse method, which is the callback that RestRequest calls when it gets a response back from the web server.

Finally – the Algorithm

Finally we get to the implementation of the algorithm itself: and with the underpinnings in place, it turns out to be one big LINQ query:

public IObservable<TimeSpan> DetermineClockSkew()
{
    var timeOffset = (from tick in Observable.Interval(TimeSpan.FromMilliseconds(100)).Take(5)
                      from timeCheck in GetServerTime().Timestamp()
                      let estimatedOneWayLatency =
                          (timeCheck.Timestamp.UtcDateTime - timeCheck.Value.ClientReportedTime).
                              TotalMilliseconds/2
                      let estimatedTimeAtServer =
                          timeCheck.Value.TimeAtServer + TimeSpan.FromMilliseconds(estimatedOneWayLatency)
                      let predictedClockSkew = estimatedTimeAtServer - timeCheck.Timestamp
                      select new {predictedClockSkew, estimatedOneWayLatency})
        .MinBy(p => p.estimatedOneWayLatency)
        .Select(p => p.predictedClockSkew);

    return timeOffset;
}

Let's step through this line by line:

• Line 1: We start things off by generating a stream of 5 tick events, at 100 millisecond intervals – we’re going to ping the server 5 times, and use the result with the shortest round trip time
• Line 2: Each time we hear a tick we fire off a what’s-the-time request to the server using GetServerTime which we discussed above. The Timestamp() method will stamp each response that we get back from the server with the time at which we received it.
• Line 3: For each response we get back from the server we estimate the one-way latency from server to client by taking half of the round-trip time (when we send a request to the server we send our own time, and the server echoes this back in the ClientReportedTime property of the message)
• Line 4: Now that we have an estimate of how long the message took to get from the server, we can estimate the time at the server at this precise moment.
• Line 5: We can now figure out the difference between the clock on the phone and the server’s clock by comparing the time at which we received the server’s response with our estimate of the server’s time at that moment.
• Line 6: We stash our estimate of the latency, and the estimate of the clock skew into an anonymous type
• Line 7 - 8: We look over all 5 attempts at estimating the clock skew, and we take the one which had the smallest latency. Notice how MinBy returns an IObservable rather than an actual value as it might in good old Linq-to-Objects: it has to be this way, otherwise the whole method would block waiting for all the web requests to complete, rather defeating the our use of asynchronous calls elsewher.

So there you have it: thanks to Christian, the WCF REST framework and the Rx team, a neat and elegant way of determining clock skew between server and phone.

See it in Action

Remember, for a limited time only you can see this in action by downloading my game to your phone or Emulator (for free!) and playing with your friends. Full source code to the whole game is also available.

Finally, judging for the Windows Phone 7 app contest finishes on March 31st, so make sure you check out the entries, and be sure to tell Red Gate what you think of mine!

Simon Squared – We have Multi-player: Days 4, 5 and (ahem!) 6

OK. So you let me have one more day, and I took 3. But I got it working. After 6 days of work, I can show you Simon Squared in all its Windows Azure powered multi-player glory:

Simon Squared Multi-player, nicely showcasing my Multiple-instances-of-the-WP7-emulator hack

So what took me so long? Surely after getting the Windows Phone 7 emulator to talk to the Windows Azure emulator, finding a UI library for XNA and enabling multiple instances of the WP7 emulator to run at once, everything else should have been easy? If only!

Here’s what happened.

Day 4

9:00 Start by setting up a Windows Azure subscription, making use of my MSDN subscription benefits: it's pretty generous: 750 hours/month of Compute Time, 10 GB storage, 7/14 GB traffic in/out.

I get a little worried when the sign up screen tells me that I'll receive an activation email "within 24 hours" - but I'm relieved to see that within a couple of minutes of being redirected to the Azure management console (very swish Silverlight app) my subscription appears, and I can start creating deployments.

10:11 Got side-tracked into a discussion with my boss about possibilities for future products and offerings based on the work we've been doing here

12:45 Implemented the phone side behaviour for running the count down to the games beginning. Making heavy use of the Reactive Framework (which comes built into WP7) for hiding away the messiness of asynchronous calls.

13:45 Problem with WCF REST: if I return a derived type of message from my operation, the client only receives the properties defined on the base type. This is the problem that the KnownType attribute solves elsewhere in WCF, but it doesn’t seem to be working for WCF REST.

15:22 Discovered that WCF REST uses XmlSerializer by default rather than DataContractSerializer - so KnownTypes woudn't work. Also discovered that RestSharp is using XmlSerializer when sending data to the server, so not serializing inherited types in the way that DCS is expecting.

15:39 Now got inherited message types passing correctly: WCF Rest and RestSharp are both pretty pluggable, so I was able to implement a custom ISerializer in RestSharp, and a custom MediaTypeProcessor in WCF Rest, both using DataContractSerializer instead of XmlSerializer

17:30 Caught out by Http response caching. The symptom was that I’d launch a fresh HttpWebRequest,  but get back stale data. Investigation showed that the requests weren’t ever leaving the phone – it never showed up in Fiddler. I spent ages prodding my Reactive Framework query, convinced that it wasn’t issuing the requests properly. Then I remembered Http Caching. In my haste, and newbie-ness, I solved the problem by putting a NoCache on all the responses on the server side. A better way would probably have been to set the WebRequest.CachePolicy.

Day 5

09:00 Thought occurs to me that I might get on faster if I don't have to keep waiting for the Azure Emulator to start up and shut down every time I build my server. Why not just run my server project directly in IIS? I try it – and WCF Http stops working. I eventually discover that the CTP bits I’m using don’t yet support HTTPS, and I had a HTTPs binding configured on my default Website in IIS.

10:30 Fortunately Cassini works - why did it take me an hour and half to discover that!

10:43 Start work on refactoring puzzle generation, so I can do it on the server side. The goal is that the server will generate rounds of 5 puzzles, and send them out to all the players in advance. Within a round, the server will send out a message saying “start puzzle X at Time T” – and thus synchronize the experience of each of the players

12:00 Working on updating the Phone side to play the puzzles that server sends out.

13:16 Finished refactoring the Game screen so that it is backed by a GameController - allowing me to implement a SinglePlayerGameController and MultiPlayerGameController.

17:20 Got some simple state machines working on the client and server. It looks something like this:

17:40 With two instances of the emulator running, I can now see the same puzzles displayed on each. The problem is, they’re horribly out of sync – player 1 might get the puzzle two seconds ahead of player 2, which for small puzzles means he’s certain to win!

Day 6

09:21 Implemented Christian’s algorithm to estimate the clock skew between server and phone. The implementation is pretty straight-forward using the Reactive Framework, and quite elegant, if I do say so myself. Remind me to show you, sometime.

10:33 Couldn’t figure out why changes I was making in the code weren’t having any effect when I ran the game. Then discovered that Visual Studio had some how set itself to “Never Build on Run”.

12:00 It’s working! Multiple players now receive the puzzles beautifully in sync, one after another. As soon as one player completes a puzzle, it vanishes from all the other players screens, and the count-down to the next puzzle begins

13:20 Implemented a scoreboard to show the position at the end of each round

14:30 Having deployed the game to my phone, I challenge Vinod to a dual – me on the emulator, him on the phone. I now discover a serious downside to being a games developer: once a game reaches playability, you inevitably get sucked into playing when you should just be testing.

15:35 Use Expression Design 4 to recreate a game logo that Neil designed using Word. What do you think?

17:05 Spitting and polishing. Tidying up the various screens. I might not have time to make them pretty, but at least I can make the buttons big enough to touch. That’s the downside of testing with an emulator using a mouse – it doesn’t alert you to the imprecision of fat fingers!

17:30 Waiting for Windows Azure to deploy the server. Visual Studio 2010 is supposed to be able to package and deploy your project straight into the cloud, but I couldn’t get that to work (something about it not being able to resolve the URL of my storage account). Instead, I uploaded the package manually.

17:45 Still waiting for Windows Azure to start up my role instance. Give up and go home.

21:00 Wife gives me permission to interrupt our traditional Thursday evening film (The Prestige last night, – recommended) to see whether the Azure role has started up yet. It hasn’t, so in the time-honoured way I tell it to reboot – and that fixes it. A few minutes later, she’s thrashing me – mainly because she’s using the phone, and I’m using the emulator over Remote Desktop (my laptop is old, and doesn’t support XNA games in the emulator) – in that setup the emulator doesn’t respond to my mouse moves!

What I need is another phone.

Ah, what’s this I see – the Windows Phone Marketplace Developer Newsletter, announcing Windows Phone 7 Handsets for Developers. The Newsletter says that local evangelists have codes for free phones to give away. I think Mike Ormond is the WP7 evangelist for the UK …

Multi-player enabling my Windows Phone 7 game: Day 3 – The Server Side

I’m building a game to enter in Red Gate’s Windows Phone 7 App competition. I built the core of the game in 3 days from a standing start. Now I’ve challenged myself to make the game multi-player in 3 days, using Windows Azure for the server side.  Day 1 was spent getting up to speed with Windows Azure and networking on WP7. On Day 2 I got the UI sketched out. Here’s Day 3:

8:30 Begin the day with excellent news. With barely any badgering at all Red Badger have kindly offered me an non-expiring build of their XPF UI library for XNA. Thanks chaps!

And thanks to everyone who's shown support by tweeting the last couple of blog posts. I can't tell you how motivating it is as I set to work again this morning.

8:45 Microsoft have released the January 2011 Update to the Windows Phone Developer Tools (on the 4th of February mind, but I'll let that pass). This is the one that brings the promised Copy-Paste support and some performance updates. Being the good boy I am, I install it before beginning anything else.

9:32 Staring work on the server. I've never designed anything RESTful before, so I've taken a quick look at a couple of articles: one at quite a high level, and one in a little more detail.  A logical place to begin is starting a game. So what I think I need to do is define a /Game resource on my server, and make it respond to POST requests.

11:45 Hooked up AutoFac on the server so that I can inject dependencies into my REST services.

12:00 A good chunk of the day gone, and I still haven't implemented starting a game. WCF isn't playing nice at the moment: I can't get it to respond to the POST request I'm making to my Service.

12:01 Writing the above fixed the problem! Well, not quite. As soon as I had written that, I thought of fiddling with the UriTemplate on the WebInvoke attribute. Since I was posting a request to the /Games resource, and I had registered my Games service using routes.AddServiceRoute<GamesService>("Games", serviceConfiguration) I thought I could get away without specifying a value for UriTemplate on WebInvoke. Turned out I needed to specify "/"!

12:05 Next: list all available games, so that a player can choose one to join.

12:45 Done! I can now create several games through the Start a Game screen, and see them all listed on the Join a Game screen.

16:15 The afternoon passes in a blur. I’ve added several operations to my service, so that the basic game setup screens are working. This is what I’ve got on the server side:

Resource	Http Method	Meaning
/Games	POST	Start a Game
/Games	GET	List existing games
/Games/{id}/Players	POST	Join a game
/Games/{id}/Players	GET	List players belonging to a game

Notice any conspicuous absences? That’s right: there are no operations to support actually playing a game. Where has the time gone? But before I can move on, I should at least check that this service works if multiple devices connect to it.

16:25 Fustrated: the Microsoft Windows Phone 7 emulator is single-instance. How can I test a multi-player game if I can only run one copy of it?! I’ve got to find a way round this.

17:30 Eureka! After a journey deep into the bowels of the Microsoft.SmartDevice.Connectivity API and its associated XML datastore I’ve found a way to have multiple instances of the emulator running at once, and even debug them simultaneously.

Hey! Hang on just a minute: there’s already a post on my blog about that – is someone with a time machine playing around with me here?

But never mind that. My 3 days are up, and all I’ve got to show for it are the game sign-up screens. So where do we go from here? Well, I’m going to make my excuses and plead for an extra day – just one more day, and I’m sure I can get it working!

My excuses:

• It took me ages to get the Windows Phone 7 emulator talking to the Windows Azure emulator: the good folks at Microsoft haven’t connected the dots here.
• A good two hours or more of my first day was taken up with the hunt for a working UI library for XNA. Thanks to Red Badger for saving me further frustrating hours.
• Running multiple instances of the emulator really shouldn’t be that hard.

So what do you think? Can I have another day?