Tuesday 25 August 2009

Introducing HymnSheet – free, simple song presentation software for Churches and Schools

Our church jumped into the computer age a few years back. We ditched the Overhead Projector that had served us faithfully for many years, and acquired one of its digital brethren, along with a laptop. Ah, OHPs – those were the days – amazing what they could do with just a few lights, lenses, mirrors, and on a good day, no smoke. Anyway, I volunteered to sort out some software to do what the old OHP did for us – display songs on the screen for the Sunday School scholars to sing.

Powerpoint was an obvious first candidate, but rejected after a few moments consideration: our Sunday School leader likes to be able to add choruses to the mix at a moments notice: imagine try to insert slides into a Powerpoint presentation on the fly!

Then I scoured the ‘net for more specialised alternatives – and found plenty.But I noticed two things about all of them: firstly, they cost money; and secondly, all of them did too much – we prefer not to have our attention distracted from worship by snazzy graphics and alpha-blended video, or by nursery notices popping up to inform Mrs Jones that little Johnny needs his nappy changing.

What self-respecting programmer would not see this state of affairs as a challenge? Surely it would only take a few evenings to bash out something that would work exactly the way we wanted it to?

So (a few)^10 evenings later I had a little application ready to use. And we’ve been using it successfully for the last four years. Dragging and dropping song titles into a playlist sure beats sorting through a deck of transparencies.

Ever since I started this blog, I’ve been intending to put the application and source code online in the hope that others will find it useful. Finally I’ve got round to it. It is my birthday gift to you all (28 today – I reckon I’ll start feeling grown up in about two years time!).

What you should know about HymnSheet

So here it is, imaginatively named HymnSheet: simple song presentation software, especially suitable for children’s services, and, I imagine, School assemblies and music lessons. It features:

  • Support for dual monitors (or a laptop and a projector). One monitor displays the control screen, the other the song display
  • Drag-and-drop interface for putting songs into your play-list and for reordering them on the fly
  • Ability for you to highlight the current line of the song, to assist children learning to read
  • Easy to use song editor for adding your own songs to the song library
  • Whole song displayed at once on screen, with key shortcuts for scrolling between verses and the chorus of the song.

DualMonitorSupportGet started by downloading the installer (note that you’ll need the .Net Framework installed on your computer) and reading the user guide. I’ve also made available some sample hymns and choruses (mostly of the more traditional kind) to get you started.

If you find this useful, you might like to help me persuade my wife that it was worth my while - if you’re sufficiently persuasive, she may allow me the time to add some new features:

Points of interest in the Source Code

For the more technically minded, I’ve made the source code to HymnSheet available – feel free to adapt it and add to it if you want – I’d love to hear from you if you do. Some points of interest in the source code :

  • The code is written in VB.Net.  I know, I’m sorry. Reason is, I started working on this in the days before I had broadband. At the time I only had a CD of Visual Basic 2005 Express Beta which I’d picked up at a conference. By the time I had a a proper language installed on my machine, I didn’t fancy the job of rewriting ;-). Maybe one day…
  • Songs are stored in xml format. I use an XSL transform to turn this into HTML (with CSS to style it) and then display it in a Webbrowser control (maximised to fill the whole of the display screen). This gives you complete control over the way songs are presented. In particular, you might want to change the fonts or colours – the settings for these are in the songs.css file.
  • If you’ve never taken a look at the Webbrowser control, you should – it’s cool. As well as letting you load HTML from memory, it provides a managed DOM view of the HTML document, complete with events; you can also invoke any scripts embedded in the page.
  • I use some Javascript to animate the movement between verses. From IE 7 onwards, the default security settings prevent javascript from running in a HTML document that is loaded from memory (as is the case in HymnSheet). If you host the Webbrowser control in an application and want to enable javascript, you need to set a key in the registry: [HKEY_LOCAL_MACHINE or HKEY_CURRENT_USER]\SOFTWARE\Microsoft\Internet Explorer\Main\FeatureControl\FEATURE_BLOCK_LMZ_SCRIPT\[You executable name.exe], with value of 0. I do this in the installer, but I’ve also created a class with a method that will enable scripting for the current application: look for it in the source code – it’s called EnableWebBrowserScripting.
  • I’ve included the source code for the installer (see HymnSheet.wxs). If you want to build this, you’ll need to install WiX, the Windows Installer XML toolkit; you might also want to try out the excellent WixEdit for editing Wix scripts.

Tuesday 4 August 2009

An introduction to UI Automation – with spooky spirographs

A few weeks ago, I unearthed a hidden gem in the .Net framework: the UIAutomation API. UI Automation made its entrance as part of .Net 3.0, but was overshadowed by the trio of W*Fs – if they’d named it Windows Automation Foundation it might have received more love! UIAutomation provides a robust way of poking, prodding and perusing any widget shown on the Windows desktop; it even works with Silverlight. It can be used for many things, like building Screen Readers, writing automated UI tests – or for creating a digital spirit to spook your colleagues by possessing Paint.Net and sketching spirographs.

Spirograph in Paint.Net

The UI Automation framework reduces the entire contents of the Windows Desktop to a tree of AutomationElement objects. Every widget of any significance, from Windows, to Menu Items, to Text Boxes is represented in the tree. The root of this tree is the Desktop window. You get hold of it, logically enough, using the static AutomationElement.RootElement property. From there you can traverse your way down the tree to just about any element on screen using two suggestively named methods on AutomationElement, FindAll and FindFirst.

Each of these two methods takes a Condition instance as a parameter, and it uses this to pick out the elements you’re looking for. The most useful kind of condition is a PropertyCondition. AutomationElements each have a number of properties like Name, Class, AutomationId, ProcessId, etc, exposing their intimate details to the world; these are what you use in the PropertyCondition to distinguish an element from its siblings when you’re hunting for it using one of the Find methods.

Finding Elements to automate

Let me show you an example. We want to automate Paint.Net, so first we fire up an instance of the Paint.Net process:

private string PaintDotNetPath = @"C:\Program Files\Paint.NET\PaintDotNet.exe";

...

var processStartInfo = new ProcessStartInfo(paintDotNetPath);
var process = Process.Start(processStartInfo);

Having started it up, we wait for it to initialize (the Delay method simply calls Thread.Sleep with the appropriate timespan):

process.WaitForInputIdle();
Delay(4000);

At this point, Paint.Net is up on screen, waiting for us to start doodling. This is where the UIAutomation bit begins. We need to get hold of Paint.Net’s main Window. Since we know the Process Id of Paint.Net, we’ll use a PropertyCondition bound to the ProcessId property:

var mainWindow = AutomationElement.RootElement.FindChildByProcessId(process.Id);

You won’t find the FindChildByProcessId method on the AutomationElement class: it’s an extension method I’ve created to wrap the call to FindFirst:

public static class AutomationExtensions
{
   public static AutomationElement FindChildByProcessId(this AutomationElement element, int processId)
   {
       var result = element.FindChildByCondition(
           new PropertyCondition(AutomationElement.ProcessIdProperty, processId));

       return result;
   }

   public static AutomationElement FindChildByCondition(this AutomationElement element, Condition condition)
   {
       var result = element.FindFirst(
           TreeScope.Children,
           condition);

       return result;
   }
}

Having found the main screen, we need to dig into it to find the actual drawing canvas element. This is were we need UISpy (which comes as part of the Windows SDK). UISpy lays bare the automation tree of the desktop and the applications on it. You can use it to snoop at the properties of any AutomationElement on screen, and to make snooping a snap, it has a particularly helpful mode where you can Ctrl-Click an element on screen to locate the corresponding AutomationElement in the automation tree (click the mouse icon on the UISpy toolbar to activate this mode). Using these special powers it doesn’t take long to discover that the drawing canvas is an element with AutomationId property set to “surfaceBox”, and is a child of another element, with AutomationId set to “panel”, which in turn is a child of another element with [snip - I’ll spare you the details], which is a child of the Paint.Net main window.Spying on Paint.Net with UISpy

To assist in navigating this kind of hierarchy (a task you have to do all the time when automating any non-trivial application), I’ve cooked up the FindDescendentByIdPath extension method (the implementation of which is a whole other blog post). With that, finding the drawing canvas element is as simple as:

// find the Paint.Net drawing Canvas
var canvas = mainWindow.FindDescendentByIdPath(new[] {
    "appWorkspace",
      "workspacePanel",
        "DocumentView",
          "panel",
            "surfaceBox" });

Animating the Mouse

Now for the fun part. Do you remember Spirographs? They are mathematical toys for drawing pretty geometrical pictures. But have you ever tried drawing one freehand? Well here’s your chance to convince your friends that you have artistic talents surpassing Michelangelo’s.

Jürgen Köller has very kindly written up the mathematical equations that produce these pretty pictures, and I’ve translated them into a C# iterator that produces a sequence of points along the spirograph curve (don’t worry too much about littleR, bigR, etc. – they’re the parameters that govern the shape of the spirograph):

private IEnumerable<Point> GetPointsForSpirograph(int centerX, int centerY, double littleR, double bigR, double a, int tStart, int tEnd)
{
   // Equations from http://www.mathematische-basteleien.de/spirographs.htm
   for (double t = tStart; t < tEnd; t+= 0.1)
   {
       var rDifference = bigR - littleR;
       var rRatio = littleR / bigR;
       var x = (rDifference * Math.Cos(rRatio * t) + a * Math.Cos((1 - rRatio) * t)) * 25;
       var y = (rDifference * Math.Sin(rRatio * t) - a * Math.Sin((1 - rRatio) * t)) * 25;

       yield return new Point(centerX + (int)x, centerY + (int)y);
   }
}

So where are we? We have the Paint.Net canvas open on screen, and we have a set of points that we want to render. Conveniently for us, the default tool in Paint.Net is the brush tool. So to sketch out the spirograph, we just need to automate the mouse to move over the canvas, press the left button, move from point to point, and release the left button. As far as I know there’s no functionality built into the UIAutomation API to automate the mouse, but the WPF TestAPI (free to download from CodePlex) compensates for that. In its static Mouse class it provides Up, Down, and MoveTo methods that do all we need.

private void DrawSpirographWaveOnCanvas(AutomationElement canvasElement)
{
   var bounds = canvasElement.Current.BoundingRectangle;

   var centerX = (int)(bounds.X + bounds.Width /2);
   int centerY = (int)(bounds.Y + bounds.Height / 2);

   var points = GetPointsForSpirograph(centerX, centerY, 1.02, 5, 2, 0, 300);

   Mouse.MoveTo(points.First());
   Mouse.Down(MouseButton.Left);

   AnimateMouseThroughPoints(points);

   Mouse.Up(MouseButton.Left);
}

private void AnimateMouseThroughPoints(IEnumerable<Point> points)
{
   foreach (var point in points)
   {
       Mouse.MoveTo(point);
       Delay(5);
   }
}

Clicking Buttons

Once sufficient time has elapsed for your colleagues to admire the drawing, the last thing our automation script needs to do is tidy away – close down Paint.Net, in other words. This allows me to demonstrate another aspect of UIAutomation – how to manipulate elements on screen other than by simulating mouse moves and clicks.

Shutting down Paint.Net when there is an unsaved document requires two steps: clicking the Close button, and then clicking “Don’t Save” in the confirmation dialog box. As before, we use UISpy to discover the Automation Id of the Close button and its parents so that we can get a reference to the appropriate AutomationElement:

var closeButton = mainWindow.FindDescendentByIdPath(new[] {"TitleBar", "Close"});

Now that we have the button, we can get hold of its Invoke pattern. Depending on what kind of widget it represents, every AutomationElement makes available certain Patterns. These Patterns cover the kinds of interaction that are possible with that widget. So,for example, buttons (and button-like things such as hyperlinks) support the Invoke pattern with a method for Invoking the action, list items support the SelectionItem pattern with methods for selecting the item, or adding it to the selection, and Text Boxes support the Text pattern with methods for selecting a range of text and querying its attributes. On MSDN, you’ll find a full list of the available patterns.

To invoke the methods of a pattern on a particular AutomationElement, you need to get hold of a reference to the pattern implementation on the element. First you find the appropriate pattern meta-data. For the Invoke pattern, for example, this would be InvokePattern.Pattern; other patterns follow the same convention. Then you pass that meta-data to the GetCurrentPattern method on the AutomationElement class. When you’ve got a reference to the pattern implementation, you can go ahead an invoke the relevant methods.

Once again, I’ve made all this a bit easier by creating some extension methods (only the InvokePattern is shown here; extension methods for other patterns are available in the sample code):

public static class PatternExtensions
{
   public static InvokePattern GetInvokePattern(this AutomationElement element)
   {
       return element.GetPattern<InvokePattern>(InvokePattern.Pattern);
   }

   public static T GetPattern<T>(this AutomationElement element, AutomationPattern pattern) where T : class
   {
       var patternObject = element.GetCurrentPattern(pattern);

       return patternObject as T;
   }
}

With that I can now click the close button:

closeButton.GetInvokePattern().Invoke();

Then, after a short delay to allow the confirmation dialog to show up, I can click the Don’t Save button:

// give chance for the close dialog to show
Delay();

var dontSaveButton = mainWindow.FindDescendentByNamePath(new[] {"Unsaved Changes", "Don't Save"});

Mouse.MoveTo(dontSaveButton.GetClickablePoint().ToDrawingPoint());
Mouse.Click(MouseButton.Left);

For variation, I click this button by actually moving the mouse to its centre (line 6) then performing the click.

All the code is available on GitHub.

Bowing out

When I first read about UI Automation, I got the impression that it was rather complicated, with lots of code needed to make it do anything useful. I tried using Project White (a Thoughtworks sponsored wrapper around UIAutomation), thinking that would save me from the devilish details. It turned out the Project White introduced complexities of its own, and actually, UI Automation is pretty straightforward to use, especially when oiled with my extension methods. I’ve had a lot of fun using it to create automated tests of our product over the last couple of months.

Update

16/7/2014: following the demise of code.mdsn.microsoft.com, I’ve moved the code to GitHub. I’ve also updated it so that it works with Paint.Net 4.0