Tuesday 4 August 2009

An introduction to UI Automation – with spooky spirographs

A few weeks ago, I unearthed a hidden gem in the .Net framework: the UIAutomation API. UI Automation made its entrance as part of .Net 3.0, but was overshadowed by the trio of W*Fs – if they’d named it Windows Automation Foundation it might have received more love! UIAutomation provides a robust way of poking, prodding and perusing any widget shown on the Windows desktop; it even works with Silverlight. It can be used for many things, like building Screen Readers, writing automated UI tests – or for creating a digital spirit to spook your colleagues by possessing Paint.Net and sketching spirographs.

Spirograph in Paint.Net

The UI Automation framework reduces the entire contents of the Windows Desktop to a tree of AutomationElement objects. Every widget of any significance, from Windows, to Menu Items, to Text Boxes is represented in the tree. The root of this tree is the Desktop window. You get hold of it, logically enough, using the static AutomationElement.RootElement property. From there you can traverse your way down the tree to just about any element on screen using two suggestively named methods on AutomationElement, FindAll and FindFirst.

Each of these two methods takes a Condition instance as a parameter, and it uses this to pick out the elements you’re looking for. The most useful kind of condition is a PropertyCondition. AutomationElements each have a number of properties like Name, Class, AutomationId, ProcessId, etc, exposing their intimate details to the world; these are what you use in the PropertyCondition to distinguish an element from its siblings when you’re hunting for it using one of the Find methods.

Finding Elements to automate

Let me show you an example. We want to automate Paint.Net, so first we fire up an instance of the Paint.Net process:

private string PaintDotNetPath = @"C:\Program Files\Paint.NET\PaintDotNet.exe";

...

var processStartInfo = new ProcessStartInfo(paintDotNetPath);
var process = Process.Start(processStartInfo);

Having started it up, we wait for it to initialize (the Delay method simply calls Thread.Sleep with the appropriate timespan):

process.WaitForInputIdle();
Delay(4000);

At this point, Paint.Net is up on screen, waiting for us to start doodling. This is where the UIAutomation bit begins. We need to get hold of Paint.Net’s main Window. Since we know the Process Id of Paint.Net, we’ll use a PropertyCondition bound to the ProcessId property:

var mainWindow = AutomationElement.RootElement.FindChildByProcessId(process.Id);

You won’t find the FindChildByProcessId method on the AutomationElement class: it’s an extension method I’ve created to wrap the call to FindFirst:

public static class AutomationExtensions
{
   public static AutomationElement FindChildByProcessId(this AutomationElement element, int processId)
   {
       var result = element.FindChildByCondition(
           new PropertyCondition(AutomationElement.ProcessIdProperty, processId));

       return result;
   }

   public static AutomationElement FindChildByCondition(this AutomationElement element, Condition condition)
   {
       var result = element.FindFirst(
           TreeScope.Children,
           condition);

       return result;
   }
}

Having found the main screen, we need to dig into it to find the actual drawing canvas element. This is were we need UISpy (which comes as part of the Windows SDK). UISpy lays bare the automation tree of the desktop and the applications on it. You can use it to snoop at the properties of any AutomationElement on screen, and to make snooping a snap, it has a particularly helpful mode where you can Ctrl-Click an element on screen to locate the corresponding AutomationElement in the automation tree (click the mouse icon on the UISpy toolbar to activate this mode). Using these special powers it doesn’t take long to discover that the drawing canvas is an element with AutomationId property set to “surfaceBox”, and is a child of another element, with AutomationId set to “panel”, which in turn is a child of another element with [snip - I’ll spare you the details], which is a child of the Paint.Net main window.Spying on Paint.Net with UISpy

To assist in navigating this kind of hierarchy (a task you have to do all the time when automating any non-trivial application), I’ve cooked up the FindDescendentByIdPath extension method (the implementation of which is a whole other blog post). With that, finding the drawing canvas element is as simple as:

// find the Paint.Net drawing Canvas
var canvas = mainWindow.FindDescendentByIdPath(new[] {
    "appWorkspace",
      "workspacePanel",
        "DocumentView",
          "panel",
            "surfaceBox" });

Animating the Mouse

Now for the fun part. Do you remember Spirographs? They are mathematical toys for drawing pretty geometrical pictures. But have you ever tried drawing one freehand? Well here’s your chance to convince your friends that you have artistic talents surpassing Michelangelo’s.

Jürgen Köller has very kindly written up the mathematical equations that produce these pretty pictures, and I’ve translated them into a C# iterator that produces a sequence of points along the spirograph curve (don’t worry too much about littleR, bigR, etc. – they’re the parameters that govern the shape of the spirograph):

private IEnumerable<Point> GetPointsForSpirograph(int centerX, int centerY, double littleR, double bigR, double a, int tStart, int tEnd)
{
   // Equations from http://www.mathematische-basteleien.de/spirographs.htm
   for (double t = tStart; t < tEnd; t+= 0.1)
   {
       var rDifference = bigR - littleR;
       var rRatio = littleR / bigR;
       var x = (rDifference * Math.Cos(rRatio * t) + a * Math.Cos((1 - rRatio) * t)) * 25;
       var y = (rDifference * Math.Sin(rRatio * t) - a * Math.Sin((1 - rRatio) * t)) * 25;

       yield return new Point(centerX + (int)x, centerY + (int)y);
   }
}

So where are we? We have the Paint.Net canvas open on screen, and we have a set of points that we want to render. Conveniently for us, the default tool in Paint.Net is the brush tool. So to sketch out the spirograph, we just need to automate the mouse to move over the canvas, press the left button, move from point to point, and release the left button. As far as I know there’s no functionality built into the UIAutomation API to automate the mouse, but the WPF TestAPI (free to download from CodePlex) compensates for that. In its static Mouse class it provides Up, Down, and MoveTo methods that do all we need.

private void DrawSpirographWaveOnCanvas(AutomationElement canvasElement)
{
   var bounds = canvasElement.Current.BoundingRectangle;

   var centerX = (int)(bounds.X + bounds.Width /2);
   int centerY = (int)(bounds.Y + bounds.Height / 2);

   var points = GetPointsForSpirograph(centerX, centerY, 1.02, 5, 2, 0, 300);

   Mouse.MoveTo(points.First());
   Mouse.Down(MouseButton.Left);

   AnimateMouseThroughPoints(points);

   Mouse.Up(MouseButton.Left);
}

private void AnimateMouseThroughPoints(IEnumerable<Point> points)
{
   foreach (var point in points)
   {
       Mouse.MoveTo(point);
       Delay(5);
   }
}

Clicking Buttons

Once sufficient time has elapsed for your colleagues to admire the drawing, the last thing our automation script needs to do is tidy away – close down Paint.Net, in other words. This allows me to demonstrate another aspect of UIAutomation – how to manipulate elements on screen other than by simulating mouse moves and clicks.

Shutting down Paint.Net when there is an unsaved document requires two steps: clicking the Close button, and then clicking “Don’t Save” in the confirmation dialog box. As before, we use UISpy to discover the Automation Id of the Close button and its parents so that we can get a reference to the appropriate AutomationElement:

var closeButton = mainWindow.FindDescendentByIdPath(new[] {"TitleBar", "Close"});

Now that we have the button, we can get hold of its Invoke pattern. Depending on what kind of widget it represents, every AutomationElement makes available certain Patterns. These Patterns cover the kinds of interaction that are possible with that widget. So,for example, buttons (and button-like things such as hyperlinks) support the Invoke pattern with a method for Invoking the action, list items support the SelectionItem pattern with methods for selecting the item, or adding it to the selection, and Text Boxes support the Text pattern with methods for selecting a range of text and querying its attributes. On MSDN, you’ll find a full list of the available patterns.

To invoke the methods of a pattern on a particular AutomationElement, you need to get hold of a reference to the pattern implementation on the element. First you find the appropriate pattern meta-data. For the Invoke pattern, for example, this would be InvokePattern.Pattern; other patterns follow the same convention. Then you pass that meta-data to the GetCurrentPattern method on the AutomationElement class. When you’ve got a reference to the pattern implementation, you can go ahead an invoke the relevant methods.

Once again, I’ve made all this a bit easier by creating some extension methods (only the InvokePattern is shown here; extension methods for other patterns are available in the sample code):

public static class PatternExtensions
{
   public static InvokePattern GetInvokePattern(this AutomationElement element)
   {
       return element.GetPattern<InvokePattern>(InvokePattern.Pattern);
   }

   public static T GetPattern<T>(this AutomationElement element, AutomationPattern pattern) where T : class
   {
       var patternObject = element.GetCurrentPattern(pattern);

       return patternObject as T;
   }
}

With that I can now click the close button:

closeButton.GetInvokePattern().Invoke();

Then, after a short delay to allow the confirmation dialog to show up, I can click the Don’t Save button:

// give chance for the close dialog to show
Delay();

var dontSaveButton = mainWindow.FindDescendentByNamePath(new[] {"Unsaved Changes", "Don't Save"});

Mouse.MoveTo(dontSaveButton.GetClickablePoint().ToDrawingPoint());
Mouse.Click(MouseButton.Left);

For variation, I click this button by actually moving the mouse to its centre (line 6) then performing the click.

All the code is available on GitHub.

Bowing out

When I first read about UI Automation, I got the impression that it was rather complicated, with lots of code needed to make it do anything useful. I tried using Project White (a Thoughtworks sponsored wrapper around UIAutomation), thinking that would save me from the devilish details. It turned out the Project White introduced complexities of its own, and actually, UI Automation is pretty straightforward to use, especially when oiled with my extension methods. I’ve had a lot of fun using it to create automated tests of our product over the last couple of months.

Update

16/7/2014: following the demise of code.mdsn.microsoft.com, I’ve moved the code to GitHub. I’ve also updated it so that it works with Paint.Net 4.0

40 comments:

Anders Kjellerup Jacobsen said...

Great post!

Dan Dumitru said...

1. I'm happy you're back to loooong technical articles

2. A small typo:
"you use these properties to to distinguish an element from its siblings"

3. Really nice post. And although I'll probably never use UI Automation code, it was a pleasant read. And good thing I know now it exists.

Unknown said...

Dan and Anders,
Thanks for the Feedback. To be honest, I was surprised at the popularity of this post.

Can I ask your opinion on my "filler" posts (the shorter ones that seem to be all I have time for most of the time): clearly you and others prefer longer technical posts, but are the shorter ones worthwhile? Do you find them interesting/amusing, or do you just skip as soon as you see there's no technical content?

Dan Dumitru said...

From my side, I don't find the need for the filler posts.
I have you on my reader, so I won't get bored if you only write one technical article per month, or so.

Anonymous said...

I think it's bad style to create a wrapper function for something as simple as Thread.Sleep. Is "Thread.Sleep" too much to type? Readers have to check the definition of Delay upon first seeing it.

But thanks for the great Automation tips!

Unknown said...

@Anonymous,
Thanks for your comment. However, I think that having a method called Delay is more intention-revealing. The fact that it calls Thread.Sleep is an implementation detail - what if I decided that in some cases it was more efficient to Spin-wait?

jeux ps2 said...

Amazing article here! I like this site. Thanks for sharing such technical information here. I am interested in graphical design field. So this information will be useful to me. I like this site. I will visit this site in future too.

Kirill Osenkov said...

Awesome article! Thanks!

Unknown said...

Kirill,
Glad you liked it - and thanks for the link on your blog!

Hari Sreekanth Upadrasta said...

Hi,
I have a question. Can UIAutomation be used on any thing that is visible on Windows.

Unknown said...

No, not everything that is visible. UI widgets have to implement particular interfaces in order to be controllable through UIAutomation.

The best way to check if you can control something with UIAutomation is to open UISpy and activate the Control-Click mode (see the article above). Then click on the thing you're interested in. If it's controllable, UISpy will show it in its tree of controls.

Román Fresneda Quiroga said...

Hey, I know this post is old, but I found this now reading through my blog roll. Maybe you already know about it, might be worth it a try: http://white.codeplex.com/

Anonymous said...

It's a great post,however i would like to know how much programming one should know to start with UI Automation? Also, when comparing white tool and UI Automation, what would be more useful? Any suggestions will be appreciated. Thanks in advance

Anonymous said...

how to write automated tests for applications like sharepoint which contains silverlight controls and html tags....Is it possible to test html tags with UI Automation

Anonymous said...

Excellent post, thanks for the info. Can the UIAutomation API be used for web testing? I have a Silverlight widget that is accessible via a browser. I'm used to using WatiN for browser automation and was trying to leverage White for Silverlight, thus far I've hit some snags and wanted to explore other options.

Unknown said...

John, Thanks!

I've used UIAutomation for Silverlight testing very successfully, without needing to involve White.

Anonymous said...

As john asked is UI automation api be used for web testing... i,e widget visible only in browser not in UI SPY....if yes, can a sample code be provided for a scenarios like <object id="silverlightClient" style="display:none;" data="data:application/x-silverlight-2...


and for tags inside iframe tag..

Unknown said...

@Anonymous, I don't believe UIAutomation can be used for standard html widgets in the browser, only Silverlight ones. In general, if you can't see it in UISpy, you can't automate it.

As to elements inside iframes: I think it's possible but you'd probably have to locate your iframe on screen, then use the AutomationElement.FromPoint method.

Anonymous said...

Is there any way to test html widgets along with Silverlight ones, because this is case of testing different web parts of Sharepoint. Can a simple sample code be provided doing both in a single test case.....

Unknown said...

@Anonymous,
For testing non-silverlight content, you'd probably what to use WaitN, Selenium, or one of the alternatives mentioned in this web page (http://stackoverflow.com/questions/426190/)

Anonymous said...

i am familiar with WebDriver, but the problem is i have to hook to already opened browser which is not supported in WebDriver....my test scenario is do some action with UI automation on silverlight web part in Sharepoint and test whether that got reflected in some non-silverlight webpart... any ideas to get hold of already opened browser...or to get hold of html content of page and performing actions on it...

Anonymous said...

First thanks for a very quick responses....Any idea to find Silverlight object using UI Automation... the link http://stackoverflow.com/questions/3408742/how-do-i-find-a-silvlerlight-object-on-an-asp-net-page-using-microsoft-ui-automa/4076240#4076240 provides the example, but it works only if there is only one object tag.... in my application i have more.. how should i handle it....any idea?.... also one doubt, will windowless property value true killing accessiblity in UI SPY? then what should be done to handle such situation?

Unknown said...

I don't have a good answer for when there is more than one object tag. The best I can suggest is trying to locate the object tag on screen, then using AutomationElement.FromPoint.

Anonymous said...

thanks for the info, right now i am trying with watin and white. One strange problem i got is, i have an application exe shortcut on the desktop. When i launch it manually i am able to see all the controls of the pane in the UI spy, but the same when i do programatically the visiblity is upto pane, not the contents of the pane... any idea regarding this?

Unknown said...

I'm afraid I don't know why that should be happening. Could it be a timing issue - just a case of needing to wait a second or so after launching the app before it finishes populating its screen. I find that I need to use Thread.Sleep() quite liberally in my UIAutomation testing code...

Anonymous said...

already I used Thread.sleep.....Just now i found solution to this issue.... may be this is appropriate or not i donno.... i opened visual studio in administrator mode and ran the test which resulted in this error. In some article it is given about Window visibility related to some access rights, so i thought of opening visual studio normally.... and it worked for me.....

Unknown said...

Ah! That was the other thing I had in the back of my mind - you can't automate a process unless the controlling process has the same or greater privileges than the target process (a gaping security hole would be created otherwise!).

Anonymous said...

Hi SAM
Really nice Article. Your Article makes my life easy. Thanks a lot.

Thanks
Attiq

Anonymous said...

Hi Sam,

How to Automate a IR or FireFoX. Please help.
I want a program that capture the selected Text from any Browser (IE,Chrome, FireFox)

Please help me
Thanks

Feriebolig Fanø said...

Thanks alot for this post. It was excactly what I was looking for.. I need to modify the functionality of a program, that I dont have the source for. This way of programming will let me control/monitoring the program from outside the program - and make wise decissions, if an error occurs.
Thanks a ton!

Anonymous said...

Thanks for those extension file. Really works and performed well on my computer.

Joseph Taylor said...

Thanks a lot for this article, I was struggling to get started with UI Automation really helped me get my head around it. It's running in our test suite enabling us to automate our Silverlight components while WebDriver does the rest.

A10165022 said...

Thanks for reminding me of spirograph!  I loved that toy!  Am getting it for my son for xmas.  

Sankaronline7 said...

var grid = rootElement.AsQueryable(TreeScope.Descendants).First(o => o.ClassName == "GridControl" && o.ControlType == ControlType.DataItem);
 
While implementing the above code i was getting the grid as null.
What should i do?
Please help me out.
Thanks
Sankar.

Marcelo Finkielsztein said...

Very useful post! thanks!

Two comments:

[1] Trying to use your technique, I found that i needed to wait for Windows to become "stable" after a new form is loaded (at least that is what i 'think' it happens!!), which (to me) explains why those "Delay(2000)" instructions; so I believe i will modify your Automation-Extensions class slightly. I am adding a "while" loop with a stopwatch around the FindChildByCondition functions.

Now the method loops while the result is null (and the elapsed time is less than 60 seconds; to avoid the risk of infinite loops). After 60 seconds, the 'while' loop will give up and return a null result, but at least during that "grace" period, it will reattempt a few times.
After each cycle, i inserted a moment of Delay() and a "Application.Doevents" statement, with the intention to let "the dust settle".

A preliminary version looks like this. Later on will refactor and make it more flezible, configurable, etc, but for the time being . . .


public static AutomationElement FindChildByProcessId(this AutomationElement element, int processId)
{
AutomationElement result = null;
Stopwatch stopwatch = new Stopwatch();
while (stopwatch.Elapsed < TimeSpan.FromSeconds(60) && result == null)
{
result = element.FindChildByCondition(new PropertyCondition(AutomationElement.ProcessIdProperty, processId));
System.Threading.Thread.Sleep(500);
System.Windows.Forms.Application.DoEvents();
}
stopwatch = null;
return result;
}

Does this make sense? i look forward to receiving your feedback about this!!!

[2] I need to automate two apps, developed in-house. One Windows form and one web-app (asp.net 2.0). For the Windows form, no doubt i am using your excellent code. Question is: What would you recommend for the web app ?? should i use the very same technique?

THANK YOU!
--
Marcelo

mars said...

Hi, thanks for the example!

One thing want to point out: the sample source code needs to change the canvas id path item "surfaceBox" to "documentBox" for the newest Paint.NET.



Cheers!

Diego Barbosa said...

Hello, Very Nice Article. I'm trying to follow the tutorial but i'm having a bit of trouble in line:



element.FindChildByCondition(new PropertyCondition(AutomationElement.ProcessIdProperty, processId));



VS can't find a reference to method "FindChildByCondition". I have searched the web for answeres but i couldn't find any, i know that is a noob mistakes, but i can't seem to find the solution.

thanks a lot.

Samuel Jack said...

It's an extension method defined in my AutomationExtensions class. To make sure the compiler can find it, you need to add "using AutomationExample;" to the top of your class (I'm assuming, of course, that you've included AutomationExtensions.cs in your project)

Ajay said...

Do you know if we can use it to automate application with custom controls by using automation peers in MVVM architecture,.

mony said...

شركة
مكافحة حشرات ورش مبيدات بالقطيف

Post a Comment