Trying out Agile

As I mentioned a few weeks back, we've started work on a new product. So far, this has entailed much writing of Functional Specifications, many meetings, and very little gathering of momentum. Whilst on holiday back in August, doing my stint of baby-sitting duty outside my daughter's bedroom, giving my wife a break, I spent some time pondering on the best way to run the project, and how we could get things moving.

I'd taken along a couple of books (more on these below) about project management, and they each recommended Agile as a way of managing projects. The more I thought about Agile in the context of our project, the more I felt it was a good idea. Returning to work, it didn't take much to persuade my fellow developer, Roman that it would be nice to try it out. Our Project Manager came on board with the idea quite quickly, and now the Director sponsoring the project has given his blessing to adopting it.

For those of you who have never considered Agile before, I thought I'd jot down my thoughts about it, and point you in the direction of some reading material that I've found helpful. Any confirmed Agilists, or Anti-Agilists should feel free to chip in where I've got it right or wrong.

What is Agile?

Agile isn't one methodology: it's a whole family of them. Think of it as an old-fashioned Taylor's shop, where you can go to choose your cloth and colour, and have a methodology made to measure. Members of the family include DSDM, Crystal Clear, the better known Scrum, and the infamous (in some circles at least) EXtreme Programming (XP). They all vary in formality, terminology, documentality (I thought I'd been clever and coined a word, but it seems I wasn't the first), but at the heart of each is an emphasis on flexibility and feedback.

Flexibility is an attribute we all wish our work possessed at the end of a project when the client decides that what he's got isn't what he wants, even though it is exactly what he signed up to at the beginning. Unfortunately, by that late stage, it's usually too late to accommodate the kind of drastic changes that the client invariably has in mind. An Agile methodology side-steps that problem by delivering the work in increments, making sure that the customer gets the things that he values most early on in the project, and giving him opportunity to change his mind about what is important at each stage as he sees how things are going, and when new ideas strike him.

For many developers, Feedback is what they get at the end of a project in the form of a dressing-down from an irate executive who has just had to explain to a customer why they finally got the software six months late, and 100% over budget. The developer would no doubt like to feed back to the executive, that the reason for this is that she was asked to commit to a plan made before the requirements were even agreed or developers assigned to the project; to say nothing of the design, meticulously refined during the first few weeks of the project, that had to be thrown out when a key feature was added half-way through. Mention Agile to this kind of executive, and he'll snort, and dismiss it with the put-down that there wouldn't even be a plan if Agile had been followed. Which isn't true. Agileists might only spend a couple of hours scribbling estimates on post-it notes before they dive into coding; but they do that at regular intervals throughout the project when they know what progress they've made on implementing whole, working, tested features, so can anticipate with increasing confidence how much work remains to do.

Agile works very much on just-in-time, and just-enough principles. Requirements are gathered, often in the form of user stories, short descriptions, in the language of the user, describing something that the system needs to do. Enough of these are captured to define the first version of the system, and they include just enough detail for the group of developers on the project to give an order-of-magnitude estimate (often in Story Points) to each Story. At this stage attention turns to Iterations.

Iterations are the building blocks of Agile plans. Whatever you call them (Scrum calls them Sprints and others call them Time boxes), they are periods of time in which a new, potentially releasable, version of software is created. Customers priorities the stories, and pick out the ones they'd like for an iteration. Developers unpack each story to discover what tasks are involved, and how long each will take, in the process creating a detailed iteration plan. More discussion with the customer fleshes out each story, and provides a list of Acceptance tests that will indicate when the feature is done. The team get their heads together to design as much of the architecture as they need for that iteration; maybe the new designs will involve  refactoring code they have written before - but they always work with the safety net of fully automated unit tests to protect them. 

With each Story coded, tested and documented, the new version of the software is ready for demonstration to the customer. Seeing it might give him new ideas. But no matter; they just go on the list and get prioritised along with everything else. The Developers reflect on the iteration passed. They review their expectations about how many Iterations they expect the whole release to take, then round they go again on the next cycle¹.

The first agile practices began back in the 1980s when developers and managers realised that for many software projects the Waterfall, plan-and-design-it-all-first, methodology wasn't working. Agile built up its fan base through the 1990s, with Kent Beck introducing Extreme Programming in 1996. In 2001 a group of Agile leaders got together and created the Agile Alliance and formalised the Agile Manifesto which states the principles that unite their different methodologies. Agile now has wide adoption, with big names like Cisco, Google, Microsoft and Yahoo all using it to a greater or lesser extent.

How can I find out more?

• I first learnt about Agile in Alistair Cockburn's book, Crystal Clear. This describes the Crystal Clear Agile methodology which is perhaps the the one that refugees from Waterfall projects will feel most comfortable with, as it is more moderate than XP. Alistair leaves no area of the project lifecycle untouched, giving advice about how to carry out each part in an Agile manner. He has useful advice on the roles that different people in the project play, and what artifacts each should produce.
• Johanna Rothman has written the excellent book Manage It! that covers a multitude of Project Management techniques, not just Agile one.  Reading her book, it seems to me that Agile emerges as her favourite way of running projects.
• VersionOne, the developers of the agile planning tool that we're currently using (because they do a free version!), have put together a useful overview about Agile, Agile 101.
• Gabrielle Benefield, formerly Director of Agile Development at Yahoo has co-authored a Scrum Primer
• Last of all, I must mention Mike Cohn: I wish I'd discovered his work earlier. On his website, mountaingoatsoftware.com, he has a number of useful articles and presentation about how and why Agile works, and also his blog. I found his Introduction to Agile Estimating and Planning very helpful. I'm currently reading two of his books, User Stories Applied and and the more general Agile Estimating and Planning. Both of these books appear within the top 20 of a recent list of the Top 100 Software books, so I'm clearly not the only one who thinks these books are well worth your financial and cognitive investment. They'll be useful whatever flavour of Agile you might be considering. Each has an informative extended case study describing how Agile plays out in practice on a project.

Footnotes

1. Agilists of other denominations might detect a bias towards XP and Scrum in my description: that's because proponents of these two methodologies seem to be most vocal on the web, and have contributed most to my stack of reading material.