The more I read on agile practices the more I hear some fairly edgy ideas about requirements gathering, design, and architecture in the agile world. While I understand the concept of embracing change, there has to be a line in the sand somewhere of how your company performs its business and how you build software to enable that business. As I work with clients adopting Scrum I find different levels of the YAGNI (Your Ain't Gonna Need It) principle and the idea of building extensible software that gracefully accepts change. My struggle is that I need to find a happy medium with the majority of my clients. Martin Fowler's article "Is Design Dead?" makes an argument for something in between also, but he focuses more with contrasting against the more edgy concepts in XP. Unfortunately a large number of agile pundits have taken up the idea that design up front is a wasteful practice as is gathering any type of detailed requirements. These are not just the inexperienced newbies blogging from their soap boxes, but well respected and experienced thought leaders in our industry.

So is up front requirements gathering and design really dead?

I have never been a agile purist and I think that for my clients some hybrid is more prudent. In the past I have been on Scrum teams that designed on the fly and kept the YAGNI mantra. Sometimes this resulted in some patchwork architectures and some disjointed business themes in the application. While I do think it is a great idea to embrace change, a basic vision for both your business case and architecture/design of any software solution are needed to keep the smaller iterations on at least some common path. This vision should be able to change as well, but there are some fundamental things that most probably will ring true for the solution. Determining that line is a black art. Too little and you risk being all over the place in your deliverables. Too much and you start becoming too rigid and have lost the benefits of an agile approach.

So how do you know how much up front work to do?

If I had a textbook answer I would give it to you. And it would most likely be wrong. Every case is different. Each client's situation and environment is different. As a company's business evolves the line might have to move also. But I do think that forming a basic business case as the initial vision and some high level architecture and design is almost always warranted. Even Ken Schwaber, one of the originators of Scrum, advises to start as soon as you have enough backlog items to start a sprint. You do not need grand functional specifications and loads of UML diagrams. As is often quoted "The plan is useless, but the planning is essential." When determining if you need to detail a specific business case ask yourself how essential is it to our core business? Higher value requirements always get the most attention. As far as architecture/design, defining the basic components and their interactions at a high level should be enough to ensure your deliverables follow a cohesive path. When determining how extensible a given component should be, concentrate of the most probable changes rather than inventing all sorts of possible future scenarios.

So how can you gather requirements and perform up front design while still being agile?

If you are going to start a sprint with just enough product backlogs items then some of those product backlog items should be forming high level business cases and prototypes so that subsequent sprints will have the benefit of an uniform vision on both fronts. I also believe that you do need some level of documentation for these outside of just the backlogs. A wiki is a great place for them since by its very nature embraces change (and the good ones can provide history for those people who just cannot get over the idea of this type of change). To take that idea to the next step, Team Foundation server allows you to add links to web pages on backlog items so that you can tie the documentation to the user functionality and development tasks. If you want to go low-tech use flip charts and save all the paper. Post it on the walls of the room where the team is literally surrounded by the business and architectural vision.

Even if you are well into an agile project, you can always have the stakeholders redefine the vision if you feel you are straying too far because of only thinking about the next sprint. Change is inevitable, but your company's core business fundamentally changing in not very probable. And good design should elegantly accept change also. I do not think up front gathering of requirements or design is dead. I just think that in agile environments it has evolved and shed its more useless features.