You can replace “Architecture Documentation” with any kind of artifact used to describe the design or implementation of your application. This can also apply to requirements, but for this article we’ll stick with technical documents. This happens quite a bit in shops that have a heavy waterfall or big design up front process. The architects work feverishly at the beginning of a development effort to think up an application architecture that will meet every expected need for the entire solution for the duration of the project. They will then painstakingly record this technical wonder in various forms of documentation including lots of UML diagrams because if it’s documented in UML, it has to be right. And much like the architect in the second Matrix movie, they believe that their design is “ a work of art, flawless, sublime.”

And most often when you get into the trenches of actually developing the software, ideas that sounded good on paper often fall apart in implementation and have to be modified to varying degrees. And like good little developers we immediately run back to the now out of date documentation and correct it. Right? Of course, and we deliver the updated documentation to Bigfoot on magical unicorns.

So why does this happen?

• Too Much Design Detail Up Front – Unless the application you are building is exactly like that of those you have built many times before with the exact same team, on the exact same infrastructure, with the exact same technology, and you have at your disposal the ability to manipulate the world to be the exact same as when you developed each of those previous applications, your lovely design will most likely have to be modified during implementation. Often we try to mitigate this fact by documenting even more as if the sheer onslaught of technical details will negate this age old truth.
• Concentrating on Building Frameworks and Not Features – As developers we love to solve the world’s problems whether it asked us to or not. The wasteland of shelfware is littered with the bodies of “flavor of the month” frameworks well intended developers created in hopes of alleviating all foreseeable issues in the future. This also tends to make the design more complex (although most who do it argue the opposite), and requires a certain level of documentation up front just so the developers who work on it can get their heads around how to implement and use it.
  
  This can also result in bloated software that is hard for others to maintain. While a solid architecture often leverages industry proven frameworks like Entity Framework, nHibernate, etc., it is important to weigh each framework you adopt (or write yourself) with the justification of the cost it will incur. We must also look at many of the non-technical implications that can have a large part in the decision like the level of expertise of the people implementing and supporting the application.

Here are some practices and principles I think help:

• Emergent Design – Some think Agile prescribes no design and just slinging code from the start of a project. This is a common misconception. The practice of emergent design is more about designing at a high level initially to clarify the big technical questions, but using tests and developing in iterations to validate design choices as you make them. It is a counter to the so called “ivory tower” architects who stop writing production code and work in more theoretical matters. There are times when this type of work is very much warranted but some detailed specs along with a reference application cannot be deemed as technical gospel until it is proven with production ready implementations. How much design you do up front and how much you allow to evolve along the way is a very contextual decision that should be evaluated for each project.
• Design as a Team – When you design the application as a team, then there is less need to document up front since everyone will be on the same page. It also tends to ensure developers stick to the architecture and design approaches, since they had a hand in crafting them. Another benefit is that junior developers are exposed to elements of the solution and the process for creating it that they might not have had a chance to experience if the application is designed only by the senior team members and then handed to them to implement later.
• The YAGNI Principle – This stands for “You Ain’t Gonna Need It” (I’m southern so we use ain’t instead of aren’t) which is a mantra used by Agile programmers to resist the urge to gold plate software. The more bells and whistles we try to add, the more moving parts it adds and the more points of possible failure we add. I have definitely seen this taken way too literally but stopping and really thinking about whether some extra technical detail is truly needed to meet the business need is always a good practice. The more well known and less controversial version of this is KISS (keep it simple stupid).
• Self Documenting Code – Agile does not mean you do not document. If someone tells you that, then you have my permission to punch them in the ear. Read the Agile Manifesto and you will see it says comprehensive documentation. So there is a certain level of up front design and documentation, but treat that documentation as perishable the second you implement it in code. Once the code is written then the “Code is the truth.” Writing the code in a way that its intent is obvious, even to those not extremely technical, is a great way to preserve the truth of the design. Aside form the code, there are many applications out there that can pull information out of the code into more formal documentation and diagrams so that the intent can be expressed outside of code such as nDoc, Visual Studio 2010’s Architecture Explorer features, and many more.
  • A Side Note on Code Comments – I have attended quite a few sessions/presentations over the years that say comments are evil. Well you can shut your cake holes because I like comments. I am not talking about silly, obvious comments like this:
    
    //Declare new Customer.
    Customer newCustomer = new Customer();
    
    I am talking about things like XML comments in C# for all public members of a class, and sparingly used comments that sometimes explain why something was implemented the way it was. I also follow the practice of using comments as the stubs for the code I am about to write (something I picked up years ago from Code Complete), but I normally replace those comments with the actual code.
    
    But do understand that when I say “self documenting code”, that does not refer to comments, but rather the formatting and naming of those code structures. If I have to read your comments to understand the intent of your code then I do take issue with that.

You hear them all the time in software development shops, these common phrases that always get a laugh in my presentations because people are so familiar with them. And while we hear them and maybe even say them ourselves, we seldom take a closer look at the underlying issues with our software development principles and practices that cause us to hear and say them so often. Let’s take a look at some of the more common phrases and see what issues they often reveal and what Agile practices might help alleviate them.

“It Works on My Box.”

This is by far my favorite because I hear it (and I admit that I say it) quite a bit. It is most often uttered by a developer who is told some feature they authored is not working on the server or another developer’s machine. It is so common that there are t-shirts and even a song. Very funny but it can be indicative of several issues:

• Developer Apathy – Often the phrase is accompanied with a shrug as the developer walks away. Not a very good team effort to drop the ball like that and leave someone else to pick it up and try to figure out what is wrong.
• A Poor Definition of Done – If your process allows developers to simply get something built and tested on their personal machine, then you will hear this phrase more than you would like.
• Siloed Development and Late Integration – Often teams split up tasks and go away for long periods of time to write code in a vacuum, coming together late in the process to start seeing if what they have written separately will work when integrated together. Which it rarely does right off the bat.

So what Agile principles and practices can help prevent you from hearing this phrase over and over?

• Continuous Integration - Since CI builds (and tests hopefully) software that is checked in on a separate environment, these types of issues are caught earlier. If you implement more robust stop gap measures such as TFS 2010’s new gated check in feature, you can proactively prevent bugs of this type from even getting into the source on the server. I personally think this is one of the low hanging fruits of Agile (more specifically XP) since you have to build, integrate, and deploy the solution sometime anyway, this just makes you do it sooner and more often. There are many good solutions out there besides TFS such as Team City and Cruise.
• Clear Definition of Done – Even though there is an implicit definition of done most developers have in their heads, you should sit down with your team and clarify what everyone expects to have been completed when someone claims a certain coding task is “done”. I suggest writing in down on a big sheet of paper and posting on the wall in the team room. So when someone says “Oh yeah, that’s done.” you can point to it and ask “Really?”. This can also help remove another common developer phrase from your teams vocabulary: “Done done.”

Stay tuned for our next entry: “Don’t Read the Architecture Documentation, It is Not Up To Date Anymore.”

When I talk to clients about adopting the basic values from the Agile Manifesto, I tell them to imagine sliders between the four entries on the left and right. A good adoption is like mixing a band and as the band plays you adjust the mix constantly to get the best sound. Purists may claim you must adopt each value to its fullest which is like sliding all the sliders on the mixing board to one side which never sounds very good.

I’ve had a client who had recently invested in offshore resources and a new remote office for the development teams. They could not immediately get the development team together so they could not fully embrace the “Individuals & Interactions” value or some of the practices attributed to it. So we used some tools like Skype, WebEx, and Team Foundation Server and put in a little process around communications between the sites to help retain some of the collaborative aspects of the team. So I see this like moving the slider on my Agile mixing board a little over to the right and the “Process & Tools” side. This client does see the value of having everyone together so there is a plan that once they can feasibly move development to one location, they will do so. And when they do we can remove some of the processes put in place and stop using the tools so the slider moves back to the left a little more.

Claims that you must adopt Agile in its entirety and never change your adoption to meet your current environment are not very practical for some companies. I definitely think you should always evaluate the impediments you perceive are preventing you from transitioning to Agile practices to ensure you are not reverting to muscle memory, but there are definitely real world scenarios that can make some adoptions possibly detrimental to the business if immediately (and blindly) adopted. I like this blog from Ryan Cooper about how conflict can arise from these situations.

So I think good Agile coaches are like good sound men who are constantly listening to the band and as the environment changes they adjust the mix appropriately. We listen to the team and the stakeholders to determine how much to adopt and what may have to wait for later. Anything that you do determine should wait needs to be logged with the following details:

• What it is exactly you are not going to adopt.
• What the benefit of adopting it would be.
• The cost of not adopting the practice.
• “Smells” that will indicate you should definitely adopt the practice.

This way we acknowledge that we are not adopting one of the best practices prescribed by Agile (or whatever flavor we are adopting) and then outline the cost and the return for justification later if the smells start to occur. This adoption log can also be great fodder for retrospectives.

So listen to your team and be vigilant when the environment changes so that you adjust your mix of Agile adoption to best fit the situation.

I've been re-reading a few books lately and in the process it made me start to think of how differently I approach software development than when I first started. It seemed that each time I reach another plateau of understanding one or more books helped me get started on the next level. So in this post I wanted to chronicle my growth (which is still happening!) as a developer using .NET along with the key books that helped me. While I did development before .NET (VB & some Java) I am going to only stick to .NET for brevity.

Phase One: Programmer
When you first learn how to write software you learn the syntax of the language (C# in my case) and how to construct programs using that language. I myself did not go to school to be a developer (I was actually on my way to being a professional pilot) so I had very little pretense on theory or approaches. So I wrote code in a very functional way usually.

When I first started in C# I bought Programming C# from Oreilly. This link is to the fifth edition which covers C# 3.0, but I started with the first edition on 1.0 which is now out of print. I always loved the Oreilly books because they were very direct and to the point. This book was a great help in getting started with the language.

As I started to do more with .NET I bought C# in a Nutshell also by Oreilly. This is not a "how to" book but rather a reference for the C# language and the .NET framework. I turned to this book over and over again to figure out which class to use and how to use it. It was a great companion to the Programming C# book.

Phase Two: Developer
Eventually I was more sure of my abilities that I did not have to concentrate so hard on the coding part of developing software. I had a decent grasp of the language and the framework so I started wanting to write software in a more consistent and productive way. Another consultant told me to pick up Code Complete and it started me on my way to adopting some best practices that I still adhere to today. While it had nothing to do with C#, its content was applicable to any development language.

Also during this time I read (or skimmed sometime) quite a few books from WROX on several areas of programming in .NET which all helped me start to address specific problems in my applications.

Phase Three: Architect (or so I thought)
As I started to get better a writing applications using C#, I was being asked more and more to help design these applications then systems of multiple applications. I was lucky to work with many people who were much smarter than I was and they turned me on to quite a few good resources.

During one design session I was complaining about finding a solution to a particular problem and a colleague shook his head at me and asked, "Haven't you read the gang of four Design Patterns book?" I had to sheepishly admit I have never heard of it and then immediately went out and read it that week. After which I realized I had been describing a picture perfect scenario for an Abstract Factory pattern. I then bought two books on design patterns for C#: Design Patterns in C# and C# Design Patterns: A Tutorial . Using all three books I really started to delve into learning how to write more solid, reusable software.

Phase Four: Team Lead
Once I had been the architect or lead developer on a few projects I was being asked to lead development teams on bigger projects and that when I started to dabble in the project management and business analysis side of the software development lifecycle more. Either I was frustrated by the people in these positions or the project was small enough where the development team took on these duties. Either way I felt there were better ways to do both.

One of the project managers I worked with who was very good at his job loaned me a copy of the Mythical Man Month and it helped me start to think about projects in a wider scope than just the development effort. I was a big help in understanding how to interface with customers and managers and understand how to measure the progress of a project.

I was also given a copy of Software Requirements which gave me a new perspective on gathering requirements and mapping features to coding efforts. At this time I was also introduced to Rational's RequisitePro product which implemented a good deal of the concepts behind the book.

Phase Five: The Reluctant Agilist
By this time I had been leading software development projects for a few years with a good degree of success. I will admit I thought I had it all pretty under control and then a project that went bad started me rethinking how I developed software. Methods that had been tried and true for so long had failed and I was looking for some new ideas. At a new company I was re-introduced to the Scrum process for Agile development. I had heard of this before but to be honest I had dismissed it as the inmates running the asylum. Some of the engineering practices were appealing but overall I was fairly skeptical.

We did a pretty good job of adopting Scrum in all the wrong ways so I turned to Ken Schwaber's two definitive books on the topic: Agile Project Management with Scrum and The Enterprise and Scrum. To be honest I did not come away with much. Not that these are not good books but I tend to learn backwards in that I sometimes need the mechanics of something before I can relate it to the underlying concept. I then downloaded Henrik Kniberg's Scrum and XP From the Trenches and this book helped tie the concepts and mechanics together for me.

Phase Six: Agile Developer
On my next project I took all the things I had learned from our previous rough adoption of Scrum and applied it to our approach. I was lucky in that the management of my new client was very Agile friendly. I now really saw Scrum working and I started to concentrate on how to write my software in a more Agile way.

I attend a local Agile user group and won Practices of an Agile Developer. It was great in two ways because it not introduced some new concepts to me, it reaffirmed some of the lessons I had learned of the past few projects. It was a very easy to read book that provided a lot of information. At the next meeting I won the Art of Agile Development, which built upon everything I had just learned. Both these books made me want to learn Test Driven Development (TDD) better so I then grabbed Test Driven Development in Microsoft .NET.

I recently went back and reread Agile Principles, Patterns, and Practices in C# because when I was first given the book I was still not convinced about Agile and I really only skimmed the material. Going over it now had a much greater impact and there is so much great content that can be applied even outside of Agile that I suggest it to any developer regardless of their approach.

Phase Seven: The Empty Vessel
As I write this I have to say I am a little ashamed at how at some of these levels I thought I really knew everything I needed to know. It has made me really open myself up to learning as much as I can and figuring out the best way to apply that knowledge in my day to day activities. It was humbling to look at the publication dates on some of the books and think, "This has been around for awhile and I am just now learning it!"

There were more books along the way but I wanted to point out the ones that really pushed me to the next level of my development career and I hope to be constantly adding to this list for as long as I write software.

• Iteration Planning/Backlog Maintenance (-1): This is probably my most disappointing set of features. I never found the work item queries view in Team Explorer or Excel a decent interface for organizing the Product Backlog and Release/Sprint Planning. Products like RallyDev and VersionOne (which both have connectors to TFS) had easy to use, drag and drop user interfaces that made these features more tactile. Stephanie Saad shows the new Agile planning templates in this Channel 9 video. All that was really added were some Excel sections for capacity and release number crunching that we had implemented ourselves in Excel about a year ago. Microsoft may be simply relying on ISVs like Rally and VersionOne to fill in this space but I hate to buy another product (on top of VSTS/TFS's substantial cost) to get this functionality.




• Unit Testing/TDD Support (+1): While I am not a TDD zealot I do believe in test early and test often when it comes to unit testing. For those who really wanted to write their tests first in true TDD fashion it was a little kludgy to get a unit test class written without any of the references of target classes already defined. There are some new options while creating your unit tests to automatically create your target classes and namespaces as you go. Karen Lui demonstrates this in this Channel 9 video. These features are definitely going to make writing unit tests first easier.




• QA/Functional Testing (+1): This functionality I have not gotten to play around with as much so I have really only gotten an idea of how it works from this video. The new testing application, Camano, is a stand alone application that QA/functional testers can use to execute manual test and record them for automation afterwards. There are also the other cool features for recording video while testing and saving metadata about the path the tester took to get to the bug. All this info can be bundled up in a bug report that can be sent and tracked through TFS.
  
  
• Self Documenting Code (+1): Since there is less concentration on documenting architecture and design information up front in most Agile processes, techniques for making your code/application self documenting are highly recommended. But even these practices never gave you those great UML diagrams that really showed the "big picture" of how the solution as a whole was built. But now with some of the "bottom up" design features (demonstrated here), you can generate not only the standard class diagrams, but sequence diagrams also. The Architecture Explorer is also a very cool way to browse the basic structure of any .NET solution.
  
  



• Code Quality [Adherence to Architecture/Design] (+1): Enforcing code quality in a self managing Agile team is sometimes a difficult task. Pair programming and good code reviews are good ways to ensure the developers are following the basic architecture/design of the solution. The Layer Diagram (same video as above at the 30 minute mark) is a great way to layout the basic boundaries of your application that can actually enforce that the underlying code does not cross boundaries.
  
  

So far the Release/Sprint Planning and Backlog management is the only area I found wanting. Otherwise I am very eager to start using 2010!

The more I read on agile practices the more I hear some fairly edgy ideas about requirements gathering, design, and architecture in the agile world. While I understand the concept of embracing change, there has to be a line in the sand somewhere of how your company performs its business and how you build software to enable that business. As I work with clients adopting Scrum I find different levels of the YAGNI (Your Ain't Gonna Need It) principle and the idea of building extensible software that gracefully accepts change. My struggle is that I need to find a happy medium with the majority of my clients. Martin Fowler's article "Is Design Dead?" makes an argument for something in between also, but he focuses more with contrasting against the more edgy concepts in XP. Unfortunately a large number of agile pundits have taken up the idea that design up front is a wasteful practice as is gathering any type of detailed requirements. These are not just the inexperienced newbies blogging from their soap boxes, but well respected and experienced thought leaders in our industry.

So is up front requirements gathering and design really dead?

I have never been a agile purist and I think that for my clients some hybrid is more prudent. In the past I have been on Scrum teams that designed on the fly and kept the YAGNI mantra. Sometimes this resulted in some patchwork architectures and some disjointed business themes in the application. While I do think it is a great idea to embrace change, a basic vision for both your business case and architecture/design of any software solution are needed to keep the smaller iterations on at least some common path. This vision should be able to change as well, but there are some fundamental things that most probably will ring true for the solution. Determining that line is a black art. Too little and you risk being all over the place in your deliverables. Too much and you start becoming too rigid and have lost the benefits of an agile approach.

So how do you know how much up front work to do?

If I had a textbook answer I would give it to you. And it would most likely be wrong. Every case is different. Each client's situation and environment is different. As a company's business evolves the line might have to move also. But I do think that forming a basic business case as the initial vision and some high level architecture and design is almost always warranted. Even Ken Schwaber, one of the originators of Scrum, advises to start as soon as you have enough backlog items to start a sprint. You do not need grand functional specifications and loads of UML diagrams. As is often quoted "The plan is useless, but the planning is essential." When determining if you need to detail a specific business case ask yourself how essential is it to our core business? Higher value requirements always get the most attention. As far as architecture/design, defining the basic components and their interactions at a high level should be enough to ensure your deliverables follow a cohesive path. When determining how extensible a given component should be, concentrate of the most probable changes rather than inventing all sorts of possible future scenarios.

So how can you gather requirements and perform up front design while still being agile?

If you are going to start a sprint with just enough product backlogs items then some of those product backlog items should be forming high level business cases and prototypes so that subsequent sprints will have the benefit of an uniform vision on both fronts. I also believe that you do need some level of documentation for these outside of just the backlogs. A wiki is a great place for them since by its very nature embraces change (and the good ones can provide history for those people who just cannot get over the idea of this type of change). To take that idea to the next step, Team Foundation server allows you to add links to web pages on backlog items so that you can tie the documentation to the user functionality and development tasks. If you want to go low-tech use flip charts and save all the paper. Post it on the walls of the room where the team is literally surrounded by the business and architectural vision.

Even if you are well into an agile project, you can always have the stakeholders redefine the vision if you feel you are straying too far because of only thinking about the next sprint. Change is inevitable, but your company's core business fundamentally changing in not very probable. And good design should elegantly accept change also. I do not think up front gathering of requirements or design is dead. I just think that in agile environments it has evolved and shed its more useless features.

I have been presenting on Scrum and how I have implemented it processes using Team Foundation Server for the last year and almost every time I talk about it I run out of time. Both Scrum and TFS are complex entities and it is hard to talk about them both in one hour.

I am starting a new series of blog postings entitled "Comfortably Scrum" as a way to spend a little more time on a specific topic and do it justice. I eventually plan to accompany each blog post with a companion video.

I am currently working on the first post on gauging if your company is ready to adopt an agile process like Scrum.

More to come!

Since last year I have been at two different development shops that were in various stages of adopting Scrum. At the first I was the development manager and initial Scrum Master for a medium sized team that was moving from a waterfall/cowboy process to Scrum. Most recently I have been consulting with a smaller company that had previously loosely adopted Scrum, but has been bought by a larger company that has a more structured methodology. It has been a very educational experience.

One thing that I have learned is that a development team that wishes to get the most out of adopting Scrum must be a very disciplined team. It is sometime a misconception that agile methodologies are basically "the inmates running the asylum", but agile (and Scrum in my experience) requires a more mature and self-reliant team than waterfall. Scrum team are "self organizing" and are given almost total freedom to get their job done during each Sprint. They are left to determine what needs to be done, who works on what task, the approach for each item, etc. often with little to no outside influence.

This is quite a big responsibility and I have seen many people (and I include myself) struggle with it. Less mature developers may dive into a technical task and be oblivious to how their task relates to the project as a whole. Often they do not take advantage of the Daily Scrum meetings to get a good sense of how the team is progressing towards the current Sprint goal. These developers will often under estimate tasks and their deliverables will tend to be less on target.

Scrum Masters can fall into old habits also. As a Scrum Master I tended to want to "lead" the team rather than facilitate the team leading itself. Sometimes this was "muscle memory", but often I had to provide a more hands on approach when my team would start to drift away from the goals of a Sprint. I had to make a concentrated effort to let go and allow my team to run itself. I also am guilty of getting off track in the sacred Daily Scrum meetings. My team takes great joy in using my own phrase "Don't tell me about the pregnancy, show me the baby." when I do this.

I think that most every developer really knows what must be done to create a solid software solution, but what we struggle with is the discipline to execute in our daily activities. It is often easy at the beginning and then gets decidedly harder as the pressure increases. I personally struggle with this and am trying to make a real effort to make myself adhere to the principals that I believe make me a better developer.

Just this last Sprint I found myself continuing on a task that we had time boxed to a few days. At the end of the time period I was very close to wrapping it up and decided to extend the time period. Each time I got "close to finishing" another issue arose and I extended the time period again. I did finish the task and the result will greatly benefit the team, but other tasks were sacrificed. I also had to admit to myself that if another member of the team had done something similar, as the Scrum Master I would have probably called them out on it. It was another lesson it how to be more disciplined in my approach.

My hope is that each day I gain a little more experience and a little more practical approach to how I adopt (and sometimes DON'T adopt) agile/Scrum philosophies. That way the next day I become a more disciplined developer.