ARTICLES |
| ||
| ||
| ||
| ||
| ||
| ||
| ||
| ||
| ||
| ||
| ||
| ||
|
Development Methodologies - An Overview
Version: 1.2
Release Date: Oct 29, 2002
Author:Johan Borg, KEP3
Introduction
For a software project there exist dozens of different development methodologies (or lifecycle models) to choose from; anything from pure Ad Hoc programming (i.e. Code-and-Fix) via the strict risk analysis centric Spiral method to the "new" agile processes.
Here we will briefly describe the different development methodologies that we have experience with, and the advantages and disadvantages with each:
It must be emphasised that this is merely a brief discussion on these methodologies. There exists literally dozens of books on each of them and the goal is not to reiterate that information, but merely give an overview that can be used as a basis for further discussions.
Why use a Method?
A method is needed to:
- Verify the requirements (validation) and to make sure that the final system is correct (verification).
- Concentrate on the important/needed features and avoid confusing details.
- Create the formal intermediate documents that Quality Assurance and Traceability requires:
- Requirements Specification
- Design Documentation
- Test Cases / Reports
- Ensure communication within large development teams, especially if they are geographically dispersed.
Consensus it that any software project needs a development method based on modern programming practices - technical fundamentals. What must be evaluated for each and every project is how detailed this method should be.
Case example (opens in new window)
Minimum Effort
The minimum standard for a development "method" is to have a specification that contains requirements that are verifiable.
No design or coding except for the most trivial exercises should commence without a well-written detailed specification where the requirements are clearly defined and easy to create test cases from.
From this specification one can create a simple design, do the implementation and finally verify the result.
Maximum Effort
The maximum effort could be establishing a process that complies with certification (or military) requirements of for example the RTCA DO-178B. Or the extremely mission-critical software typically seen within space programs. Common is that it does not suffice to remain on the lowest level of the CMM ladder, i.e. chaos.
The Methods
Waterfall
The Waterfall method is the first published model of a software development process (1970). The basic principle is that the different processes (Analysis, Design, Coding, Testing) are done sequentially. Output from one process is input to the next.
Advantages
- The method is simple, easy to understand and work with.
- It creates very detailed intermediate documentation after each process. (Theoretically completely different teams, or even companies, can be responsible for the different processes.)
Disadvantages
- It just doesn't happen. Real projects tend not to follow a sequential flow.
- Does not accommodate changing requirements easily.
- Delivery only at the end of the project (could be years, which could be too late).
Summary
In general it is difficult to find situations where the Waterfall method is especially suitable. For the method to work the requirements have to be basically written in stone. It is never recommendable for any Graphical User Interface work. The situations we have seen where it can be suitable is for calculation modules within the manufacturing industry. These modules are typically based on sets of equations which do not change, the only thing that changes is the data.
Code-and-Fix
Code-and-Fix is what you end up with if you have not explicitly chosen a development method. It is very seldom (read never) useful, but yet commonly practised.
Advantages
- No Overhead (No planning, documentation, QA, standards enforcing, etc., just coding.)
- Requires no expertise; anybody can do this.
Disadvantages
- Extremely dangerous:
- No way of assessing progress
- No way of assessing quality
- No way of identifying risk
- If it is discovered after 3/4 of the way that the design is fundamentally flawed; just start all over again.
Summary
This "method" is best left for the extremely tiny "use and forget" programs. If you know that a task is small, and it is truly a one-off occasion, then maybe this approach can be used. For anything else it is just too dangerous.
Incremental Development
The Incremental Development (or Evolutionary Development or Staged Delivery) method was devised to counter the problems of the Waterfall method that the delivery comes very late and that requirements emerge at all stages of the project. With Incremental Development the system is delivered in increments, starting with the highest priority. The increments should be reasonably small. Project dependant of course, but the aim should be from a few weeks up to maximum two months between each release.
Advantages
- The highest priorities gets delivered quickly.
- Delivery every few weeks.
- Feedback from early increments give requirements for later increments.
- Lower risk of overall project failure.
- The highest priority systems tend to receive the most testing.
Disadvantages
- Might have to create temporary solutions to make a release.
- A lot of code may get discarded.
- Planning gets difficult.
Summary
There are very few cases where Incremental is not better than Waterfall. For any project of more than trivial size and/or complexity an incremental / iterative development process is to be preferred. When incremental development is the selected process directly from the start of a project it generally results in a more modular and flexible software design. Especially compared to the involuntary incremental that Waterfall typically suffers from. (I.e. by the time v1.0 of Waterfall is nearing release it is already apparent that it will not fulfill all requirements and must be updated. And this in areas where no change was anticipated originally.)
Prototyping
Prototyping (or Evolutionary Prototyping) is very useful when entering uncharted waters. When a new system concept or new technology is used it is extremely difficult to plan accurately and identify all the risks. Instead of proceeding with only a very vague idea about the outcome it may prove more efficient to create quick prototype(s) that can be further evaluated.
Advantages
- Learn more about requirements and exposes risks.
- User feedback very early.
- Very good for GUI work.
Disadvantages
- Users see a "working" software early and may not understand that it is built without any quality assurance or maintainability in mind.
- Difficult to make initial time and cost estimates.
- Easy to fall into trap of Code-and-Fix.
Summary
In our experience this is a very powerful development methodology especially for systems which are central components in a process change. In these situations specification documents or even pictures of dialogue windows does not suffice and a working prototype might reveal new and crucial requirements. Worth noting is that this method to a certain extent is similar to Extreme Programming in that it requires a very focused team that works outside of normal company policies. It is hardly possible to do prototyping with a geographically dispersed team.
Note that the prototyping referred to here is not prototyping as described by Fred Brooks in Mythical Man-Month [1]. Brooks proposed to "Build one to throw away". This has in general been accepted as perhaps too drastic for most applications. While it is true that large parts of a prototype might be re-written more than once during a development, the overall goal is to build an architecture that support those changes.
Spiral Development
The Spiral Development process encompasses other process models. For each iteration the sub-problem with the highest associated risk must be identified and solved. To solve the problem any type of "normal" development process (Waterfall, Incremental, Prototyping, etc.) might be employed. Which one is suitable all depends on the risks identified.
Advantages
- Takes a pro-active stance on risks with explicit risk analysis assessment and resolving stage.
- Very flexible.
Disadvantages
- Almost impossible to make initial time and cost estimates.
Summary
Suitable method for projects where risk must be understood and handled. The method is complex and time consuming and it is impossible to predict time and cost initially. Then again, experience has tought us that even with traditional methods such as waterfall, the initial predictions we do make are not necessarily very correct. (See Agile Processes below.)
Agile Processes
Most of us has lived through one or several projects with no process to guide it. The lack of process leads to unpredictability, wasted effort and repeated errors. It inflicts pain both for the customer (slipping schedules, growing budget and poor quality) and the developers (working even more overtime to produce even poorer software).
The response to this problem during the last two to three decades has been to seek steps to formalise the process; introduce rules and constraints that govern the activities and demands certain output and artifacts. But the problem is that software projects are not so simple that a few constraints and artifacts can prevent errors. So the work continue to analyse these errors and introduce more rules and constraints into the process to prevent those errors in the future.
After a number of projects we find ourselves overloaded with a huge cumbersome process that greatly impedes our ability to get projects done at all. The huge process suddenly creates the very same problems it was design to solve:
- Slows the team down to the extent that schedules slip and budgets bloat.
- Reduce the responsiveness of the team so that they always release the wrong product because the requirements have changed.
Unfortunately this leads many teams to believe they do not have enough process, so even more rules and constraints are added.
This state of affairs have been described as Runaway Process Inflation or Monumental Methodologies.
Why so difficult?
It is worthwhile to ponder for a moment over why it is so difficult to construct a process for software development. Fact is that constructing software entities are indeed more difficult than almost any other engineering task due to the inherent complexity, the need for conformity and changeability and the problem that software is basically unvisualizable. But even ignoring this it should be possible to have a process where once the requirements are defined, a design can be devised and then it is rather straightforward to construct and distribute.
The key phrase here is "once the requirements are defined". Statistics show that the typical project will experience a 25% change in requirements[2, 3]. Note that this is not to be read as "There is a 25% chance (risk) that any of our requirements will change", rather it means that "On average 25% of all our requirements for this project will change".
Clearly any huge cumbersome process that assumes that all the requirements are available early, concretely specified and immutable, is bound to fail.
It is also clear that a process is needed that allows software teams to develop quickly and to respond to changes.
The Agile Values
To respond to this challenge a group of industry experts formed the Agile Alliance and created the Agile Manifesto:
"We are uncovering better ways of developing software by doing it and helping others do it. We value:
~ Individuals and interactions over processes and tools ~
~ Working software over comprehensive documentation ~
~ Customer collaboration over contract negotiation ~
~ Responding to change over following a plan ~
That is, while there is value in the items on the right, we value the items on the left more."
Individuals and interactions over processes and tools
People are the most important ingredient of success. A good process will not save a team that does not have strong players; but a bad process can cause a strong team to fail. Similarily strong players can fail badly if they do not work as a team.
The right tools can be very important for success. However tools can easily be overemphasized. To have too many complicated, difficult to use tools, can be as bad as having no tools. Start out easy and only step up when it has been proven that it is really necessary.
Working software over comprehensive documentation
Software without documentation is a disaster. But software with too much documentation is worse. Huge documents take a great effort to produce, and an even greater effort to maintain in sync with the code. Once it is out of sync, it has turned into a huge lie that cause nothing but misdirection.
What one should have is a brief document describing overall design rationale and only the highest level structures of the system. Many teams have gotten caught up in the pursuit of documentation instead of software. Basic rule: Produce no document unless its need is immediate and significant. (The issue is after all understanding, not the documentation in itself.)
Customer collaboration over contract negotiation
Software cannot be ordered like a commodity. It is not possible to write a specification and then have someone develop it to a fixed schedule for a fixed price. Time after time this treatment of software projects have produced spectacular failures.
Successful projects are based on customer feedback on a regular and frequent basis. A successful contract should govern the way the developing team and the customer collaborate rather than details of scope and schedule for a fixed cost.
Responding to change over following a plan
It is the ability to respond to change that often determines the success or failure of a software project. It is impossible to predict the course of a software project too far into the future. There are just too many variables. A novice manager might be tempted to put the whole project into a nice PERT or Gant chart and tape it to the wall. This gives a sense of control, possibility to track task progress and cross them off the chart as they are done.
The truth is that the structure of the chart will degrade over time. The development team will learn more about the system and the customer will gain knowledge about their needs. Tasks will become obsolete, new tasks will be added. It is not just the dates that will change, it is the whole shape of the plan.
A better planning strategy is to only make detailed plans for the next few weeks, up to a month, very rough plans for the next couple of months and extremely crude plans beyond that. This gives strict guidance for immidiate work and a rough understanding of the requirements that will probably follow after that. Because of the lower resolution of the following requirements we are not heavily commited and can easily change when the requirements do. And they will.
Agile Principles
The four agile values inspired the following twelve principles. These principles are the characteristics that differentiate an agile process.
- Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
- Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
- Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
- Business people and developers must work together daily throughout the project.
- Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
- The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
- Working software is the primary measure of progress.
- Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
- Continuous attention to technical excellence and good design enhances agility.
- Simplicity--the art of maximizing the amount of work not done--is essential.
- The best architectures, requirements, and designs emerge from self-organizing teams.
- At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
Summary
The utmost professional goal for any software engineer or development team is to deliver the highest possible value to customers. And value for a customer is software that works. Not grand plans or documentation, but software. The agile processes aim to reduce the process bloat that has crept up during the last decades and instead allow development teams to concentrate on delivering value. Regardless whether one wants to follow a complete agile process, like Extreme Programming, SCRUM, Crystal or RUP (Rational Unified Process), or just parts, or not at all, it is very worthwhile to remember the values and principles of the agile manifesto.
Extreme Programming (XP)
XP is a lightweight, evolutionary (i.e. agile) software development process with the following core values:
- Simplicity
- Communication
- Feedback
- Courage
Very simplified XP can be seen as Incremental Development mixed with Prototyping, taken to the extreme. It is a reaction to the rigidity of the more formal traditional development processes available. With XP change is expected and embraced. In practise XP relies on a highly experienced team where programmers work in pairs producing code and having users constantly present to give feedback. Deliveries are very frequent (weekly or even daily).
Advantages
- Can produce very high quality software very quickly.
- User feedback always present.
- Change is expected and embraced.
Disadvantages
- Requires a very experienced team. (And the emphasis is on team.)
- Requires constant user interaction.
- Requires an organisation that "dares to do it".
Summary
XP is without doubt the most interesting development methodology that has surfaced during the last years. The requirements are high, but if they can be met, XP can deliver results like no other methodology. It is of course controversial to make statements such as "2 people working at a single computer will add as much functionality as two working separately except that it will be much higher in quality", but several independent investigations have proved the thesis. For more information on XP, please look at the Extreme Programming web site.
Acknowledgements
Ideas for this article has been taken from lots of people, too many to list. As mentioned in the introduction this is a mere summary of the ideas, concepts and methodologies that exists. For more information, please visit any of the links provided. Or better yet, read the writings of for example Martin Fowler, Kent Beck or Robert C. Martin.
Footnotes
[1] Brooks, Frederick P., Jr. 1975. The Mythical Man-Month.
[2] Jones, Capers. 1994. Assessment and Control of Software Risks.
[3] The information given in [2] is by now 8-10 years old. The only thing that is certain is that the situation has not become better.
