A fairly direct analogy can be made between the debate over evolution vs. intelligent design and agile development at large.
The debate from the perspective of intelligent design advocates says that life, in all it's complexity could never have evolved step-by-step, and must have had an intelligent designer acting behind the scenes.
From the perspective of evolutionary scientists, the argument is almost exactly the opposite. It simply does not make sense to them that life, in all it's complexity, appeared from nothing (or rather from some already unimaginably complex being). Rather, it is said, life must have evolved slowly, step-by-step, piece-by-piece over very long periods of time.
In natural biology, evolution is in my mind a more reasonable and defensible explanation of how the world, in all it's wonder, came into being. Ironically however, in software design, we need a hybrid of the two the concepts to develop great software.
Waterfall & Intelligent Design
In my mind, waterfall is very much on the same page as what drives the model of intelligent design. It assumes that the designers (the software architects, product stakeholders, developers, etc) can produce, at the onset of the project, a holistic design that addresses all of the needs of the eventual end-users.
It further assumes that these intelligent designers can then spend 8-12 months executing said designs, and produce exactly what was originally intended.
It goes on to assume that when the end-users finally consume the product of this intelligent design, the changes required at that point will be small and inconsequential, and therefore the software will be "done" - because, after all, it was intelligently designed from the start.
Evolution & the Agile Development Model
Evolution, on the other hand, makes progress through the process of natural selection. If one "copy" of a species has a random mutation that makes it more well adapted to a particular climate, or circumstance, it is more likely to survive to the age of reproduction. This benefit is then passed on to it's children, their children and their children's children, and eventually the entire population has this new beneficial trait.
Agile development takes an approach to software design and implementation that much more closely resembles evolutionary processes. Rather than coming up with one massive plan for the entire project at the beginning, the project is split up into many small "generations" or "sprints". Each generation "evolves" from the previous one, improving on it, and incrementally, step-by-step, a sophisticated system is born.
Ironically, in the case of software development, it is actually a hybrid of the two models that is needed to produce great results. Specifically, there is no "survival" as such in the software design process. There are no opportunities for natural "mutations" to occur. Instead, while the software definitely evolves incrementally in an agile shop, it requires intelligent designers to "guide" this evolution along the way. Software engineers must introduce the mutations from generation to generate (i.e. write more code, analyze the results, rinse & repeat the process again and again), and in this way guide the evolution of the overall product being developed.
How this relates to Systemically Incremental Programming (SIPing)
As I've stated in a number of previous articles, I find it ironic that even staunch agile development advocates seem to revert back to a waterfall type of approach when it comes down to the individual bugs within a sprint.
What I've observed, is that even when an organization at large adopts and implements an agile approach to development, this in no way addresses how each developers handles 4-8 hours of work placed in front of them. An individual bug might include:
- Business Logic
- Network communication
- A data-access layer
- UI Development
- Other?
A plan for how to implement the bug might include sub-topics under each of these high level areas that address how each of those areas of the code will be addressed. As the bug gets bigger and bigger, this "tree" diagram of functionality gets bigger as well. In the end, even if it is only 10-15 hours of work, it starts to look more and more like a waterfall type of "intelligent design" for how the bug will be implemented.
Typically, such an organizational model in no way indicates the order in which the functionality will be implemented. It's usually heavy on design, and light on results. If the implementation of the data-access layer runs into glitches, the overall implementation is effected (goes long/over budget), causing the entire bug to quickly explode in terms of time and complexity. This in turn potentially holds up other important processes in the larger development effort.
A SIPed view of the same problem
SIPing approaches this bug differently. As with agile development in general, it takes a 15 hour bug, and splits it up into 10-15 one hour "sprints", or "SIPs". Each SIP might touch multiple functional areas described by the "intelligent design" originally proposed. Each SIP however would be an incremental improvement or evolution over the previous SIP. Each SIP introduces a new beneficial trait that did not exist in the software before. Each SIP is small, easy to understand, implement, test and demonstrate.
What you have to "buy" however, is that even a very sophisticated system can evolve in this way. They do not need to be designed from the top down. In fact, the benefits of this evolutionary process are the same at the bug-by-bug level as they are in the organization at large.
A SIPed plan for the same bug would take the 10 hours of work and say - in the first 2 hours, we will get a working, functional model. It will not do everything that needs to be included in the final version, but it will verify the overall process, it will be simple and easy to write, and it will be functional enough to present to the stakeholder for verification that we're heading in the right direction.
NOTE: Whether or not an outside stakeholder is shown this early version will depend on the specific feature being developed. At the very least, the developer himself is always a stakeholder, and he or she will have the opportunity to actually see this "rough draft" and make a decision about which direction to head next.
After this initial version is implemented, the entire process is re-evaluated - just like the process between sprints in an agile process. The remaining functionality is re-considered with respect to this initial "draft", some new "features" might be added to the list, others might be de-prioritized, and a second pass is then undertaken.
As with agile at large, the specific bug, defect or new feature being addressed evolves SIP-by-SIP, rather than being developed as a single, intelligently designed module - which is simply never as easy as it often appears to be at the onset.
