Evolution vs. Intelligent Design / Agile Development

SIPing: Systemically Incremental Programming


A fairly direct analogy can be made between the debate over evolution vs. intelligent design and agile development at large.

The debate from the perspective of intelligent design advocates says that life, in all it's complexity could never have evolved step-by-step, and must have had an intelligent designer acting behind the scenes.

From the perspective of evolutionary scientists, the argument is almost exactly the opposite.  It simply does not make sense to them that life, in all it's complexity, appeared from nothing (or rather from some already unimaginably complex being).  Rather, it is said, life must have evolved slowly, step-by-step, piece-by-piece over very long periods of time.  

In natural biology, evolution is in my mind a more reasonable and defensible explanation of how the world, in all it's wonder, came into being.  Ironically however, in software design, we need a hybrid of the two the concepts to develop great software.


Waterfall & Intelligent Design
In my mind, waterfall is very much on the same page as what drives the model of intelligent design.  It assumes that the designers (the software architects, product stakeholders, developers, etc) can produce, at the onset of the project, a holistic design that addresses all of the needs of the eventual end-users.

It further assumes that these intelligent designers can then spend 8-12 months executing said designs, and produce exactly what was originally intended.  

It goes on to assume that when the end-users finally consume the product of this intelligent design, the changes required at that point will be small and inconsequential, and therefore the software will be "done" - because, after all, it was intelligently designed from the start.


Evolution & the Agile Development Model
Evolution, on the other hand, makes progress through the process of natural selection.  If one "copy" of a species has a random mutation that makes it more well adapted to a particular climate, or circumstance, it is more likely to survive to the age of reproduction.  This benefit is then passed on to it's children, their children and their children's children, and eventually the entire population has this new beneficial trait.

Agile development takes an approach to software design and implementation that much more closely resembles evolutionary processes.  Rather than coming up with one massive plan for the entire project at the beginning, the project is split up into many small "generations" or "sprints".  Each generation "evolves" from the previous one, improving on it, and incrementally, step-by-step, a sophisticated system is born.

Ironically, in the case of software development, it is actually a hybrid of the two models that is needed to produce great results.  Specifically, there is no "survival" as such in the software design process.  There are no opportunities for natural "mutations" to occur.  Instead, while the software definitely evolves incrementally in an agile shop, it requires intelligent designers to "guide" this evolution along the way.  Software engineers must introduce the mutations from generation to generate (i.e. write more code, analyze the results, rinse & repeat the process again and again), and in this way guide the evolution of the overall product being developed.


How this relates to Systemically Incremental Programming (SIPing)
As I've stated in a number of previous articles, I find it ironic that even staunch agile development advocates seem to revert back to a waterfall type of approach when it comes down to the individual bugs within a sprint.

What I've observed, is that even when an organization at large adopts and implements an agile approach to development, this in no way addresses how each developers handles 4-8 hours of work placed in front of them.  An individual bug might include:

• Business Logic
• Network communication
• A data-access layer
• UI Development
• Other?

A plan for how to implement the bug might include sub-topics under each of these high level areas that address how each of those areas of the code will be addressed.  As the bug gets bigger and bigger, this "tree" diagram of functionality gets bigger as well.  In the end, even if it is only 10-15 hours of work, it starts to look more and more like a waterfall type of "intelligent design" for how the bug will be implemented.

Typically, such an organizational model in no way indicates the order in which the functionality will be implemented.  It's usually heavy on design, and light on results.  If the implementation of the data-access layer runs into glitches, the overall implementation is effected (goes long/over budget), causing the entire bug to quickly explode in terms of time and complexity.  This in turn potentially holds up other important processes in the larger development effort.

A SIPed view of the same problem

SIPing approaches this bug differently.  As with agile development in general, it takes a 15 hour bug, and splits it up into 10-15 one hour "sprints", or "SIPs".  Each SIP might touch multiple functional areas described by the "intelligent design" originally proposed.  Each SIP however would be an incremental improvement or evolution over the previous SIP.  Each SIP introduces a new beneficial trait that did not exist in the software before.  Each SIP is small, easy to understand, implement, test and demonstrate.

What you have to "buy" however, is that even a very sophisticated system can evolve in this way.   They do not need to be designed from the top down.  In fact, the benefits of this evolutionary process are the same at the bug-by-bug level as they are in the organization at large.  

A SIPed plan for the same bug would take the 10 hours of work and say - in the first 2 hours, we will get a working, functional model.  It will not do everything that needs to be included in the final version, but it will verify the overall process, it will be simple and easy to write, and it will be functional enough to present to the stakeholder for verification that we're heading in the right direction.  

NOTE: Whether or not an outside stakeholder is shown this early version will depend on the specific feature being developed.  At the very least, the developer himself is always a stakeholder, and he or she will have the opportunity to actually see this "rough draft" and make a decision about which direction to head next.

After this initial version is implemented, the entire process is re-evaluated - just like the process between sprints in an agile process.  The remaining functionality is re-considered with respect to this initial "draft", some new "features" might be added to the list, others might be de-prioritized, and a second pass is then undertaken.

As with agile at large, the specific bug, defect or new feature being addressed evolves SIP-by-SIP, rather than being developed as a single, intelligently designed module - which is simply never as easy as it often appears to be at the onset.

Sipping and Test Driven Development (TDD)

Sipping: Systemically Incremental Programming


Overview
Sipping and Test Driven Development  may initially sound like they are one and the same thing.  Indeed, they are closely related, and aim to solve many of the same underlying problems.  Additionally, while not mutually exclusive, they do compliment each other well, and using TDD within the Sipping framework will produce excellent results.

That being said, they are different.


First though, how is Sipping similar to TDD
Many of the benefits associated with TDD are also present when using the Systemically Incremental Programming.  The benefits section of TDD on wikipedia served as the basis for this list of benefits of sipping:

Sipping drives the design of a program. By focusing on a complex problem SIP by SIP, one must imagine how the functionality will be used by clients, and how all of the SIPs will fit together, creating one, coherent system. 

As described in Ordering your SIPs, the programmer is concerned with the interface before the implementation. This benefit is complementary to Design by Contract as it approaches code through test cases rather than through mathematical assertions or preconceptions.

Sipping offers the ability to take small steps when required. It allows a programmer to focus on the task at hand, as all other complexities and distractions have been pushed into other, subsequent SIPs. Exceptional cases and error handling are not considered initially, and SIPs to create and test these extraneous circumstances are implemented separately. 

Unlike TDD, simple Sipping does not typically require more code, while, I believe, still providing many of the benefits of TDD that make the implementation times for TDD shorter.^[10] Large numbers of individual SIPs help to limit the number of defects in the code in the same way that large numbers of Tests under TDD do. The early and frequent nature of the testing helps to catch defects early in the development cycle, preventing them from becoming endemic and expensive problems. Eliminating defects early in the process usually avoids lengthy and tedious debugging later in the project.  TDD and Sipping share this benefit equally.

Sipping tends to lead to more modularized, flexible, and extensible code. This effect often comes about because the methodology requires that the developers think of the software in terms of small units that can be written and tested independently and integrated together later. This leads to smaller, more focused classes, looser coupling, and cleaner interfaces. The use of the mock object design pattern also contributes to the overall modularization of the code because this pattern requires that the code be written so that modules can be switched easily between mock versions for testing and "real" versions for deployment.

Because no more code is written than necessary to complete the SIP, it is generally easier for the developer to actually test all of this code, which he does as a matter of practice, before committing each SIP. 

How is Sipping different from TDD?
Specifically, I would categorize TDD as a subset of Sipping.  In other words, TDD does not specifically address how to break a problem apart.  Instead, it describes what to do once each "test" has been identified.  The "failing test" referred to in TDD would typically be the goal of a single SIP.

Where TDD seeks to describe how to implement one SIP, Systemically Incremental Programming describes the process of how to take complex problems, and pull them apart into separate SIPs in the first place.

TDD is also a relatively highly structured process, that relies on automated testing systems (Unit Testing for example).  Sipping is a more general set of guidelines that aims to solve a similar problem, but does not address directly how each "SIP" should be validated.  TDD (in many cases) will be an excellent choice for how to validate that the SIP has been completed to spec, but sipping works even if Testing is not the method of validation

Interestingly, going back to Wikipedia, virtually all of the "Vulnerabilities" listed for TDD do not apply to Sipping because most of the vulnerabilities of TDD are related to the overhead of initially creating and maintaining the TDD tests.  Sipping does not require this overhead.  Again, if used in conjunction with TDD, they may still be present - but they are absolutely not a prerequisite of Sipping.

The one vulnerability that both share (which was, however, not listed as a vulnerability of TDD) is that the incremental design process that both encourage, does not necessarily result in the most efficient overall system design.  Subsequent refactoring of the code may be required if performance is identified at any point during the testing process.  See Sipping, Code Efficiency and Preemptive Optimization for more information on this issue.

Conclusion
As you can see, Sipping TDD are closely related to one another, but not mutually exclusive.  The biggest difference is that in it's fundamental methodology design, Sipping does not require unit tests to be written for each SIP, but rather, this would be an optional extension to the implementation of an individual SIP.  This reduces the required overhead to starting using the Sipping method - as it can be applied to any project, any time, anywhere - giving SIP developers many of the advantages of TDD, without the test code overhead.

Ordering your SIPs

Sipping: Systemically Incremental Programming


Overview
The process of splitting up a project into SIPs is just the first part of the process.  Once the individual SIPs have been identified, the next step is deciding the order in which to implement the SIPs.
This is a critical part of Sipping.  Usually, the first SIP to implement is the one that most quickly hooks up the desired functionality into a system that can be used to test it.  Once this high level SIP has been implemented, more detailed SIPs can be implemented to further test and extend the functionality original developed.
Let's take a sample project.  We are going to write a file search tool that will:

• Look through a directory (and optionally sub directories) for files with specific names (including wild cards)
• Be able to filter for specific file types (extensions)
• Be able to look in specific file types for a text
• Be able to sort the results according to one or more sorting algorithms

Source Files

The FileSearcher class
This problem can be broken down into a variety of SIPs.   In thinking about the problem, I find it helpful to first decide the overall system architecture.  In this case, I think that I would like to design a class called FileSearcher.  This class will collect all of the related configuration data such as RootDirectory, SortOrder, ValidExtensions, TextExtensions, etc into one place.
During this early thought process, I make some high level design decisions.  These are by no means final decisions.  Quite to the contrary, they are simply goals or objectives that will help to guide the design process.  At this point, all I can do is to image a very fully functional class called FileSearcher that will provide (centrally) all of the functionality that we're looking to incorporate into our model.  As we add new features, this class will provide the framework that all of our new functionality will be added to.
With this high level design in mind, the first SIP that I will design (and code by the way) is the underlying framework that all of the other features will be built on.  In other words, in the first SIP I almost always want to have a functional prototype of the final design.  It usually won't do most of what I actually want - but it will do a basic implementation of the final design - and hopefully will work in a way that is as close to the final design as possible.
For the first SIP, I usually skip to the last step, and design the framework of the final class that I want.  In this case, here's what my first sip looks like...


SIP 1: Create a basic FileSearcher class
The objective of this step is to create a functional enough FileSearcher class that it can actually be used.  I will create a simple UI to test my class.  My plan with SIP 1 is to create the following functionality:
FileSearcher:

• Create the FileSearcher class
• Add a String RootDirectory { get; set; } property that will define the root of the search area
• Define an event called FileFound that will be fired each time a matching file is found
• Define a void Search() method that will start a search of the RootDirectory and fire the FileFound event for each file found.

That's it for the FileSearcher class.  It will (as you can see) still not filter files, search sub-directories, sort or really do any of the other features that we might want to implement.  But it will do something, and will let us design a UI to test our class.

Testing UI:

• A windows form with a ListBox, Button and FileSearcher property.
• When the button is clicked, it will clear the ListBox's Items, and call FileSearcher.Search().
• In the FileFound event handler, the file found will simply be added to the ListBox.

Again - that's it.  Once implemented, the application can be run.  Since the RootDirectory property is set (by default) to 'C:\', what should show up in the ListBox when the Search button is clicked, is a list of files in the root of your C Drive.  Here's the code:

FileSearcher.cs

    1public class FileSearcher
    2{
    3    public FileSearcher()
    4    {
    5        this.RootDirectory = "C:\\";
    6    }
    7    /// <summary>
    8    /// Event that will be fired each time a new file is discovered
    9    /// </summary>
   10    public event EventHandler FileFound;
   11    protected virtual void OnFileFound(FileInfo fi)
   12    {
   13        if (this.FileFound != null) this.FileFound(fi, EventArgs.Empty);
   14    }
   15
   16    /// <summary>
   17    /// RootDirectory to search for files in
   18    /// </summary>
   19    public String RootDirectory { get; set; }
   20
   21    /// <summary>
   22    /// The main search method
   23    /// </summary>
   24    public void Search()
   25    {
   26        // Search the home Directory
   27        this.SearchDirectory(this.RootDirectory);
   28    }
   29
   30    /// <summary>
   31    /// Internal search method that will fire the file found event
   32    /// </summary>
   33    /// <param name="directory"></param>
   34    private void SearchDirectory(string directory)
   35    {
   36        // Search through the files in the directory
   37        foreach (String file in Directory.GetFiles(directory))
   38        {
   39            this.OnFileFound(new FileInfo(file));
   40        }
   41    }
   42}

At this point, the class is very simple.  In fact, by design, it is as simple as I can possibly make it while still providing enough functionality to actually make it a useful enough class to be testable.
By keeping it so simple and focused, my goal is to make it the best possible class that I can at this point.  All of the directory names should make sense, work together, be cohesive, and of course - functional.  The most important decision being made at this point is, what's the best overall architecture.  Questions that could be asked at this point include:

• Do I want to pass parameters into the Search() method, or as implemented here, have all of the configuration options (like the RootDirectory property) as class level properties?
• Do I want to have a SearchResultEventArgs class, and make the FileFound event an EventHandler<SearchResultEventArgs> instead of a simple EventHandler event?  This is definitely something that will be examined in future SIPs, and there is a high probability that this will be enhanced later.  For now, a simple FileInfo as the sender is totally sufficient for our purposes.

And the consuming form Form1.cs:

    1public partial class Form1 : Form
    2{
    3    public Form1()
    4    {
    5        InitializeComponent();
    6
    7        // Respond when a file is found
    8        this.FileSearcher.FileFound += new EventHandler(FileSearcher_FileFound);
    9    }
   10
   11    /// <summary>
   12    /// Add any files found to the searchResults ListBox
   13    /// </summary>
   14    void FileSearcher_FileFound(object sender, EventArgs e)
   15    {
   16        searchResults.Items.Add(sender);
   17    }
   18
   19    /// <summary>
   20    /// The FileSearcher object that will do all of the searching
   21    /// </summary>
   22    private FileSearcher FileSearcher = new FileSearcher();
   23
   24    /// <summary>
   25    /// What to do when the Search button is clicked
   26    /// </summary>
   27    private void searchButton_Click(object sender, EventArgs e)
   28    {
   29        this.searchResults.Items.Clear();
   30        this.FileSearcher.Search();
   31    }
   32}

All the Form1 does is call FileSearcher.Search(), and then add the files found to the listbox.  At this point, you can't configure the RootDirectory, and no other search features are implemented.

The goal of the first SIP is almost always to put enough of a framework in place, that the rest of the features desired can be easily tested.  The order of the rest of the SIPs becomes less important, but I will lay out the rest of the SIPs as I would be inclined to implement them.

• SIP 2: Recursion
• SIP 3: File Pattern Matching
• SIP 4: Valid Extension Restriction
• SIP 5: Text File Searching
• SIP 6: Sorting

With this list in front of us, I'd like to point something out.  For simply listing files, using a simple EventHandler is totally sufficient.  This is why, for the first pass, using a simple EventHandler made more sense.  When we add recursion, this EventHandler will still work.  Even after SIPs 3 & 4, when we've added the ability to search for files with specific name and extension filtering, it should still work.

Once we get to SIP 5 however, and are searching through text files for specific text, we might need a SearchResultEventArgs that can provide more information than a simple FileInfo can convey.  Specifically, I can picture wanting to return details about what matched, a list of matching lines, portions of the matching text, etc.

Preemptive Optimization

It is very important to point out that while we could preemptively put this structure into place now, at this point we don't really know enough to actually do that effectively.  To some extent, the Sipping method is specifically designed to prevent this kind of work from happening "before it's time".  Instead, that change should be made systemically, as an incremental change to the FileSearcher class, when we get to that SIP!

There's no need to rush - we'll get there soon enough.  Doing it now would simply confuse the process, make the current task more complicated, and at this point, really not add anything to the equation (other than confusion/complexity).

Putting this structure in place now (when it is not yet needed) would be overkill.  If all we ever implemented was this first FileSearcher that could simply search 1 directory of files, with no filtering or sorting of any kind - the EventHandler delegate is totally adequate.  Leave it at that for now, and trust that it will be easy enough to add (as it's own SIP) down the road.

Ready for the next SIP?  Let's incrementally add the ability to search sub directories.


SIP 2: Sub-Directories
I would now like to add the ability to search through sub-directories.  I don't necessarily want to always search recursively, but the class should have the ability to do so.  At this point, because I don't want to have to search through my entire hard drive every time I click search, I'll also update the testing UI configure a smaller RootDirectory, and turn ON the RecursiveSearch property.
FileSearch.cs

    1public class FileSearcher
    2{
    3...
    4    /// <summary>
    5    /// Determines if the search algorithm searches recursively
    6    /// </summary>
    7    public Boolean RecursiveSearch { get; set; }
    8
    9    /// <summary>
   10    /// Internal search method that will fire the file found event
   11    /// </summary>
   12    /// <param name="directory"></param>
   13    private void SearchDirectory(string directory)
   14    {
   15        // If we are searching recursively, look through directories first
   16        if (this.RecursiveSearch)
   17        {
   18            foreach (String subDirectory in Directory.GetDirectories(directory))
   19            {
   20                this.SearchDirectory(subDirectory);
   21            }
   22        }
   23
   24        // Search through the files in the directory
   25        foreach (String file in Directory.GetFiles(directory))
   26        {
   27            this.OnFileFound(new FileInfo(file));
   28
   29        }
   30    }
   31}

This adds to the file searcher class a Boolean RecursiveSearch property that, when true, will search through Subdirectories of the RootDirectory in addition to what it did before.  This adds  ~5 new lines of code, and a huge amount of new functional. 
Now we test - and ensure that we don't just get the RootDirectory files, but also all subfolders and files.  We might also at this point test, that if we set the RecursiveSearch to false, that again, we only get the RootDirectory files.
Our class is starting to take shape.  Let's take another SIP.


SIP 3: File pattern matching
Lets add the ability to filter the exact files that we're searching for.  Given the infrastructure that we've already put in place, this should be another minor tweak (Systemic Incremental Programming change) to the system.

    1public class FileSearcher
    2{
    3...
    4    /// <summary>
    5    /// Pattern of files to search for
    6    /// </summary>
    7    public String SearchPattern { get; set; }
    8
    9    /// <summary>
   10    /// Add an overloaded search that will set the SearchPattern to look for.
   11    /// </summary>
   12    /// <param name="searchPattern"></param>
   13    public void Search(String searchPattern)
   14    {
   15        this.SearchPattern = searchPattern;
   16        this.Search();
   17    }
   18...
   19    public void Search()
   20    {
   21        // Make sure we have a search pattern
   22        if (String.IsNullOrEmpty(this.SearchPattern)) this.SearchPattern = "*.*";
   23
   24        // Search the home Directory
   25        this.SearchDirectory(this.RootDirectory);
   26    }
   27...
   28    private void SearchDirectory(string directory)
   29    {
   30...
   31        // Search through the files in the directory
   32        foreach (String file in Directory.GetFiles(directory, this.SearchPattern))
   33...
   34    }
   35}

You may notice that when adding the "SearchPattern" property, I also chose to add an overloaded version of Search that could update this search pattern for us.  This is an improvement that was "easy" to see, because adding support for search patterns was all I was thinking about that this phase (SIP).  
Our system can now search for files matching a specific search pattern and can search recursively.  It still can't look for specific extensions (except as provided by the search pattern),  search for text inside of files or sort the results.  Those will come.  Let's add Extension filtering next.


SIP 4: Valid Extension Filtering
For this feature, I'm picturing a separate filter entirely from the SearchPattern.  Basically, I'm picturing a List<String> ValidExtensions { get; } property can that be used to set valid extensions for the overall search.  In looking at our current system, the most logical place for this seems to be in the OnFileFound method.  In other words, right before the event gets fired, make sure that the file being returned matches our list of valid extensions (if any are present).
Here again, we should design a simple, powerful, well tested feature, because this is only thing that we're thinking about in SIP 4.  Everything else already works.  It should also be really easy to tweak out testing UI to validate that what we put in place does in fact, systemically modify our class, exactly as expected.
FileSearcher.cs:

    1public class FileSearcher
    2{
    3    /// <summary>
    4    /// Event that will be fired each time a new file is discovered
    5    /// </summary>
    6    public event EventHandler FileFound;
    7    protected virtual void OnFileFound(FileInfo fi)
    8    {
    9...
   10        if (this.FileFound != null)
   11        {
   12            // Check if there are any valid extensions specified
   13            if (this.ValidExtensions.Count > 0)
   14            {
   15                // Return if the extension isn't a valid extension
   16                if (this.ValidExtensions.Count(ext => 
   17                    ext.ToLower() == fi.Extension.ToLower()) == 0) return;
   18            }
   19            this.FileFound(fi, EventArgs.Empty);
   20        }
   21    }
   22
   23   /// <summary>
   24    /// List of valid extensions to filter our search with
   25    /// </summary>
   26    public List<String> ValidExtensions { get; private set; }
   27...
   28}

This SIP adds a new List<String> ValidExtensions { get; } property that can be used to filter the final search results to a specific list of valid extensions.  If any ValidExtensions are specified, then it now counts how many of those extensions match the current file, and if 0 match, it returns before firing the FileFound event.
Simple, minor, incremental change to the system, which can now be fully tested against our previous code to verify that it works, just as expected.
I think we're ready for SIP 5 - text file searching.  

SIP 5: Text Searching
An interesting thing happened when implementing this SIP which is A) Not uncommon when Sipping and B) Yet another reason that sipping proves to be so effective.  Specifically, while originally, this feature seemed superficially at least to be a relatively easy thing to implement, a variety of complications showed themselves, which have caused me to add 4 more SIPs to our plan.
Specifically, when first approaching this SIP I quickly realized that to begin with, I was going to leave the existing EventHandler in place, and push the more sophisticated EventHandler<SearchResultEventArgs> implementation to a later SIP.  This decision was based on the fact that this SIP was already complicated enough, without adding the complexity of restructuring how the event actually gets fired.
Also, when I first tested the SIP, it included a variety of files that were not text files.  When they weren't text files, I was not searching them, but I was including them in the result.  This was obviously a mistake (bug), but because this was the only feature that I was focusing on, I realized that this could (in certain situations at least) actually be the desired behavior.  In other words, I am going to add a SIP at the end of my list that will add a new property called ExcludeNonTextFilesFromSearch (which will default to true).  If it was set to true, all matching Non-text files would be included, but only text files that matched would be included.
Another "bug" that was exposed through testing was that I had a TextExtension that was not in the list of ValidExtensions.  Again, as a later SIP I could add a feature that said "bool AssumeTextExtensionsAreValid"
In summary, completing this SIP in the simplest way that I could, created 3 new SIPs to add to our list
New SIPs:

• SIP 7: SearchResultEventArgs
• SIP 8: Add bool AssumeTextExtensionsAreValid property
• SIP 9: Add bool ExcludeNonTextFilesFromSearch property.

I choose not to implement these now to keep my changes small, targeted and incremental.  I will come back to these features at a later time though.

    1public class FileSearcher
    2{
    3...
    4    protected virtual void OnFileFound(FileInfo fi)
    5    {
    6        if (this.FileFound != null)
    7        {
    8...
    9            // If TextFileSearchString string is specified, try to search for it
   10            if (!String.IsNullOrEmpty(this.TextFileSearchString))
   11            {
   12                // See if this is a valid text file to look through
   13                if (this.TextExtensions.Count(ext =>
   14                    ext.ToLower() == fi.Extension.ToLower()) > 0)
   15                {
   16                    // Search strings
   17                    String searchString = String.Empty;
   18                    String fileContents = String.Empty;
   19
   20                    // Check if the match should be done case insensitively
   21                    if (this.CaseInsensitiveTextFileSearch)
   22                    {
   23                        searchString = this.TextFileSearchString.ToLower();
   24                        fileContents = File.ReadAllText(fi.FullName).ToLower();
   25                    }
   26                    else
   27                    {
   28                        searchString = this.TextFileSearchString;
   29                        fileContents = File.ReadAllText(fi.FullName);
   30                    }
   31
   32                    // If the file doesn't contain the text, return
   33                    if (!fileContents.Contains(searchString)) return;
   34                }
   35                // If this isn't a text file, don't include it in the results
   36                else return;
   37            }
   38
   39            // Fire the FileFound event
   40            this.FileFound(fi, EventArgs.Empty);
   41        }
   42    }
   43...
   44    /// <summary>
   45    /// List of extensions that should be searched as text
   46    /// </summary>
   47    public List<String> TextExtensions { get; private set; }
   48
   49    /// <summary>
   50    /// Specific text to search for within text files
   51    /// </summary>
   52    public String TextFileSearchString { get; set; }
   53
   54    /// <summary>
   55    /// Should text file searches be case insensitive?
   56    /// </summary>
   57    public Boolean CaseInsensitiveTextFileSearch { get; set; }
   58...
   59}

Conclusion
I am not going to put all of the SIPs into this article, but you can see how the problem evolves slowly, SIP by SIP.  When thinking about the order of your SIPs, I would suggest the following guidelines:

1. The most important SIP is the 1st one.  The objective of the first SIP should be to put the basic framework in place so that the rest of the SIPs can be tested, one by one, as they are developed.
2. The order of the rest of the SIPs is substantially less important.  That being said, I tend to implement the simplest ones first, so that by the time I'm getting to the more complicated SIPs, the system has already been well tested, and logical inconsistencies in the design have been worked out under simpler scenarios.  In this way, when I get to the more complicated SIPs (like text searching in this case), the rest of the system was already in place, and I could really focus on just the text searching, which was already complicated enough.
3. Look for opportunities to split your SIPs.  Often, items that appear to be one SIP when starting a project, turn out to be 3, 4 or even more SIPs when you actually dig into the code.  This is what Sipping is designed to handle.  When you're not sipping, as a step gets more complicated, traditional approaches usually just try to juggle more and more balls, and often we end up with Spaghetti.  The whole point of sipping is that we have a system that is specifically designed to allow us to say "This is getting complicated, let me push these other 3 'features' to a later SIP."  They are almost always good ideas, but getting distracted with new ideas is usually what gets developers into trouble.  If it's really a good idea, it should fit beautifully into the system if/when we choose to tackle that new feature - as it's own SIP!

Hopefully this has helped to further explain how Sipping works, and how the SIP implementation order can be used to keep your project On-track, On-budget, and defect free!

The Final Code:

    1public class FileSearcher
    2{
    3    public FileSearcher()
    4    {
    5        this.RootDirectory = "C:\\";
    6        this.RecursiveSearch = true;
    7        this.ValidExtensions = new List<String>();
    8        this.TextExtensions = new List<String>(new String[] { ".txt" });
    9    }
   10
   11    /// <summary>
   12    /// Event that will be fired each time a new file is discovered
   13    /// </summary>
   14    public event EventHandler FileFound;
   15    protected virtual void OnFileFound(FileInfo fi)
   16    {
   17        if (this.FileFound != null)
   18        {
   19            // Check if there are any valid extensions specified
   20            if (this.ValidExtensions.Count > 0)
   21            {
   22                // Return if the extension isn't a valid extension
   23                if (this.ValidExtensions.Count(ext =>
   24                    ext.ToLower() == fi.Extension.ToLower()) == 0) return;
   25            }
   26
   27            // If a TextFileSearchString string specified, try to search for it
   28            if (!String.IsNullOrEmpty(this.TextFileSearchString))
   29            {
   30                // See if this is a valid text file to look through
   31                if (this.TextExtensions.Count(ext =>
   32                    ext.ToLower() == fi.Extension.ToLower()) > 0)
   33                {
   34                    // Search strings
   35                    String searchString = String.Empty;
   36                    String fileContents = String.Empty;
   37
   38                    // Check if the match should be done case insensatively
   39                    if (this.CaseInsensitiveTextFileSearch)
   40                    {
   41                        searchString = this.TextFileSearchString.ToLower();
   42                        fileContents = File.ReadAllText(fi.FullName).ToLower();
   43                    }
   44                    else
   45                    {
   46                        searchString = this.TextFileSearchString;
   47                        fileContents = File.ReadAllText(fi.FullName);
   48                    }
   49
   50                    // If the file doesn't contain the text, return
   51                    if (!fileContents.Contains(searchString)) return;
   52                }
   53                // If this isn't a text file, don't include it in search results
   54                else return;
   55            }
   56
   57            // Fire the FileFound event
   58            this.FileFound(fi, EventArgs.Empty);
   59        }
   60    }
   61
   62    /// <summary>
   63    /// RootDirectory to search for files in
   64    /// </summary>
   65    public String RootDirectory { get; set; }
   66
   67    /// <summary>
   68    /// Pattern of files to search for
   69    /// </summary>
   70    public String SearchPattern { get; set; }
   71
   72    /// <summary>
   73    /// List of valid extensions to filter our search with
   74    /// </summary>
   75    public List<String> ValidExtensions { get; private set; }
   76
   77    /// <summary>
   78    /// List of extensions that should be searched as text
   79    /// </summary>
   80    public List<String> TextExtensions { get; private set; }
   81
   82    /// <summary>
   83    /// Specific text to search for within text files
   84    /// </summary>
   85    public String TextFileSearchString { get; set; }
   86
   87    /// <summary>
   88    /// Should text file searches be case insensitive?
   89    /// </summary>
   90    public Boolean CaseInsensitiveTextFileSearch { get; set; }
   91
   92    /// <summary>
   93    /// Determines if the search algorithm searches recursively
   94    /// </summary>
   95    public Boolean RecursiveSearch { get; set; }
   96
   97    /// <summary>
   98    /// Add an overloaded version of search that will set the search pattern
   99    /// </summary>
  100    /// <param name="searchPattern"></param>
  101    public void Search(String searchPattern)
  102    {
  103        this.SearchPattern = searchPattern;
  104        this.Search();
  105    }
  106
  107    /// <summary>
  108    /// The main search method
  109    /// </summary>
  110    public void Search()
  111    {
  112        // Make sure we have a search pattern
  113        if (String.IsNullOrEmpty(this.SearchPattern)) this.SearchPattern = "*.*";
  114
  115        // Search the home Directory
  116        this.SearchDirectory(this.RootDirectory);
  117    }
  118
  119    /// <summary>
  120    /// Internal search method that will fire the filefound event
  121    /// </summary>
  122    /// <param name="directory"></param>
  123    private void SearchDirectory(String directory)
  124    {
  125        // IF we are searching recursively, look through directories first
  126        if (this.RecursiveSearch)
  127        {
  128            foreach (String subDirectory in Directory.GetDirectories(directory))
  129            {
  130                this.SearchDirectory(subDirectory);
  131            }
  132        }
  133
  134       // Search through the files in the directory
  135       foreach (String file in Directory.GetFiles(directory, this.SearchPattern))
  136       {
  137           this.OnFileFound(new FileInfo(file));
  139       }
  140    }
  141}