Before starting any development project, start by envisioning the perfect environment in which you want to develop said project. If this environment does not exist, start by creating it - then move on to the project at hand. Any services, features, helper methods/functions that can be handled abstractly by the environment should be considered 1st. Only after "the playground" infrastructure has been setup - then move onto the actual project at hand.
The interesting thing about this process is that it is completely recursive, from a large - think operating system scale project - all the way down to an individual line of code. Literally (as in, the literal definition of literally - not the figurative one!)
In other words, you're going to develop an operating system. This system will need a file system, memory management, process management, thread management, etc. ... eventually. But to begin with, whichever of these features are implemented, all operating systems need a bootstrap module. You (definitionally) can not have an operating system without a bootstrapping function.
So, let's start by designing the much smaller concept of a simple kernel. Not the final, full OS level kernel - but just the smallest kernel that I can think of. i.e. If we were given 12 hours to write an OS - could we develop a kernel that could bootstrap a piece of hardware to some sort of "operational" state in 12 hours? Even though I've never done anything even remotely like this before, I think that with the appropriate training, I could create a kernel in 12 hours.
In fact, there are probably 5 or 6 different ways that a "kernel" could be written. This is where the creative/editing/design process really kicks into gear. Because, now (hopefully as just a thought experiment at this point), each of the different approaches can be considered, in the larger context of the 6 months full OS development project that we're actually working on. Most likely 1 or 2 of these models will show a substantial benefit over the others.
These 2 now become "finalists" as the potential model for our kernel. At this point, it probably makes the most sense to just pick one and give it a try. Within the first 3 hours, we hopefully have something productive. In fact, we probably have enough to be able to really (in depth) re-evaluate our direction, and make sure that our assumption about the two finalists was accurate. Often at this point in the process, it becomes absolutely apparent that the other approach (or possibly, even one of the 3 that was originally discarded) is actually a much better way to solve this problem.
The key to note at this point in the process is that it's REALLY cheap to switch completely the design at hand.
If we take the full 12 hours before we re-evaluate we're already pretty "locked in" to the design that we started out with. Be open to change. Be ready for the fact that many to most of the decisions that you make might be wrong (fail!). But if you're open to that idea, you're much more likely to recognize the failure, and change course, early!
Sunday, September 29, 2013
Saturday, September 28, 2013
Axioms, Limitations and Freedom of Thinking
Axioms, Limitations and Freedom of Thinking
Argue for your limitations, and sure enough, they're your's.
-- Richard Bach, Illusions
Where do ideas come from?
What prevents ideas from coming?
What I think prevents creative, outside the box thinking, is in most cases arguments for limitation. In other words, one is never going to break free of "the box", if step one is arguing that there's an unbreakable, cubic shell surrounding your idea! If you start by coming up with all the reasons that you can't get outside the box, it's unlikely that you're going to find a path out (even if one exists right under your nose).
I've spent many years thinking about this quote, and I think that today, for possibly the first time, I may have begun to understand why arguments for limitations are so costly.
I would contend, that what we "know" is always based on axioms. Unfortunately, "thinking outside the box" often involves changing ideas that were originally considered to be axiomatic.
When arguing for your limitations, things usually unfold as follows:
Dave: "I want to go skydiving, but I hear you only fall 125 MPH - I want to fall faster. How can I do this?"
Bob: "Sorry Sir, it simply can't be done."
Bob: (often unspoken) "...Because that's how terminal velocity works, and unless you're going to gain 50 lbs, you're just not going to go any faster than that. Sorry."
Dave: Ya know, I think you might be right. Oh well.
Rather than starting by arguing for your limitations, consider this discussion:
Dave: "How can we accomplish Project XYZ?"
Bob: "Well sir, that seems tricky. It appears that for this to happen, A would have to change fairly substantially. Hmm, B also, does not seem like a viable option. Although... ya know, if we swapped D and E, then we could remove C altogether and that would solve the problem! Hazah!"
Dave: "Fantastic - when can we get started?"
The point is, that if Bob starts out by stating the obvious - that it can't be done, he has already taken the "outside the box" idea that D and E could be swapped off the table. Because for his original statement to hold true (and everyone always wants to be "right"), then D and E CAN'T switch places, so why on earth would he even think about it?
Notice that bob started at A, and first ensured that A and B were not viable solutions. Only then did he consider the better solution.
Always be open to any idea being "wrong". That way, you won't miss the really good ideas when they're staring you right in the face.
Writing code is like a mediation/negotiation
There seem to always be two sides of a story when writing code, and to some extent, the process of finding "the right" code, or the most elegant solution ends up proceeding as follows:
I just allow the code to essentially battle itself, with me as the programmer acting as the mediator in the negotiation.
So, the negotiation typically starts with the calling system asking attempting to call a function on the responding system that does not originally exist. Consider this an "opening offer" where the calling system provides as much (or as little) detail as IT thinks is necessary for the responding system to answer the question or request.
This will generate a matching method in the responding system. Now, we try to respond to the calling system's request, specifically from the perspective of the responding system. Hopefully, the information provided is sufficient and an answer can simply be provided.
In many cases though - the responding system will need more information - so it should "request" this additional information by adding a parameter to it's function signature. This will generate an error in the calling system's code.
By switching back to the calling system's perspective, providing the additional parameter is usually possible. But if it is not directly possible, either the calling system needs to go get the relevant information, or provide the next best option available.
By switching back to the responding system now, maybe the alternate information provided actually works, or may cause the responding system to change it's usage again.
This process continues until a mutually agreeable set of code is found.
An important note about this process is that at the point that the original "opening offer" was presented, a final solution to the problem was NOT completely understood. This process is the method to FIND the solution. It is NOT presumed that the solution is known at the beginning of this process.
I just allow the code to essentially battle itself, with me as the programmer acting as the mediator in the negotiation.
So, the negotiation typically starts with the calling system asking attempting to call a function on the responding system that does not originally exist. Consider this an "opening offer" where the calling system provides as much (or as little) detail as IT thinks is necessary for the responding system to answer the question or request.
This will generate a matching method in the responding system. Now, we try to respond to the calling system's request, specifically from the perspective of the responding system. Hopefully, the information provided is sufficient and an answer can simply be provided.
In many cases though - the responding system will need more information - so it should "request" this additional information by adding a parameter to it's function signature. This will generate an error in the calling system's code.
By switching back to the calling system's perspective, providing the additional parameter is usually possible. But if it is not directly possible, either the calling system needs to go get the relevant information, or provide the next best option available.
By switching back to the responding system now, maybe the alternate information provided actually works, or may cause the responding system to change it's usage again.
This process continues until a mutually agreeable set of code is found.
An important note about this process is that at the point that the original "opening offer" was presented, a final solution to the problem was NOT completely understood. This process is the method to FIND the solution. It is NOT presumed that the solution is known at the beginning of this process.
Saturday, December 29, 2012
A Study in Failure
Today, I was out playing in the snow with my daughter and 2 nieces. A neighbor came by and made a huge snowball just by rolling it down the hill that we were by. He inspired me to try to make an even bigger snowball!!
The First Failure: This should be simple...
Seemed simple enough! If he could make a snowball that big just rolling straight down the hill, just imagine how big I could make one if I started by going back and forth a few times at the top of the hill before rolling it down. If I could make a 3 foot snowball at the top of the hill, just think how big it would get by the time it had rolled all the way down to the bottom of the hill!
Back and forth. Back and forth. Pretty quickly it got too big to hold back and so down the hill it went! Bigger and bigger it got.... and then it broke into pieces half way down the hill; the largest one, no bigger than my head.
Failure #2: We clearly need a stronger core!
Hmm.... obviously the first attempt had not been strong enough to survive the roll down the hill. This time I would really pack the ball at the top to make sure I had a solid core.
So, I rolled and packed, rolled and packed (by lifting and gently back onto dropping it on the ground). It got bigger - and it was definitely much stronger than before. But by the time it got to be about 2 feet in size, it broke apart - even before I tried to roll it down the hill!
Failures 3, 4 and 5: No control of the rolling direction
Learning from this failure, I decided to try packing it by hand. It was much slower going, but man was it packed tight! By the time it got to be the same size as the first one, it was so solid it probably weighed more the twice as much! It was like a rock.
In fact, it was so heavy that I realized it might actually get further if I could roll it across the hill - across a much gentler slope. It would never get out of control and just think how big it would get...
So - we started to roll it, but gravity had other plans...
Useless strengthening techniques
I had also spent almost 1/2 the time building the core:
And... a relatively tiny snowball!
After all of that effort - almost 45 minutes of rolling and packing at the top of the hill, removing leaves and adding "fiber", and the resulting snowball was only about 3 1/2 feet in size. Smaller than the original!
What to do?
Having failed again and again to make a "big enough" snowball, I decided to take what I had learned and try once more.
I thought to myself - what if I didn't try to use the hill at all. That way, I would never need a big, solid core to "survive" a run down the hill. Instead, I could just try to make the lightest, fluffiest core possible - since just pushing it back and forth on a flat surface would never require it to be "strong".
So once again, I started to roll... Relatively quickly it was bigger than any of our previous attempts. I needed help, and got it from my daughter, niece and... Brian, who had rolled the original snowball.
With all of us working together, we were able to build, and build and build and build - until it was too big to push anymore. I think if we'd had more people - we probably could have made it even bigger.
Finally - success. But success that was only possibly by pushing through, and learning from the five failures that had preceded it.
The First Failure: This should be simple...
Seemed simple enough! If he could make a snowball that big just rolling straight down the hill, just imagine how big I could make one if I started by going back and forth a few times at the top of the hill before rolling it down. If I could make a 3 foot snowball at the top of the hill, just think how big it would get by the time it had rolled all the way down to the bottom of the hill!
 |
| Preparation... |
Back and forth. Back and forth. Pretty quickly it got too big to hold back and so down the hill it went! Bigger and bigger it got.... and then it broke into pieces half way down the hill; the largest one, no bigger than my head.
 |
| Fail! |
Failure #2: We clearly need a stronger core!
Hmm.... obviously the first attempt had not been strong enough to survive the roll down the hill. This time I would really pack the ball at the top to make sure I had a solid core.
So, I rolled and packed, rolled and packed (by lifting and gently back onto dropping it on the ground). It got bigger - and it was definitely much stronger than before. But by the time it got to be about 2 feet in size, it broke apart - even before I tried to roll it down the hill!
Fail.
Failures 3, 4 and 5: No control of the rolling direction
Learning from this failure, I decided to try packing it by hand. It was much slower going, but man was it packed tight! By the time it got to be the same size as the first one, it was so solid it probably weighed more the twice as much! It was like a rock.
In fact, it was so heavy that I realized it might actually get further if I could roll it across the hill - across a much gentler slope. It would never get out of control and just think how big it would get...
So - we started to roll it, but gravity had other plans...
 |
| Preparation - Actual Path (Fail) - Desired Path (gentle slope) |
I had also spent almost 1/2 the time building the core:
- Removing leaves from the outside (which I felt might be weakening the ball)
- And leaving (or even adding) sticks, twigs, etc (which I felt might help to strengthen the ball)
While the final core did not break apart (certainly a successful improvement over previous attempts) the solution had not really assisted in achieving the stated goal and had probably just added weight, making it harder to push.
Unhelpful Success... (failure)
And... a relatively tiny snowball!
After all of that effort - almost 45 minutes of rolling and packing at the top of the hill, removing leaves and adding "fiber", and the resulting snowball was only about 3 1/2 feet in size. Smaller than the original!
 |
| Fail! |
Having failed again and again to make a "big enough" snowball, I decided to take what I had learned and try once more.
I thought to myself - what if I didn't try to use the hill at all. That way, I would never need a big, solid core to "survive" a run down the hill. Instead, I could just try to make the lightest, fluffiest core possible - since just pushing it back and forth on a flat surface would never require it to be "strong".
So once again, I started to roll... Relatively quickly it was bigger than any of our previous attempts. I needed help, and got it from my daughter, niece and... Brian, who had rolled the original snowball.
With all of us working together, we were able to build, and build and build and build - until it was too big to push anymore. I think if we'd had more people - we probably could have made it even bigger.
 |
| The final snowball |
Saturday, December 1, 2012
CEE (Computer Emotion Engine)
I've been thinking for many months now about what makes us (human beings) so different from computers (machines). One of the main things that I had previously identified, was this massive divide between biological and non-biological organisms. To be more precise:
- I saw "emotion" (or some form thereof) in virtually every biological organism that I looked at from the smallest single celled organism struggling to "get to the light".
- At the same time, I was struck by the complete (100%) lack of "emotion" in all of the various (non-biological) machines around us.
Further thinking on the subject of this duality lead me to the question of whether "life" might be the source of what we call emotion, and therefore if it was possible that machines would simply never "feel" as we do. Period.
This idea was further reinforced by the fact that, when trying to imagine how I would go about programming a computer to "feel", I couldn't come up with even the remotest idea of where to start. I imagined having virtually unlimited resources and hundreds of developers at my disposal, and was disappointed with myself that I wouldn't even know where to begin work on such a project.
Today that all changed. After a discussion with my friend Matt Cherwin last week, I realized that there "might" be a path from where we are now, to where I've been trying to get. In our discussion, Matt maintained that the what we call "emotions" are really just the interactions of a number of complex systems. I maintained that IBM's Watson, for example, seemed like the kind of complex system that he was describing, and yet didn't display anything that even began to approach what we refer to as emotion. At the same time, I could think back on our own evolution, and see even the simplest of organisms that did seem to display such characteristics. This seemed like an unbridgeable divide!
Over the weekend I continued to ponder on our discussion, and today I think that I might have made a breakthrough in my own thinking on the subject. Specifically, if I redefine what I mean by "emotion" as follows, I think that we might very quickly end up with a system that, surprisingly to me, very much starts to feel like human emotion.
Basic System Design
So, here are some overarching concepts to this CEE (Computer Emotion Engine - pronounced SEE, see? :)
The "System" would be comprised of a base class called EmotionBase that defined some basic characteristics of all emotions in "the system". At the very least, these properties would include the name of the emotion and an Emotion Indicator (EI) scale - for example 1-100 which would indicate the extent of this emotion.
This class would further allow for weighted input sources and an algorithm that would combine these input sources together into one final EI number for that emotion.
Specific Use-Case
So, with this simple starting point, let's take a specific use case. The goal of this use-case is to allow our computer to interact with us at an "emotional level". In other words, I want to be able to ask my computer how happy it is, and have it respond, not randomly, but based on a variety of specific "inputs". To begin with, I might just have a little emoticon icon in the start bar that either Smiled or Frowned based on how "happy" it was.
So, the emotion created would be called Happiness as follows.
- 100 = :)
- 50 = :|
- 0 = :(
Now - what dictates happiness on our computer? Here's where the various inputs come into play. Some factors that could play into the computer's "happiness" include.
- Free disk space (get's unhappy as disk space runs out)
- CPU Usage - (happy as long as the CPU isn't totally and consistently pinned)
- CPU Heat (get's unhappy as the CPU heat goes higher).
- Available system resources
- etc.
So - our "happiness" emotion would have 4-5 values, input on a regular basis, and the computer's happiness would change as these values change.
Other examples of emotions that could be built on a similar type of model include:
- Boredom
If the computer isn't "doing anything", it could get more and more bored. If it's got 15 programs open, is running a backup routine in the background, and doing a virus scan while also downloading a 300 MB service-pack, it would not be board at all (0). If it's just sitting there, it would be quite "bored" (100). - Nervousness
If the virus scanner has recently detected a number of problems, it's running out of disk space on drive F, the CPU is running at 95%, and 3 new, unidentified processes are running - this could make the computer quite "nervous". - Anger
100's of unidentified (and new) errors in the system event log could cause the computer to get angry!
So - the first question you might be inclined to ask is - what's the benefit of such an "emotion engine"? I can think of a number of them. To begin with, it would allow us to change how the computer "chooses" what to do, and potentially how we interact with it, to be based on such emotions. Since computers don't have such such emotional programming at this point, this will seem quite odd - but if such a system were in place, here are some examples of how things might look different.
Right now, we run virus scans at 3:00 AM, Mon-Friday and again on Sunday night. We do this because we assume that these will be times of inactivity. Instead, we could program our virus scans to run "whenever you're bored, or if you are nervous about new viruses for any reason".
We could program the system to look for and clean up unused disk space when it gets "concerned' about low disk space. Now, what makes the computer "nervous" about disk space could vary from system to system in a way that our current "precise", "non emotion based" systems make more difficult. For example, if one system has 50GB of free space, maybe it's not "nervous" at all, because it's only a 70GB drive. On the other hand, a 5 TB drive with only 50GB of space should make the system very nervous. Similarly, if the system has 50GB of free space, and it knows that it has a project coming up that is going to need 60 GB of space, that could make it anxious about this - causing it to seek out a remedy to this anxiousness eg. freeing up some more space.
From our perspective, it would change the "language" that we use to communicate with our machines to match our own "emotional" verbiage.
Now - all this said, there are limits to where these benefits begin/end. We don't ever want to ask our word-processor to print a document and get back the response "No, I don't really 'feel' like it...". So - this would not be a simple thing to "get right". That being said, in my limited thinking on this so far, I think that A) It could be done and B) it could provide some powerful benefits.
How the system could be extended
To really be helpful, CEE emotions would have to be programmed carefully. In addition, the "decision engines" of applications and larger systems would need to evolve to take advantage of these emotions. As with people, this would be a delicate balancing act that would need to be managed carefully.
That being said, it seems like new emotions could be designed to serve specific purposes. In addition, common "system emotions" could be extended by third parties. In other words, the system could allow for 3rd parties to write "plugins" of sorts that could influence the emotional state of the machine.
This "emotion model" might need to evolve so that certain emotions were kept "private", while others were shared more freely.
To begin with, the emotion state of the machine might be more global, but might eventually get quite specific, and granular for specific sub-systems to take advantage of. In other words, overall, the system might be quite happy, while at the same time being very nervous about the length of time since it's last backup had run.
The emotion systems would also need to feed back on each other in quite a sophisticated way that would be difficult to "nail" on a first pass (version 1) of such a system. For example, if the overall system has a happiness quotient, this overall happiness index should really (eventually at least) be influenced by the individual (local) emotions of:
- The various applications running
- The hardware, and the state of each of the individual hardware components
- Various performance indicators
- etc.
The interplay of these "emotions" together would be difficult to "get right" - that's for sure.
In addition to systems being able to make decisions based on these emotions, "the system" could also evolve to better and better answer the question "Why are you sad?", "Why are you scared?", etc.
Two types of emotions
Seems like there are at least 2 different types of emotions. There are singular and bidirectional emotions. There may be others, but these are the only two I've thought of so far. Examples of singular emotions/feelings are:
Two types of emotions
Seems like there are at least 2 different types of emotions. There are singular and bidirectional emotions. There may be others, but these are the only two I've thought of so far. Examples of singular emotions/feelings are:
- Fear
- Hunger
Most emotions/feelings however seem to (consistently) have two opposing ends of the spectrum:
- Happy/Sad
- Love/Hate
- Energetic/Tired
- Good/Bad
It's possible that all of these could be singled out into one "Frame" that works for all of them, but I have so far been unable to think of the opposite of "afraid".
Conclusion
The bottom line, is that - for me at least, for the first time - I feel like I have a model that would produce a machine (computer) that could possibly pass a Turing test, specifically designed to detect emotion - that would not be "fake" (ie. Not just a rule-set that dictates: "20% of the time, say you're 'happy'." or "If it's dark, say you're 'scared'.") and further, could actually provide a different way for us (human beings) to interact with machines/computers/programs/etc.
Software vs. Civil Engineering
How Software Engineering is radically different from Civil Engineering, and why confusing the two is so dangerous and costly.
Software developers like to call themselves engineers. I admit, that I have referred to myself as a software engineer on more than one occasion - though in recent years I have come to believe that this identifier is both inaccurate, unproductive, costly and even dangerous to our industry.
It is empowering to think of ourselves as "engineers". It gives us a credibility that is not conferred by the title "developer". But it also categorizes what we do in a way that I believe, in many cases, leads to poor software design, and messy systems implementations.
How Civil Engineering is so different from Software Design
I read an article some time ago by Chris Aitchison called "You are not a software engineer". It was the first time I had read someone else describe why they thought the term "Software Engineer" was an inappropriate term for what we do - and I think that it provides a fantastic metaphor to replace it.
I've spent a lot of time thinking about this issue though - and would like to expand on what he wrote, and to provide an alternate explanation of how we might approach software design that is more effective.
Building a Bridge, or a Skyscraper, or a Road requires a massive amount of planning up front, and detailed specifications - before the project begins. One of the major reasons that such detailed plans are needed up front is that changes to these types of projects, once the project has begun, are generally very expensive. In some cases, prohibitively so.
One can not start building a 50 story skyscraper, then decide to change it to a 100 story skyscraper 80% of the way through the project. This is simply not done. It would, in almost every case be both cost prohibitive, as well as being physically unfeasible in most cases as well.
Software, however, is a completely different animal.
Don't get me wrong - I am NOT arguing that planning should not, or does not need to be, a central part of the software design process. But, that being said, what I am arguing is that the approach to software design and implementation can be done in a radically different way than a civil engineering project - because of the nature of how software works.
Often, with a physical engineering project, models are created up front to show the client what the final product will look like. Small scale version, or prototypes might be created to work through some of the physical aspects of how certain parts might work together. (Obviously much of this is now done "virtually", with the help of computer aided design (CAD) programs - but that doesn't change the underlying point here). These prototypes and models however are not generally functional thought, and have no bearing on the final project that must be delivered.
In other words, it is not possible to create a 1/10th scale version of a skyscraper, and then "scale it up" to full size. It is not, generally speaking, productive to create a bridge "iteratively". Building physical structures simply do't work that way.
Software, by contrast can, and more and more these days, is developed iteratively. Software can evolve. Software can grow. If you design a house, and want to change the footprint of the house, this would require you to pull down the existing structure, pouring a new foundation, and then building a new house. In software design, this is often not the case!
A different approach
Thinking of software, as though it is cost constrained in the same way as physical engineering, leads us to approach software as though it is a civil engineering project. We want "full functional specifications", blueprints as it were, before starting the project. We then expect to be able to climb into a cave for 4-6 months, and come out after that time with a finished product that exactly implements the desired functionality, "to spec". Then, when the client looks at the final product and decides that they don't like 50% of it, the entire process repeats itself. Implementations go over budget (on both time and cost), clients are left unhappy, and ultimately - projects fail.
The agile development methodology is designed to address this to some extent, but ultimately, what is needed is a more fundamental shift in our perspective of what software is, and how it should be developed.
Software is not a physical structure, and can be changed, cost effectively, in a way that makes it fundamentally different than anything that human kind has created in it's history. We should take advantage of this flexibility in our approach to it, from the top of the design metaphor, all the way down to the individual developer, working on a single widget within the larger project.
Software can, and should...evolve. It should grow. This is why the description of us as software "gardeners", is a much better description of us than software engineers.
Software developers like to call themselves engineers. I admit, that I have referred to myself as a software engineer on more than one occasion - though in recent years I have come to believe that this identifier is both inaccurate, unproductive, costly and even dangerous to our industry.
It is empowering to think of ourselves as "engineers". It gives us a credibility that is not conferred by the title "developer". But it also categorizes what we do in a way that I believe, in many cases, leads to poor software design, and messy systems implementations.
How Civil Engineering is so different from Software Design
I read an article some time ago by Chris Aitchison called "You are not a software engineer". It was the first time I had read someone else describe why they thought the term "Software Engineer" was an inappropriate term for what we do - and I think that it provides a fantastic metaphor to replace it.
I've spent a lot of time thinking about this issue though - and would like to expand on what he wrote, and to provide an alternate explanation of how we might approach software design that is more effective.
Building a Bridge, or a Skyscraper, or a Road requires a massive amount of planning up front, and detailed specifications - before the project begins. One of the major reasons that such detailed plans are needed up front is that changes to these types of projects, once the project has begun, are generally very expensive. In some cases, prohibitively so.
One can not start building a 50 story skyscraper, then decide to change it to a 100 story skyscraper 80% of the way through the project. This is simply not done. It would, in almost every case be both cost prohibitive, as well as being physically unfeasible in most cases as well.
Software, however, is a completely different animal.
Don't get me wrong - I am NOT arguing that planning should not, or does not need to be, a central part of the software design process. But, that being said, what I am arguing is that the approach to software design and implementation can be done in a radically different way than a civil engineering project - because of the nature of how software works.
Often, with a physical engineering project, models are created up front to show the client what the final product will look like. Small scale version, or prototypes might be created to work through some of the physical aspects of how certain parts might work together. (Obviously much of this is now done "virtually", with the help of computer aided design (CAD) programs - but that doesn't change the underlying point here). These prototypes and models however are not generally functional thought, and have no bearing on the final project that must be delivered.
In other words, it is not possible to create a 1/10th scale version of a skyscraper, and then "scale it up" to full size. It is not, generally speaking, productive to create a bridge "iteratively". Building physical structures simply do't work that way.
Software, by contrast can, and more and more these days, is developed iteratively. Software can evolve. Software can grow. If you design a house, and want to change the footprint of the house, this would require you to pull down the existing structure, pouring a new foundation, and then building a new house. In software design, this is often not the case!
A different approach
Thinking of software, as though it is cost constrained in the same way as physical engineering, leads us to approach software as though it is a civil engineering project. We want "full functional specifications", blueprints as it were, before starting the project. We then expect to be able to climb into a cave for 4-6 months, and come out after that time with a finished product that exactly implements the desired functionality, "to spec". Then, when the client looks at the final product and decides that they don't like 50% of it, the entire process repeats itself. Implementations go over budget (on both time and cost), clients are left unhappy, and ultimately - projects fail.
The agile development methodology is designed to address this to some extent, but ultimately, what is needed is a more fundamental shift in our perspective of what software is, and how it should be developed.
Software is not a physical structure, and can be changed, cost effectively, in a way that makes it fundamentally different than anything that human kind has created in it's history. We should take advantage of this flexibility in our approach to it, from the top of the design metaphor, all the way down to the individual developer, working on a single widget within the larger project.
Software can, and should...evolve. It should grow. This is why the description of us as software "gardeners", is a much better description of us than software engineers.
Monday, October 10, 2011
Evolution vs. Intelligent Design / Agile Development
SIPing: Systemically Incremental Programming
A fairly direct analogy can be made between the debate over evolution vs. intelligent design and agile development at large.
The debate from the perspective of intelligent design advocates says that life, in all it's complexity could never have evolved step-by-step, and must have had an intelligent designer acting behind the scenes.
From the perspective of evolutionary scientists, the argument is almost exactly the opposite. It simply does not make sense to them that life, in all it's complexity, appeared from nothing (or rather from some already unimaginably complex being). Rather, it is said, life must have evolved slowly, step-by-step, piece-by-piece over very long periods of time.
In natural biology, evolution is in my mind a more reasonable and defensible explanation of how the world, in all it's wonder, came into being. Ironically however, in software design, we need a hybrid of the two the concepts to develop great software.
Waterfall & Intelligent Design
In my mind, waterfall is very much on the same page as what drives the model of intelligent design. It assumes that the designers (the software architects, product stakeholders, developers, etc) can produce, at the onset of the project, a holistic design that addresses all of the needs of the eventual end-users.
It further assumes that these intelligent designers can then spend 8-12 months executing said designs, and produce exactly what was originally intended.
It goes on to assume that when the end-users finally consume the product of this intelligent design, the changes required at that point will be small and inconsequential, and therefore the software will be "done" - because, after all, it was intelligently designed from the start.
Evolution & the Agile Development Model
Evolution, on the other hand, makes progress through the process of natural selection. If one "copy" of a species has a random mutation that makes it more well adapted to a particular climate, or circumstance, it is more likely to survive to the age of reproduction. This benefit is then passed on to it's children, their children and their children's children, and eventually the entire population has this new beneficial trait.
Agile development takes an approach to software design and implementation that much more closely resembles evolutionary processes. Rather than coming up with one massive plan for the entire project at the beginning, the project is split up into many small "generations" or "sprints". Each generation "evolves" from the previous one, improving on it, and incrementally, step-by-step, a sophisticated system is born.
Ironically, in the case of software development, it is actually a hybrid of the two models that is needed to produce great results. Specifically, there is no "survival" as such in the software design process. There are no opportunities for natural "mutations" to occur. Instead, while the software definitely evolves incrementally in an agile shop, it requires intelligent designers to "guide" this evolution along the way. Software engineers must introduce the mutations from generation to generate (i.e. write more code, analyze the results, rinse & repeat the process again and again), and in this way guide the evolution of the overall product being developed.
How this relates to Systemically Incremental Programming (SIPing)
As I've stated in a number of previous articles, I find it ironic that even staunch agile development advocates seem to revert back to a waterfall type of approach when it comes down to the individual bugs within a sprint.
What I've observed, is that even when an organization at large adopts and implements an agile approach to development, this in no way addresses how each developers handles 4-8 hours of work placed in front of them. An individual bug might include:
Typically, such an organizational model in no way indicates the order in which the functionality will be implemented. It's usually heavy on design, and light on results. If the implementation of the data-access layer runs into glitches, the overall implementation is effected (goes long/over budget), causing the entire bug to quickly explode in terms of time and complexity. This in turn potentially holds up other important processes in the larger development effort.
What you have to "buy" however, is that even a very sophisticated system can evolve in this way. They do not need to be designed from the top down. In fact, the benefits of this evolutionary process are the same at the bug-by-bug level as they are in the organization at large.
A SIPed plan for the same bug would take the 10 hours of work and say - in the first 2 hours, we will get a working, functional model. It will not do everything that needs to be included in the final version, but it will verify the overall process, it will be simple and easy to write, and it will be functional enough to present to the stakeholder for verification that we're heading in the right direction.
NOTE: Whether or not an outside stakeholder is shown this early version will depend on the specific feature being developed. At the very least, the developer himself is always a stakeholder, and he or she will have the opportunity to actually see this "rough draft" and make a decision about which direction to head next.
After this initial version is implemented, the entire process is re-evaluated - just like the process between sprints in an agile process. The remaining functionality is re-considered with respect to this initial "draft", some new "features" might be added to the list, others might be de-prioritized, and a second pass is then undertaken.
As with agile at large, the specific bug, defect or new feature being addressed evolves SIP-by-SIP, rather than being developed as a single, intelligently designed module - which is simply never as easy as it often appears to be at the onset.
A fairly direct analogy can be made between the debate over evolution vs. intelligent design and agile development at large.
The debate from the perspective of intelligent design advocates says that life, in all it's complexity could never have evolved step-by-step, and must have had an intelligent designer acting behind the scenes.
From the perspective of evolutionary scientists, the argument is almost exactly the opposite. It simply does not make sense to them that life, in all it's complexity, appeared from nothing (or rather from some already unimaginably complex being). Rather, it is said, life must have evolved slowly, step-by-step, piece-by-piece over very long periods of time.
In natural biology, evolution is in my mind a more reasonable and defensible explanation of how the world, in all it's wonder, came into being. Ironically however, in software design, we need a hybrid of the two the concepts to develop great software.
Waterfall & Intelligent Design
In my mind, waterfall is very much on the same page as what drives the model of intelligent design. It assumes that the designers (the software architects, product stakeholders, developers, etc) can produce, at the onset of the project, a holistic design that addresses all of the needs of the eventual end-users.
It further assumes that these intelligent designers can then spend 8-12 months executing said designs, and produce exactly what was originally intended.
It goes on to assume that when the end-users finally consume the product of this intelligent design, the changes required at that point will be small and inconsequential, and therefore the software will be "done" - because, after all, it was intelligently designed from the start.
Evolution & the Agile Development Model
Evolution, on the other hand, makes progress through the process of natural selection. If one "copy" of a species has a random mutation that makes it more well adapted to a particular climate, or circumstance, it is more likely to survive to the age of reproduction. This benefit is then passed on to it's children, their children and their children's children, and eventually the entire population has this new beneficial trait.
Agile development takes an approach to software design and implementation that much more closely resembles evolutionary processes. Rather than coming up with one massive plan for the entire project at the beginning, the project is split up into many small "generations" or "sprints". Each generation "evolves" from the previous one, improving on it, and incrementally, step-by-step, a sophisticated system is born.
Ironically, in the case of software development, it is actually a hybrid of the two models that is needed to produce great results. Specifically, there is no "survival" as such in the software design process. There are no opportunities for natural "mutations" to occur. Instead, while the software definitely evolves incrementally in an agile shop, it requires intelligent designers to "guide" this evolution along the way. Software engineers must introduce the mutations from generation to generate (i.e. write more code, analyze the results, rinse & repeat the process again and again), and in this way guide the evolution of the overall product being developed.
How this relates to Systemically Incremental Programming (SIPing)
As I've stated in a number of previous articles, I find it ironic that even staunch agile development advocates seem to revert back to a waterfall type of approach when it comes down to the individual bugs within a sprint.
What I've observed, is that even when an organization at large adopts and implements an agile approach to development, this in no way addresses how each developers handles 4-8 hours of work placed in front of them. An individual bug might include:
- Business Logic
- Network communication
- A data-access layer
- UI Development
- Other?
A plan for how to implement the bug might include sub-topics under each of these high level areas that address how each of those areas of the code will be addressed. As the bug gets bigger and bigger, this "tree" diagram of functionality gets bigger as well. In the end, even if it is only 10-15 hours of work, it starts to look more and more like a waterfall type of "intelligent design" for how the bug will be implemented.
Typically, such an organizational model in no way indicates the order in which the functionality will be implemented. It's usually heavy on design, and light on results. If the implementation of the data-access layer runs into glitches, the overall implementation is effected (goes long/over budget), causing the entire bug to quickly explode in terms of time and complexity. This in turn potentially holds up other important processes in the larger development effort.
A SIPed view of the same problem
SIPing approaches this bug differently. As with agile development in general, it takes a 15 hour bug, and splits it up into 10-15 one hour "sprints", or "SIPs". Each SIP might touch multiple functional areas described by the "intelligent design" originally proposed. Each SIP however would be an incremental improvement or evolution over the previous SIP. Each SIP introduces a new beneficial trait that did not exist in the software before. Each SIP is small, easy to understand, implement, test and demonstrate.
What you have to "buy" however, is that even a very sophisticated system can evolve in this way. They do not need to be designed from the top down. In fact, the benefits of this evolutionary process are the same at the bug-by-bug level as they are in the organization at large.
A SIPed plan for the same bug would take the 10 hours of work and say - in the first 2 hours, we will get a working, functional model. It will not do everything that needs to be included in the final version, but it will verify the overall process, it will be simple and easy to write, and it will be functional enough to present to the stakeholder for verification that we're heading in the right direction.
NOTE: Whether or not an outside stakeholder is shown this early version will depend on the specific feature being developed. At the very least, the developer himself is always a stakeholder, and he or she will have the opportunity to actually see this "rough draft" and make a decision about which direction to head next.
After this initial version is implemented, the entire process is re-evaluated - just like the process between sprints in an agile process. The remaining functionality is re-considered with respect to this initial "draft", some new "features" might be added to the list, others might be de-prioritized, and a second pass is then undertaken.
As with agile at large, the specific bug, defect or new feature being addressed evolves SIP-by-SIP, rather than being developed as a single, intelligently designed module - which is simply never as easy as it often appears to be at the onset.
Subscribe to:
Posts (Atom)