Monday, October 29, 2007
On the concept of Form (1)
The point is, doing so would take too much [free] time, delaying my writings for several more weeks or even months. This is exactly the opposite of what I want to do with this blog (see my post blogging as destructuring), so I'll commit the ultimate sin :-) and talk about form in a rather unstructured way. I'll start a random sequence of posting on form, writing down what I think is relevant, but without any particular order. I won't even define the concept of form in this first instalment.
Throughout these posts, I'll borrow extensively from a very interesting (and warmly suggested to any software designer) book from Christopher Alexander. Alexander is better known for his work on patterns, which ultimately inspired software designers and created the pattern movement. However, his most releavant work on form is Notes on the Synthesis of Form. When quoting, I'll try to remember page numbers, in which case, I'll refer to the paperback edition, which is easier to find.
Alexander defines two processes through which form can be obtained: the unselfconscious and the selfconscious process. I'll get back to these two concepts, but in a few words, the unselfconscious process is the way some ancient cultures proceeded, without an explicit set of principles, but relying instead on rigid tradition mediated by immediate, small scale adaptation upon failure. It's more complex than that, but let's keep it simple right now.
Tradition provides viscosity. Without tradition, and without explicit design principles, the byproducts of the unselfconscious process will quickly degenerate. However, merely repeating a form won't keep up with a changing environment. Yet change happens, maybe shortly after the form has been built. Here is where we need immediate action, correcting any failure using the materials at hand. Of course, small scale changes are sustainable only when the rate of change (or rate of failure) is slow.
Drawing parallels to software is easy, although subjective. Think of quick, continuous, small scale adaptation. The immediate software counterpart is, very naturally, refactoring. As soon as a bad smell emerge, you fix it. Refactoring is usually small scale, using the "materials at hand" (which I could roughly translate into "changing only a small fraction of code"). Refactoring, by definition, does not change the function of code. Therefore, it can only change its form.
Now, although some people in the XP/agile camp might disagree, refactoring is a viable solution only when the desired rate of change is slow, and only when the gap to fill is small. In other words, only when the overall architecture (or plain structure) is not challenged: maybe it's dictated by the J2EE way of doing things, or by the Company One True Way of doing things, or by the Model View Controller police, and so on. Truth is, without an overall architecture resisting change, a neverending sequence of small-scale refactoring may even have a negative large-scale impact.
I've recently said that we can't reasonably turn an old-style, client-server application into a modern web application by applying a sequence of small-scale changes. It would be, if not unfeasible, hardly economic, and the final architecture might be far from optimal. The gap is too big. We're expecting to complete a big change in a comparatively short time, hence the rate of change is too big. The viscosity of the previous solution will fight that change and prevent it from happening. We need to apply change at an higher granularity level, as the dynamics in the small are not the dynamics in the large.
Curiously enough (or maybe not :-), I'll be talking about refactoring the next two days. As usual, I'll try to strike a balance, and get often back to good design princples. After all, as we'll see, when the rate of change grows, and/or when the solution space grows, the unselfconscious process must be replaced by the selfconscious process.
Labels: book reference, design, form, profession
Wednesday, October 24, 2007
More on evolving (or rewriting) existing applications
In my previous post I've only scratched the surface of a large problem. If rewriting the application from the core is so dangerous (unless you can break the feedback loop in the diagram, which you can, but not for free), are we left with any viable option?
We all heard about refactoring in the last few years. Refactoring tools are getting better, to the point that you can actually trust them with your production code. However, refactoring can only bring you so far. You won't change a fat client into a web application by incremental, small scale refactoring. But careful refactoring is truly helpful when dealing with small scale problems. I've recently seen some C/C++ code full of comments, and guess what, the backtalk of those comments (see my "Listen to your tools and materials") is actually "please remove me and refactor the code instead". Refactoring is also hard when the code is strongly coupled with an outdated (or ill-structured) relational database, without any kind of insulation layer. So, refactoring is an interesting technique, and can often be applied in parallel with some more aggressive strategy, but is not really helpful when you need large scale changes.
An interesting strategy, that can be applied in more than a few cases, is the modular rewriting of applications. The concept is not new: you exploit (or improve by refactoring and then exploit) the existing architecture, by gradually replacing modules. With a little oversimplification, we could say that the real problem is which modules you want to replace first.
Again, this depends on a number of factors:
- do you have a different target architecture? If so, is there any module that you can replace in the existing system, and which will effectively move, almost unchanged, to the new architecture?
- are you just tired of maintenance problems? Then use quadrants to isolate the best candidates. Troubled modules with small size, or troubled modules with minimal coupling from the rest, should be your first candidates. Get as much value as you can, as soon as you can. Get momentum. We're not in the 80s anymore; flat value delivery curves are a recipe from project cancellation.
- are you "just" changing technology, or are you trying to innovate the application? Honestly, most companies can't innovate. Not by themselves, and quite often, not even with external help. They're prisoners of their own artifacts. They think of their problems in terms of the existing implementation. They will never, for instance, conceive a real web 2.0 application, because they can't think that way.
If you want to innovate the application, start with innovative features in mind. Design new modules, then bridge them with the existing system. While designing the bridge, push the envelope a little: can we erode a little of the existing system, an replace some subsystem with something more appropriate?
Doing so will move you toward an Incremental Funding project, which will gradually erode the existing system until it will make economic sense to scrap the rest.
- Note that this assumes that innovation is happening at the fringes of your system, not at the core. This is often true (see also "Beyond the core" by Chris Zook, Harvard Business School Press, for a business-oriented perspective), but sometimes, the real innovation requires a new core. Say you need a new core. Are you sure? Maybe you "just" need a new product, barely integrated with the existing system. Don't say no so soon. Play with the idea. Don't get into a self-fulfilling expectation by telling yourself that everything must be tightly integrated into a new core.
- Are you moving toward a truly better architecture, or just following a technology trend? Maybe your company was much less experienced when the current system was built. Still, are you sure the new core won't become brittle very soon? Do you really need a large core? Can't you decompose the system toward (e.g.) a loosely-coupled, service-oriented architecture, instead of a tightly-coupled, database-centric architecture (maybe with a web front end, big deal :-).
- can you apply some non-linear (a.k.a. lateral) thinking? Not familiar with lateral thinking? De Bono should be a required reading for software developers, and even more so for project managers and designers!
- is your new/target architecture aligned with your company strategy? If not, think twice!
OK, it's getting late again. See you soon, I hope, with some reflections on the concept of form (vs function) in the software realm.
Labels: book reference, link, project management
Sunday, October 14, 2007
Evolving (or rewriting) existing applications
I'm often involved in very different activities, from weird bugs at the hardware/software interface, to coding tricky parts, to designing [small] application-specific frameworks, to making sense of nonsensical requirements. Recently, however, I've been helping a few customers make some rather fundamental decision about how to evolve (or rewrite) their applications.
Indeed, a significant subset of existing applications have been developed with now-obsolete technologies ("old" languages, framework, components, architectures), or even obsolete business concepts (e.g. a client-side application sold as a product, instead of a web application sold as a service).
Now, when you're planning the next generation of a successful application, you often end up trapped into a very natural, logical, linear thinking: we start from the core of the application, build a new/better/modern one, then we keep adding features till we have a complete application.
Sometimes, it's not even about starting at the core: you start with a large framework, and expect that you'll be able to build your applications in a snap (when the humongous framework is completed).
Now, this seems very logical, and if you look at it from a technology perspective, it makes a lot of sense. By dealing with the core first, you have the best chance to make a huge impact into the foundations, making them so much better. All those years on the market taught you a lot, so you know how to make the core better.
It's also easier this way: the new application will be entirely based on the new technology, from the bottom up. No need to mix old and new stuff.
As usual, a successful development strategy is completely context dependent! If your application is small, the natural strategy above is also the best overall strategy. In a relatively short time (since the application is small) you'll have a clean, brand-new application.
Unfortunately, the strategy does not scale at all to large applications. Let's see a few (well known, and often ignored) problems:
- The value delivery curve is pretty flat. Looks more like fig.1 in an old post of mine, as you can't really sell the new application till is finished.
People usually argue that by choosing the "right" core, they can start selling the core before the whole application has been ported. Yeah, sure, maybe in some alternative reality, but truth is, in many cases your existing customers won't downgrade to a less powerful, usually incompatible application (unless they're having major headaches from the old app).
New prospects won't be so exhilarated by a stripped-down core, either. Over the years, for a number of reasons, it' very likely that most innovation happened at the fringes of the old application, not at the core. You're now taking this stuff away for a potentially long time. Your old product will be competing with your new product, and guess what, lack of features tends to be rather visible.
- Although management may seem initially inclined to support your decision to rebuild everything from scratch, they won't stay silent as the market erodes. Soon they will (very reasonably) ask you to add some stuff to the existing application as well.
Now, by doing so, you'll slow down the development of the new application (resource contention) and you'll also create a backlog of features to be ported to the new application, once it is finished.
- All this conjures for very long development times. If you're working with intrinsically unstable technologies (like web applications), there is even a significant chance that your new application will be technologically obsolete before it gets to the market!
Let's try to model these (and a few more) issues using a diagram of effects
You may want to spend a little time looking at the different arrows, and especially at self-sustaining feedback loops. It's not hard to see that this is a recipe for having hard times. Yet companies routinely embark into this, because redoing the core is the most logical, most technologically sound thing to do. Unfortunately, it doesn't always make business sense
As usual, there are always other choices. As usual, they have their share of problems, and often requires better project management practices and skilled software designers. They also happen to be very context-dependent, as you have to find a better balance between your business, your current application, your new application.
Ok, time and space are up :-), I'll add a few details later.
Labels: link, profession, project management
