Thursday, August 07, 2008 

Do we need a Theory of Wrapping?

I think it was about 10 years ago. I was teaching OOD principles to a .COM company, and a senior developer said we should really develop a theory of wrapping. We talked a little about it, but then we moved to explore other concepts. I guess I totally forgot the idea till a few weeks ago, when 3 different customers came up with thorny issues, and they all had something in common: some kind of wrapper.

Wrappers are routinely created during software development, yet not all forms of wrapping are benign. Some may look convenient in the short term, but will come back to haunt us in the long term.
Would a theory of wrapping help? Indeed, we already have some organized knowledge on wrapping: several patterns are built upon the idea of wrapping an object or a subsystem. Looking at the GoF patterns alone, Adapter is a wrapper, Facade is a wrapper, Proxy is a wrapper. Decorator, although somehow similar to proxy in implementation, isnt't really a wrapper, as it adds new responsibilities.
Looking at wrapping through patterns, however, doesn't provide much guidance about the long-term consequences of wrappers. So, let's move to anedoctal knowledge for a while.

My first troubled customer is trying to solve an architectural problem by wrapping a few functions, and perform some magic inside those functions. The magic involves threads, fibers, apartments, and stuff like that. As any true form of magic, it must not be revealed to the general public :-)), so I won't.
Magic isn't well-known for reliability, but there is chance that magic might indeed work, which in a sense is even worse. We know the real problem is elsewhere: it has been known for more than 10 years, but the troubled area has never been fixed, just hidden under several layers. The real fix would entail changing 100 to 150 classes. Personally, I would work toward making that change as painless as possible, and then do it. They would rather go the wrapping way.

My second, less troubled customer has a much simpler problem: a legacy database, which in turn is mostly a mirror of the in-memory data structures of a legacy application (still maintained and developed). We need to provide a web front end, and the general agreement is to outsource that development. We basically have three choices:
1) ask the contractor to write the web app against the existing, ugly db. That requires some documentation effort on the company side, as the db can't be used without a detailed knowledge about fields (and bit-fields inside fields).
2) clean the database (in-house), clean the legacy app (in-house), let the contractor write the web app against the clean db. Sounds nice, but requires a lof of in-house work, and even worse, it would delay the contractor. Cleaning the legacy app seems daunting, and sooner or later we want to scrap that code anyway.
3) keep the ugly db, but provide a wrapper, in the form of a facade object model. Ask the contractor to write the web app against the facade. Delay cleaning the db for a while (possibly forever) and hope that a quickly developed facade will withstand the test (or ravages) of time. Because yeah, well, we ain't got much time, or we would go with (2). By the way, we could write the facade in-house, or write the same kind of documents as in (1) and ask the contractor to write the facade.

I would love to recommend (2), but being reality-based, I recommended (3), with the facade written by the contractor. So (dare I say it!!) I actually recommended that we spend our valuable in-house time writing documentation. How non-agile is that :-)). But the thing is, there is only one guy who knows the db, and that's bad risk management. Also, if at some point we want to redesign and clean the db, a documentation of what is currently hidden inside bit-fields stored as integers would be quite valuable. Oh, by the way: we did a little experiment to estimate how long is gonna take to write that doc: about 10 man/days, easily broken in short tasks, which can partially overlap the calendar time consumed by the contractor to build the facade object model. Not so bad after all, except we still keep a crappy db and application.

My third customer doesn't really feel troubled :-). Well, in fact, he's not really troubled: I am troubled :-). Years ago, a small team ventured into building an application-specific framework. Although I often preached about creating a set of small, loosely coupled mini-frameworks, people on that team weren't so interested in listening. So they went on and created a large, tightly coupled framework (which is much easier to build, and much harder to learn, change, etc).
When the company decided to build the first (web) application based on that framework, the application developers found out (guess what :-) that the ambitious design of the large framework was quite hard to understand, to the point that their main goal was surprisingly hard to reach. I proposed that the framework developers could create a simple facade, closer to our problem and therefore easier to use. They did, the application was completed, and is still in use today. So far so good :-).
A few years went by, and more applications have been written against that framework. The original framework developers moved to application development (or leaved the company), actually leading application development. Recently, I discovered that all the subsequent applications have been written against "my" facade, which was never designed to be general-purpose. However, it was so much simpler to use than the framework that people opted to use it anyway. They tweaked the facade when necessary, so we now have multiple versions of the facade around.
Again, the "right" thing to do would have been to make the framework easy to use on the first place. The facade was my backup plan because I never managed to make a dent in the framework developers' minds. Sure, it was instrumental in writing the applications, but somehow, it backfired: it became a superstructure of its own, living well beyond my expectations. Unfortunately, it was never designed to be a general-purpose component.

Of course, not every form of wrapping is evil. For instance, I often add a small facade between my code and any third party components and libraries. That shields me from many instabilities in interfaces, and in many cases allows me to change supplier if needed. That's harder with GUI components making heavy use of design-time properties, but there are techniques to deal with that as well (if warranted). Again, some wisdom is required: it makes little sense to wrap a pervasive library or framework; it makes little sense to invest in wrapping when you're developing a short-lived corporate application, and so on. Still, as someone said (exact quote needed :-), every programming problem can be solved by adding one level of indirection, and in some cases a wrapper provides a useful level of indirection.

Can we learn a general lesson from experience? From the pattern perspective, my first customer is just trying to add some kind of proxy between subsystems. The second and third are using a plain old facade. Nothing wrong with that. Yet no one is making the "right" choice: the wrapper is used to avoid fixing the real problem. We often say words like "postpone", but in software development, "postpone" usually means "don't do it".

So, again, would a theory of wrapping help? And what should this theory tell us? As is often the case, a good theory of software design should help us to make informed decisions, and like it or not, that means informed economical decisions. All too often, sound engineering choices are discarded in favor of cheap solutions, because we have no real economic model, or even a rough economic model, to calculate the (possibly negative :-) ROI of what we propose.

The real, long-term economical consequences for my first customer are way too complex, and I have little hope that we could ever develop a theory that could even come close to help with that. However, I believe the bigger long-term cost factor of the proxy-based solution would be in the increased debugging effort that would fall on the shoulders of all the other programmers working on the same project (about 100 people, I guess). This problem, somehow, could be highlighted in a theory of wrapping. A proxy, especially when doing threads-fibers-apartments magic, makes it hard to get a clear picture of what's going on just by looking at the stack (which is a per-thread, per-fiber structure). Unfortunately, the impact of this problem seems very hard to quantify.

My second customer is facing a much simpler scenario. I think we could eventually build an economic theory that could model the maintenance cost of keeping the legacy code as-is, the risk of building a leaky abstraction as we quickly build a facade over the legacy db, thereby increasing the mass of code that we'll have to change later, as we finally get to create a better database, and compare that with the savings of not doing it. We should also be able to factor in the increased maintenance cost of keeping the facade in-synch with any changes to the db required by the evolving application. It's not rocket science. Or maybe it is, but we can actually send a rocket to Mars :-). A good economic theory should recommend, in this specific context, to go with (3). Why? Because that's what I did :-)).

The third problem, I think, is more of a process problem. There was nothing wrong in the original wrapper. However, it wasn't designed for reuse, and is should not have been reused as-is, or by tweaking. This is a direct consequence of a lack of control over design choices; the team is, well, wrapped :-)) on itself. Unfortunately, lack of control over design issues is exactly what the team wants. A good process is meant to keep that kind of disfunctional behaviour out. Note that it doesn't have to be an heavyweight process. Just a good process :-).

Unfortunately, I don't have Theory of Wrapping to offer. At least, not yet :-). So, for the time being, I'll keep dealing with wrapping using informal reasoning, mostly based on experience and intuition. Got some good idea? Leave me a comment here!

Labels: , , ,

Wednesday, November 14, 2007 

Process as the company's homeostatic system

I was learning about EPOC (Excess Post-Exercise Oxygen Consumption) a few weeks ago, for no particular reason (as I often do :-). If you ever exercised at medium-to-high intensity, you've probably experienced EPOC: after you stop training, your oxygen intake stays higher than your usual resting intake for quite a while, declining over several hours.
The reason for EPOC "is the general disturbance to homeostasis brought on by exercise" (Brooks and Fahey, "Exercise physiology. Human bioenergetics and its applications", Macmillan Publishing Company, 1984).

In the next few days, inspired by different readings, I began thinking of process as the company's homeostatic system.
I've often claimed that companies have their own immune system, actively killing ideas, concepts and practices misaligned with the true nature of the company ("true" as opposed to, for instance, a theoretical statement of the company's values).
However, the homeostatic system has a different purpose, as it is basically designed to (wikipedia quote) "allow an organism to function effectively in a broad range of environmental conditions". Once you replace "organism" with "team", this becomes the best definition of a good development process that I've ever (or never :-) been able to formulate.

It is interesting to understand that the homeostatic system is very sophisticated. It works smoothly, but can leverage several different strategies to adapt to different external conditions. In fact, when good old Gerald Weinberg classified company's cultural patterns on the 1-5 scale, he called level-2 companies "Routine" (as they always use the same approach, without much care for context), level-3 "Steering" (dynamically picking a routine from a larger repertoire) and level-4 "Anticipating" (you figure :-). Of course, any company using the same process every time, no matter the process (Waterfall, RUP-inspired, XP, whatever), is at level 2.

For instance, if we find ourselves working with stakeholders with highly conflicting requirements and we don't apply a more "formal" process based on the initial assessment of viewpoints and concerns (see, for instance, my posts Value and Priorities and Requirements and Viewpoints for a few references) because "we usually gather users stories, implement, and get feedback quickly", then we're stuck in a Routine culture. Of course, if we always apply the viewpoints, we're stuck in a Routine culture too.

Well, there would be much more to say, especially about contingent methodologies and about congruency (another of Weinberg's favorite concepts) between company's process and company's strategy, but time is up again, so I'll leave you with a question. We all know that some physical activity is good for our health. Although high-intensity exercises bring disturbance to homeostasis, in the end we get a better homeostatic system. So the question is, if process is the company's homeostatic system, what is the equivalent of good training? How do we bring a dose of periodic, healthy disturbance to make our process stronger?

Labels: , ,