Sunday, March 07, 2010 

You can't control what you can't …

… measure, Tom de Marco used to say ("Controlling Software Projects: Management, Measurement, and Estimation", 1982). Tom recently confessed he no longer subscribes to that point of view. Now, I like Tom and I've learnt a lot from him, but I don't really agree about most of what he's saying in that paper.

Sure, the overall message is interesting: earth-shaking projects have a ROI so big that you don't really care about spending a little more money. But money isn't the only thing you may need to control (what about time, and your window of opportunity?) and not each and every project can be a earth-shaking project. If you need to comply with some policy or regulation by a given date, it may well be a moot project, but you better control for time :-). More examples (tons, actually) on demand. Relinquishing control is a fascinating concept, and by all means, if you can redefine your projects so that control is no longer necessary, just do it. But frankly, it's not always an option.

Still, can we control what we can't measure? As usual, it depends. It depends on what you want to control, and on your definition of control. We can watch over some things informally, that is, using a rough, imprecise, perhaps intuitive measure ("feeling") and still keep inside reasonable boundaries. This might be enough to be "in control". As others have noted (see for instance Managing What You Can’t Measure) sometimes all we need is a feeling that we're going off track, and a sensible set of tactics to get back on.

All that said, I feel adventurous enough today :-) to offer my own version of Tom's (repudiated) law. I just hope I won't have to take it back in 30 years :-).

You can't control what you can't name.

I would have said "define", but a precise definition is almost like a measure. But if you can't even name the concept (which, yes, requires at least a very informal definition of the term), you're consciously unaware of it. Without awareness, there is no control.

I can say that better: you can't control it intentionally. For a long time, people have controlled forces they didn't fully understand, and perhaps couldn't even name, for instance in building construction. They did that through what Alexander called the unselfconscious process, by relying on tradition (which was largely based on trial and error).

I see this very often in software projects too. People doing things because tradition taught them to do so. They don't really understand why - and would react vehemently if you dare to question their approach or suggest another way. They do so because tradition provides safety, and you're threatening their safety.

The problem with the unselfconscious process is that it doesn't scale well. When the problem is new, when the rate of change in the problem domain increases, whenever the right answer can't be found in tradition, the unselfconscious process doesn't work anymore. We gotta move to the selfconscious process. You gotta learn concepts. Names. Forces. Nonlinear interactions. We gotta think before we do. We gotta ask questions. Question the unquestionable. Move outside our comfort area. Learn, learn, learn.

Speaking of learning, I've got something to say, which is why I wrote this post in the first place, but I'll save that for tomorrow :-).

Labels: , , ,

Friday, February 12, 2010 

Form vs. Function: a Space Odyssey

I was teaching Object Oriented Design past week, and I mentioned the interplay between form and function (form follows function; function follows form). I'm rather cautious not to spend too much time on philosophy, although a little philosophy shouldn't hurt, and people who know me tend to expect a short philosophical digression every now and then.

Function follows form: that is to say, the shape of an object will suggest possible uses. Form follows function: the intended usage of an object will constrain, and therefore guide, its physical shape. This is true for software objects as well. It's just a different material, something we can't immediately see and shape with our hands.

Realizing that function follows form is a pivotal moment in the development of intelligence. You probably remember the opening of 2001: A Space Odyssey. The apes are touching the monolith and, shortly after, one of them is playing with a bone and bang! - it's not just a bone anymore: it's a tool. Function follows form. This chapter is known as "The Dawn of Man", and rightly so.

Watch a little more, and you'll see a doughnut-shaped space station. That's a very good example of form following function (exercise for the reader :-)

By the way, if you have never seen that apes stuff till now :-), here it is, at least until it gets removed from YouTube...

Labels: , ,

Wednesday, July 08, 2009 

When in doubt, do the right thing

The bright side of spending most of my professional time on real-world projects is that I have an endless stream of inspiration, and what is even more important, the possibility of trying out new ideas, concepts, and methods. The dark side is that the same source of inspiration is taking away the precious time I would need to encode, structure, articulate knowledge, that therefore remains largely implicit, tacit, intuitive. The pitch black side is that quite often I'd like to share some real-world story, but I can't, as the details are kinda classified or just to protect the innocent. Sometimes, however, the story can be told with just a little camouflage.

Weeks ago, I was trying to figure out the overall architecture of a new system, intended to replace an obsolete framework. I could see a few major problems, two of which were truly hard to solve without placing a burden on everyone using the framework. Sure, we had other details to work out, but I could see no real showstoppers except for those two. The project manager, however, didn't want to face those problems. She wanted to start with the easy stuff, basically re-creating structures she was familiar with. I tried to insist about the need to figure out an overall strategy first, but to no avail. She wanted progress, right here, right now. That was a huge mistake.

Now, do not misunderstand me: I'm not proposing to stop any kind of development before you work every tiny detail out. Also, in some cases, the only real way to understand a system is by building it. However, building the wrong parts first (or in this case, building the easy parts first) is always a big mistake.

Expert designers know that in many cases, you have to face the most difficult parts early on. Why? Because if you do it too late, you won't have the same options anymore; previous decisions will act like constraints on late work.

Diomidis Spinellis has recently written a very nice essay on this subject (IEEE Software, March/April 2009). Here is a relevant quote: On a blank sheet of paper, the constraints we face are minimal, but each design decision imposes new restrictions. By starting with the most difficult task, we ensure that we’ll face the fewest possible constraints and therefore have the maximum freedom to tackle it. When we then work on the easier parts, the existing constraints are less restraining and can even give us helpful guidance.

I would add more: even if you take the agile stance against upfront design and toward emergent design, the same reasoning applies. If you start with the wrong part, the emergent design will work against you later. Sure, if you're going agile, you can always refactor the whole thing. But this reasoning is faulty, because in most cases, the existing design will also limit your creativity. It's hard to come up with new, wild ideas when those ideas conflict with what you have done up to that moment. It's just human. And yeah, agile is about humans, right? :-)

Expert designer start with the hard parts, but beginners don't. I guess I can quote another nice work, this time from Luke Hohmann (Journey of the Software Professional - a Sociology of Software Development): Expert developer's do tend to work on what is perceived to be the hard part of the problem first because their cognitive libraries are sufficiently well developed to know that solving the "hard part first" is critical to future success. Moreover, they have sufficient plans to help them identify what the hard part is. Novices, as noted often fail to work on the hard-part-first for two reasons. First, they may not know the effectiveness of the hard part first strategy. Second, even if they attempt to solve the hard part first, they are likely to miss it.

Indeed, an expert analyst, or designer, knows how to look at problems, how to find the best questions before looking for answers. To do this, however, we should relinquish preconceived choices. Sure, experts bring experience to the table, hopefully in several different fields, as that expands our library of mental plans. But (unlike many beginners) we don't approach the problem with pre-made choices. We first want to learn more about the forces at play. Any choice is a constraint, and we don't want artificial constraints. We want to approach the problem from a clean perspective, because freedom gives us the opportunity to choose the best form, as a mirror of the forcefield. By the way, that's why zealots are often mediocre designers: they come with too many pre-made choices, or as a Zen master would say, with a full cup.

Of course, humans being humans, it's better not to focus exclusively on the hard stuff. For instance, in many of my design sessions with clients, I try to focus on a few simple things as we start, then dig into some hard stuff, switch back to something easy, and so on. That gives us a chance to take a mental break, reconsider things in the back of our mind, and still make some progress on simpler stuff. Ideally, but this should be kinda obvious by now, the easy stuff should be chosen to be as independent/decoupled as possible from the following hard stuff, or we would be back to square one :-).

In a sense, this post is also about the same thing: writing about some easy stuff, to take a mental break from the more conceptual stuff on the forcefield. While, I hope, still making a little progress in sharing some useful design concept. See you soon!

Labels: , , , , , ,

Sunday, October 26, 2008 

Microblogging is not my thing...

A few weeks ago I got a phone call from a client. They want to insource a mission-critical piece of code. I talked about the concept of Habitable Software and thought I could write something here.

As I started to write, words unfolded in unexpected rivers. Apparently, I've got too much to say [and too little time].

So, I tried to use a mind map to get an overview of what I was trying to say.

Here it is (click to get a full-scale pdf):

Strictly speaking, it's not even a mind map, as I drew a graph, not a tree. I find the tree format very limiting, which is probably a side-effect of keeping a lot of connections in my mind.

Another side effect is that I find micro-blogging unsatisfactory. Sure, I could post something like:

interesting book, take a look

and get over, but it's just not my style.

Anyway, I'll try to keep this short and just add a link to the presentation on form Vs. function that I mentioned in the mind map: Integrating Form and Function. Don't mind the LISP stuff :-). That thing about the essential and contingent interpreter is great.

More on all this another time, as I manage to unravel the fabric of my mind :-)

Labels: , , , ,

Sunday, October 12, 2008 

Some Small Design Issues (part 1)

In a previous post, I talked about some small, yet thorny design problems I was facing. As I started writing about them, it became clear that real-world design problems are never so small: there is always a large context that is somehow needed to fully understand the issues.

Trying to distill the problem to fit a blog post is a nightmare: it takes forever, it slows me down to the point I'm not blogging anymore, and is exactly the opposite of what I meant when I wrote Blogging as Destructuring a few years ago. On a related note, Ed Yourdon (at his venerable age :-) is moving toward microblogging for similar reasons.

Still, there is little sensible design talk around, so what does a [good] man gotta do? Simplify the problem even more, split the tale in a few episodes, and so on.
I said "tale" because I'll frame the design problem as a story. I don't mean to imply that things went exactly this way. Actually, I wasn't even there at the time. However, looking at the existing artifacts, it seems reasonable that they somehow evolved that way.
Also, an incremental story is an interesting narrative device for a design problem, as it allows to put every single decision in perspective, and to reason about the non-linear impact of some choices.

Stage 1 - the beginning
We have some kind of industrial process control system. We want to show a few process variables on a large-size numeric display, like in the picture below:

At this point the problem is quite simple, yet we have one crucial choice to make: where do we put the new logic?
We have basically three alternatives:
1) inside an existing module/process, that is, "close to the data"
2) in a new/external process, connected via IPC to the existing process[es]. Connection might be operating in a push or pull mode, depending on update rate and so on (we'll ignore this for sake of simplicity).
3) in a new/external process, obtaining data through a [real-time] database or a similar middleware. The existing processes would have to publish the process variables on the database. The new process might pull data or be pushed data, depending on the data source.
Even at this early stage, we need to make an important architectural decision. It's interesting to see that in very small systems, where all the data is stored inside one process, alternative (1) is simpler. Everything we need "is just there", so why don't we add the display logic too?
This is how simple, lean systems get to be complex, fragile, bulky systems: it's just easier to add code where the critical mass is.
So, let's say our guys went for alternative (3). We have one data source where all the relevant variables are periodically stored.

Now, we just need to know which variables we want to show, and in which row. For simplicity, the configuration could be stored inside the database itself, like this (through an OO perspective):

Using an ugly convention, "-1" as a row value indicates that the process variable isn't shown at all.

Stage 2 - into the real world
Customers may already have a display, or the preferred supplier may discontinue a model, or sell a better/cheaper/more reliable one, and so on. Different displays have different protocols, but they're just multi-line displays nonetheless.
Polymorphism is just what the doctor ordered: looking inside the Display component, we might find something like this:

It's trivial to keep most of the logic to get and format data unchanged. Only the protocol needs to become an extension point. Depending on the definition of protocol (does it extend to the physical layer too?) we may have a slightly more complex, layered design, but let's keep it simple - there are no challenges here anyway.

Stage 3 - more data, more data!
Processes gets more and more complex, and customers want to see more data. More than one display is needed. Well, it's basically trivial to modify the database to store the display number as well.
A "display number" field is then added to the Process Variable Descriptor. Note that at this point, we need a better physical display, as the one in the picture above has hard-coded labels. We may want to add one more field to the descriptor (a user-readable name), and our protocol class may or may not need some restyling to account for this [maybe optional] information. The multiplicity between "Everything Else" and "Display Protocol" is no longer 1. Actually, we have a qualified association, using the display number as a key (diagram not shown). No big deal.
Note: at this stage, a constraint has been added, I guess by software engineers, not process engineers: the same process variable can't be shown on two different displays. Of course, a different database design could easily handle this limitation, but it wasn't free, and it wasn't done.

Hmmm, OK, so far, so good. No thorny issues. See you soon for part 2 :-).

Labels: , ,

Tuesday, April 15, 2008 

Old Problems, new Solutions

In a comment to my previous post, I mentioned an half-baked idea about a (truly) modern architecture for GUI applications.

In fact, I've been experimenting with different approaches to GUI construction for "classic" Windows applications, Web applications, Smart Clients calling web services, and even Web applications calling web services for quite a while now.

Some were dead ends. Others have been a successful intermediate step toward something better. I usually pass on the good ideas to my clients, and meanwhile, I keep trying different approaches on small scale, low-risk projects. Because the "traditional" way of building GUI applications sucks big time. So does the traditional thinking about "good" GUI constructions, that is, that you need a tightly integrated language / ide / library / operating system to do anything decent.

I also don't like the idea that I should buy-in a monolithic framework for GUI construction. Different problems are better solved in different ways: we should be free to mix and match different approaches in different forms (within the same application), or even within the single form, or component.

Over time, a few people told me not to bother: Microsoft, Sun, Apple (not to mention a thousand smart guys working for smaller companies) have been working at that for years. It's an old problem, and it's unlikely that anybody can come up with a radically different solution.

It's very poor thinking. There is no limit to human creativity. Take a look at How To Fold A T-Shirt In 2 Seconds. The problem is much older than GUI construction. Uncountable people have been faced with the same problem and they didn't come up with anything similar so far. Sure, there are pros and cons in this technique (as usual), but it's a radically different (and much faster) approach.

You may want to go through the explained version, as I did. I actually tried that a few times, and it works :-).

Labels: , ,

Friday, April 04, 2008 


I'm working on an interesting project, trying to squeeze all the available information from sampled data and make that information useful for non-technical users. I can't provide details, but in the end it boils down to reading a log file from a device (amounting to about 1 hour of sampled data from multiple channels), do the usual statistics, noise filtering, whatever :-), calculate some pretty useful stuff, and create a report that makes all that accessible to a business expert.

The log file is (guess what :-) in XML format, meaning it's really huge. However, thanks to modern technology, we just generated a bunch of classes from the XSD and let .NET do the rest. Parsing is actually pretty fast, and took basically no time to write.
In the end, we just get a huge collection of SamplingPoint objects. Each Sampling point is basically a structure-like class, synthesized from the XSD:

class SamplingPoint
public DateTime Timestamp { // get, set }
public double V1 { // get, set }
// ...
public double Vn { // get, set }

each value (V1...Vn) is coming from a different channel and may have a different unit of measurement. They're synchronously sampled, so it made sense for whoever developed the data acquisition module to group them together and dump them together in a single SamplingPoint tag.

We extract many interesting facts from those data, but for each Vi (i=1...N) we also show some "traditional" statistics, like average, standard deviation and so on.
Reasoning about average and standard deviation is not for everyone: I usually consider an histogram of the distribution much easier to understand (and to compare with other histograms):

Here we see the distribution of V1 over time: for instance, V1 had a value between 8 and 9 for about 6% of the time. Histograms are easy to read, and users quickly asked to see histograms for each V1..Vn over time. Actually, since one of the Vj is monotonically increasing with time, they also asked to see the histogram of the remaining Vi against Vj too. So far, so good.

Now, sometimes I hate writing code :-). It usually happens when my language doesn't allow me to write beautiful code. Writing a function to calculate the histogram of (e.g.) V1 against time is trivial: you end up with a short piece of code taking an array of SamplingPoints and using the V1 and Timestamp properties to calculate the histogram. No big deal.

However, that function is not reusable, exactly because it's using V1 and Timestamp. You can deal with this in at least 3 unpleasant :-) ways:

1) you don't care: you just copy/paste the whole thing over and over. If N = 10, you get 19 almost-identical functions (10 for time, 9 for Vj).

2) you restructure your data before processing. Grouping all the sampled data at a given time in a single SamplingPoint structure makes a lot of sense from a producer point of view, but it's not very handy from a consumer point of view. Having a structure of arrays (of double) instead of an array of structures would make everything so much simpler.

3) you write an "accessor" interface and N "accessors" classes, one for each Vi. You write your algorithms using accessors. Passing the right accessors (e.g. for time and V1) will get you the right histogram.

All these options have some pros and cons. In the end, I guess most people would go with (2), because that brings us into the familiar realm of array-based algorithms.

However, stated like this, it seems more like a "data impedance" problem between two subsystems than a language problem. Why did I say it's the language fault? Because the language is forcing me to access data members with compile-time names, and does not (immediately) allow me to access data members using run-time names.

Don't get me wrong: I like static typing, and I like compiled languages. I know from experience that I tend to make little stupid mistakes, like typing the wrong variable name and stuff like that. Static typing and compiled languages catch most of those stupid mistakes, and that makes my life easier.

Still, the fact that I like something doesn't mean I want to use that thing all the time. I want to have options. Especially when those options would be so simple to provide.

In a heavily reflective environment like .NET, every class can be easily considered an associative container, from the property/data member names to property/data member values. So I shold be able to write (if I wanted):

SamplingPoint sp = ... ;
double d1 = sp[ "V1" ] ;

which should be equivalent to

double d1 = sp.V1 ;

Of course, that would make my histogram code instantly reusable: I'll just pass the run-time names of the two axes. You can consider this equivalent to built-in accessors.

Now, I could implement something like that on my own, using reflection. It's not really difficult: you just have to gracefully handle collections, nested objects, and so on. Unfortunately, C# (.NET) do not allow a nice implementation of the concept, mostly for a bunch or constraints they added to conversion operators: no generic conversion operators (unlike C++), no conversion to/from Object, and so on. In the end you may need a few more casts that you'd like to, but it can be done.

I'll also have to evaluate the performance implications for this kind of application, but I know it would make my life much easier in other applications (like binding smart widgets to a variety of classes, removing the need for braindead "controller" classes). It's just a pity that we don't have this as built-in language feature: it would be much easier to get this right (and efficient) at the language level, not at the library level (at least, given C# as it is now).

Which brings me to the concept of symmetry. A few months ago I stumbled upon a paper by Jim Coplien and Zhao Liping (Understanding Symmetry in Object-Oriented Languages, published in Journal of Object Technology, an interesting, free publications that's filling the void left by the demise of JOOP). Given my interest on the concept of form in software, the paper caught my attention, but I postponed further thinking till I could read more on the subject (there are several papers on symmetry in Cope's bibliography, but I need a little more time than I have). A week ago or so, I've also found (and read) another paper from Zhao in Communications of ACM, March 2008: Patterns, Symmetry, and Symmetry breaking.

Some of their concepts sound odd to me. The concept of symmetry is just fine, and I think it may help to unravel some issues in language design.
However, right now the idea that patterns are a way to break symmetry doesn't feel so good. I would say exactly the opposite, but I really have to read their more mathematically-inclined papers before I say more, because words can be misleading, while theorems usually are not :-).

Still, the inability to have built-in, natural access to fields and properties through run-time names struck me as a lack of symmetry in the language. In this sense, the Accessor would simply be a way to deal with that lack of symmetry. Therefore it seems to me that patterns are a way to bring back symmetry, not to break symmetry. In fact, I can think of many cases where patterns "expose" some semantic symmetry that was not accessible because of (merely) syntactic asymmetry.

More on this as I dig deeper :-).

Labels: , , , ,

Thursday, March 27, 2008 

Zen and the Art of Fixing

A little known fact about me is that I like fixing things. Mostly TV sets, but over the years I've tried to repair (with various degrees of success :-) any electrical, electronic or mechanical device that needed some fixing. Indeed, when I was about 14 I made a small business out of rescuing old vacuum-tube radios, fixing and then selling them to collectors.
Even today, I occasionally talk about diagnosing and repairing a defective TV when teaching about modular design, modular reasoning, design for testability, and so on.

In fact, fixing a device has a lot in common with fixing a buggy program. Some devices (like old washing machines) are relatively simple. You just need to know some basic concepts, and apply some common sense. Sure, it may take some creativity to fix even the simplest device (especially if you can't find the right replacement parts), but overall, the problem is often self-evident, and you "only" need to resort to your ability, often just manual ability.

Electronic devices, however, can fail in complex, even puzzling ways. You need a better understanding of what's going on under the hood. You may need special tools (you can't get much far repairing a TV if you ain't got an oscilloscope) but sometimes it's just plain old intuition (or sheer luck :-). You need to know some tricks of the trade (like using a light bulb to discriminate problems in the power supply vs. horizontal deflection), but overall, what you need more is rational system thinking.

The same is true when reasoning about complex software failures. We often have a large numbers of parts (hardware, firmware, drivers, OS, libraries, our own code) which can fail for a large number of reasons. Your best allied is rational system thinking. Your worst enemy is the all-so-common uncircumstantiated certainty that the fault must lie exactly somewhere (usually outside our own code :-). Overall, I would say my experience fixing stuff has made me a better troubleshooter, helping me to find obscure bugs in systems I knew very little about.

Still, I don't like to fix computers. I guess I'm already spending too much time around computers, and besides, today the integration scale is so high that you can hardly fix a broken motherboard. However, there is always an exception, and this one is interesting enough to be worth telling.

A couple of years ago (yeah :-) I rescued a notebook before it was thrown away. After a fall, it wouldn't even turn on. The CD-ROM was visibly damaged, but the screen was intact. I took it home and left it alone for a few months, till I had some time to kill.

I'm usually lucky, and in fact, I discovered that by pushing the power plug a little heavier than usual, the notebook would indeed power on. That's usually just a broken joint; it took forever :-) to disassemble the notebook and expose the PCB, but after that, it was quite easy to fix. Now the computer would turn on, start booting XP, and die a few seconds later. I suspected the HD was damaged too, and replaced it with a spare one (with W2K installed). Similar story: it would start booting the OS, then dump a blue screen reporting that “the boot device was not accessible”.

That's usually due to a broken IDE controller, or a faulty RAM chip. I tried replacing the RAM, but still got the same problem. However, the IDE controller was somehow working, as the computer would indeed start booting. Weird :-).

I didn't give up and got another HD, with good old MS-DOS installed. The notebook booted like a charm, and all the applications seemed to work. Weird again: the IDE controller seemed fine. Again, it could be a faulty RAM chip, since MS-DOS only uses the first 640KB.

I connected a USB CD-ROM and tried an old version of Knoppix on CD: knoppix uses all the available RAM, so that was like a definitive RAM test. It worked fine, but as soon as I tried using the HD, it would simply lock up. So, RAM was fine, the IDE controller was fine under MS-DOS but failed under other operating systems.
My diagnosis was that DMA was at fault. I tried to disable DMA at the BIOS level with no luck. I also disabled DMA in that W2K HD (using another PC, of course), but it would still lock, just like Knoppix.

At that point, I contemplated using an external HD (via USB), and perhaps installing a tweaked version of XP which can boot from a USB device (details on the BartPE page). I could even rewire one of the USB ports and install the whole thing internally, since the missing CD-ROM left a lot of space. But it looked like a damn lot of work :-), so I did nothing and left the notebook around for more than one year.

A few days ago, having a little time to kill again, I tried a different experiment. I got a 1GB USB stick, downloaded the latest version of Knoppix, and put it on the stick (instead of a CD-ROM). I followed this tutorial to speed things up.
It worked fine, and booting from the USB stick was very quick. However, at that point I noticed (from the boot log) that this newer version of Knoppix ran with DMA disabled (I guess the old one I tried before didn't). Time to test my DMA controller theory!

I put an HD inside, booted Knoppix from the USB stick, and guess what, no hang-up! I could access the HD with not problem whatsoever. At that point, it was hard to resist: I had to try installing knoppix on the HD itself. Again, I followed a tutorial to speed things up: this one is about a previous version, but still valid. The proof of the pudding is in the eating: I removed the USB stick, tried booting and Lo!, it worked. The world is still somehow deterministic :-).
By the way: Knoppix takes forever to boot from the HD. Overall, it would be better to keep the USB stick, maybe rewiring a USB port to keep everything inside.

Now, in a perfect world, I would have found a way to really "fix" the notebook, that is, to repair the DMA controller. I suspect it's just another loose joint from the fall. Most likely, some pin of some surface-mounted chip suffered some mechanical stress. This world, however, ain’t perfect, so I did nothing else, leaving it as a working Knoppix notebook. Fun is over, time to get back to work :-)

Labels: ,

Wednesday, March 19, 2008 

(Simple) Metrics

I've been using metrics for a long time (certainly more than 10 years now). I've been using metrics to control project quality (including my own stuff, of course), to define acceptance criteria for outsourced code, to understand the way people work, to "smell" large projects before attempting a refactoring activity, to help making an informed refactor / rewrite decision, to pinpoint functions or classes in need of a careful review, to estimate residual bugs, an so on.

Of course, I use different metrics for different purposes. I also combine metrics to get the right picture. In fact, you can now find several tools to calculate (e.g.) code metrics. You can also find many papers discussing (often with contradictory results) the correlation between any given metric and (e.g.) bug density. In most cases, those papers are misguided, as they look for correlation between a single metric and the target (like bug density). Reality is not that simple; it can be simplified, but not to that point.

Consider good old cyclomatic complexity. You can use it as-is, and it can be useful to calculate the minimum reasonable number of test cases you need for a single function. It's also known that functions with higher cyclomatic complexity tend to have more bugs. But it's also well known that (on average) there is a strong, positive correlation between cyclomatic complexity (CC) and lines of code (LOC). That's really natural: long functions tend to have a complex control flow. Many people have therefore discounted CC, as you can just look at the highly correlated (and easier to calculate) LOC. Simple reasoning, except it's wrong :-).

The problem with that, again, is trying to use just one number to understand something that's too complex to be represented by a single number. A better way is to get both CC and LOC for any function (or method) and then use quadrants.

Here is a real-world example, albeit from a very small program: a smart client invoking a few web services and dealing with some large XML files on the client side. It has been written in C# using Visual Studio, therefore some methods are generated by the IDE. Also, the XML parser is generated from the corresponding XSD. Since I'm concerned with code which is under the programmer's control, I've excluded all the generated files, resulting in about 20 classes. For each method, I gathered the LOC and CC count (more on "how" later). I used Excel to get the following picture:

As you can see, every method is just a dot in the chart, and the chart has been split in 4 quadrants. I'll discuss the thresholds later, as it's more important to understand the meaning of each quadrant first.

The lower-left quadrant is home for low-LOC, low-CC methods. These are the best methods around: short and simple. Most code ought to be there (as it is in this case).

Moving clockwise, the next you get (top-left) is for high LOC, low CC methods. Although most coding standards tend to somehow restrict the maximum length of any given method, it's pretty obvious that a long method with a small CC is not that bad. It's "linear" code, likely doing some initialization / configuration. No big deal.

The next quadrant (top-right) is for high LOC, high CC methods. Although this might seem the worst quadrant, it is not. High LOC means an opportunity for simple refactoring (extract method, create class, stuff like that). The code would benefit from changes, but those changes may require relatively little effort (especially if you can use refactoring tools).

The lower-right quadrant is the worst: short functions with high CC (there are none in this case). These are the puzzling functions which can pack a lot of alternative paths into just a few lines. In most cases, it's better to leave them alone (if working) or rewrite them from scratch (if broken). When outsourcing, I usually ask that no code falls in this quadrant.

For the project at hand, 3 classes were in quadrant 3, so candidate for refactoring. I took a look, and guess what, it was pretty obvious that those methods where dealing with business concerns inside the GUI. There were clearly 3 domain classes crying to be born (1 shared by the three methods, 1 shared by 2, one used by the remaining). Doing so brought to better code, with little effort. This is a rather ordinary experience: quadrants pinpoint problematic code, then it's up to the programmer/designer to find the best way to fix it (or decide to leave it as it is).

A few words on the thresholds: 10 is a rather generous, but somewhat commonly accepted threshold for CC. The threshold for LOC depends heavily on the overall project quality. I've been accepting a threshold of 100 in quality-challenged projects. As the quality improves (through refactoring / rewriting) we usually lower the threshold. Being a new development, I adopted 20 LOC as a rather reasonable threshold.

As I said, I use several different metrics. Some can be used in isolation (like code clones), but in most cases I combine them (for instance, code clones vs. code stability gives a better picture of the problem). Coupling and cohesion should also be considered as pairs, never as single numbers, and so on.
Quadrants are not necessarily the only tool: sometimes I also look at the distribution function of a single metric. This is way superior to what too many people tend to do (like looking at the "average CC", which is meaningless). As usual, a tool is useless if we can't use it effectively.

Speaking of tools, the project above was in C#, so I used Source Monitor, a good free tool working directly on C# sources. Many .NET tools work on the MSIL instead, and while that may seem like a good idea, in practice it doesn't help much when you want a meaningful LOC count :-).

Source Monitor can export in CSV and XML. Unfortunately, the CSV didn't contain the detailed data I wanted, so I had to use the XML. I wrote a short XSLT file to extract the data I needed in CSV format (I suggest you use the "save as" feature, as unwanted spacing / carriage returns added by browsers may cripple the result). Use it freely: I didn't put a license statement inside, but all [my] source code in this blog can be considered under the BSD license unless otherwise stated.

Labels: , , ,

Saturday, February 09, 2008 

Do Less

In many software projects lies some dreaded piece of code. Legacy code nobody wants to touch, dealing with a problem nobody wants to look at. In the past few days, I've been involved in a (mostly unpleasant :-) meeting, at the end of which I learnt that some "new" code was actually dealing with a dreaded problem by calling into legacy code. Dealing with that problem inside the "new" code was deemed risky, dirty, hard work.

Now, as I mentioned in my Work Smarter, Not Harder post, I don't believe in hard work. I believe in smart work.

After thinking for a while about the problem, I formulated an alternative solution. It didn't come out of nowhere: I could see the problem frame, and that was a language compiler/interpreter frame, although most of the participants didn't agree when I said that.
They had to agree, however, that my proposed solution was simple, effective, risk-free, and could be actually implemented in a few days of light :-) work. Which I take as an implicit confirmation that the problem frame was right.
I also had to mix some creativity, but not too much. So I guess the dreaded problem can now be solved by doing much less than expected.

This could explain the title of this post, but actually, it doesn't. More than 20 years ago, good ol' Gerald Weinberg wrote:
Question: How do you tell an old consultant from a new consultant?
Answer: The new consultant complains, "I need more business." The old consultant complains, "I need more time."

I guess I'm getting old :-), and I should probably take Seth Godin's advice. Seth is best known for "Purple Cow", a wonderful book that ought to be a mandatory reading for everyone involved in creating a new product. The advice I'm thinking of, though, comes from Do Less, a short free pamphlet he wrote a few years ago. There he writes:
What would happen if you fired half of your clients?

Well, the last few days really got me thinking :-)).

Labels: , ,

Sunday, February 03, 2008 

A few free tools

A few free tools I've found myself using and/or recommending in the last few days:

Refactor!™ for C++
A refactoring plug-in for Visual Studio. Even if you only use Extract Method, you'll like it. Downside: your IDE may get a bit sluggish.

A different refactoring/code mining tool. If you work on a large project, you'll love the ability to discover "redundant" #include in your header files. Disclaimer: I haven't tried that feature on really sick include files :-). If you do, let me know...

(thanks to Fulvio Esposito who told me about it)
A Rational Rose lookalike in Delphi (open source). One order of magnitude faster than the average java behemoth. Limited to UML 1.4, but easy to use, reasonably stable, does a reasonable job at importing Rose files, and the "auto layout" feature is better than average.

(thanks to Roberto Rossi who told me about it)
An open-source virtual machine for Windows / Linux. Reasonably stable, faster than average. I wish I had more time to learn about the internals and the architecture!

As usual, guys, this is not an endorsement or whatever else. Try that stuff at your own risk :-).

Labels: ,

Sunday, December 02, 2007 

More synchronicity

While reading the same issue of IEEE Software that I mentioned in my previous post, I came across a paper from Ed Yourdon, "Celebrating Peopleware's 20th Anniversary". I mentioned Peopleware while answering a comment to Process as the company's homeostatic system, but so far, it's just a coincidence.

It got more interesting as I went further, as Ed says:
"my colleague Larry Constantine and I had borrowed an even earlier collection of Alexander's ideas from his 1964 book Notes on the Synthesis of Form as the basis for the structured-design concepts of coupling and cohesion."
Oh, look, Notes on the Synthesis of Form, that's another interesting coincidence :-).

Speaking of cohesion, I should note that the process described by Alexander (modeling relationships between requirements as a weighted graph) has strong resemblances to the process adopted in KAOS (which I've mentioned in several posts now). The purpose is different however, as Alexander aims to derive clusters of highly cohesive requirements mechanically, while KAOS is leaning more on the soft side, allowing people to "see" the interplay between requirements.

Funny enough, in the same issue there is a paper by Simon Buckingham Shum ("There's Nothing Like a Good Argument..."), which describes a tool (Compendium) "providing a flexible visual interface for managing the connections between information and ideas" (from the compendium website).
I haven't tried it out yet, but from the screenshots, it seems to embody everything we need to apply a KAOS-like technique to requirements analysis, and also keep track of major design decisions.

Gee, everything seems so interconnected these days :-)

Labels: , , ,

Wednesday, October 24, 2007 

More on evolving (or rewriting) existing applications

Just when I wanted to be more present in my blog, I had to survive the loss of about 200 GB of data. Nothing vital, but exactly for that reason, not routinely backed up either. Anyway, sometimes it's a relief to lose a little information :-).

In my previous post I've only scratched the surface of a large problem. If rewriting the application from the core is so dangerous (unless you can break the feedback loop in the diagram, which you can, but not for free), are we left with any viable option?

We all heard about refactoring in the last few years. Refactoring tools are getting better, to the point that you can actually trust them with your production code. However, refactoring can only bring you so far. You won't change a fat client into a web application by incremental, small scale refactoring. But careful refactoring is truly helpful when dealing with small scale problems. I've recently seen some C/C++ code full of comments, and guess what, the backtalk of those comments (see my "Listen to your tools and materials") is actually "please remove me and refactor the code instead". Refactoring is also hard when the code is strongly coupled with an outdated (or ill-structured) relational database, without any kind of insulation layer. So, refactoring is an interesting technique, and can often be applied in parallel with some more aggressive strategy, but is not really helpful when you need large scale changes.

An interesting strategy, that can be applied in more than a few cases, is the modular rewriting of applications. The concept is not new: you exploit (or improve by refactoring and then exploit) the existing architecture, by gradually replacing modules. With a little oversimplification, we could say that the real problem is which modules you want to replace first.

Again, this depends on a number of factors:

- do you have a different target architecture? If so, is there any module that you can replace in the existing system, and which will effectively move, almost unchanged, to the new architecture?

- are you just tired of maintenance problems? Then use quadrants to isolate the best candidates. Troubled modules with small size, or troubled modules with minimal coupling from the rest, should be your first candidates. Get as much value as you can, as soon as you can. Get momentum. We're not in the 80s anymore; flat value delivery curves are a recipe from project cancellation.

- are you "just" changing technology, or are you trying to innovate the application? Honestly, most companies can't innovate. Not by themselves, and quite often, not even with external help. They're prisoners of their own artifacts. They think of their problems in terms of the existing implementation. They will never, for instance, conceive a real web 2.0 application, because they can't think that way.
If you want to innovate the application, start with innovative features in mind. Design new modules, then bridge them with the existing system. While designing the bridge, push the envelope a little: can we erode a little of the existing system, an replace some subsystem with something more appropriate?
Doing so will move you toward an Incremental Funding project, which will gradually erode the existing system until it will make economic sense to scrap the rest.

- Note that this assumes that innovation is happening at the fringes of your system, not at the core. This is often true (see also "Beyond the core" by Chris Zook, Harvard Business School Press, for a business-oriented perspective), but sometimes, the real innovation requires a new core. Say you need a new core. Are you sure? Maybe you "just" need a new product, barely integrated with the existing system. Don't say no so soon. Play with the idea. Don't get into a self-fulfilling expectation by telling yourself that everything must be tightly integrated into a new core.

- Are you moving toward a truly better architecture, or just following a technology trend? Maybe your company was much less experienced when the current system was built. Still, are you sure the new core won't become brittle very soon? Do you really need a large core? Can't you decompose the system toward (e.g.) a loosely-coupled, service-oriented architecture, instead of a tightly-coupled, database-centric architecture (maybe with a web front end, big deal :-).

- can you apply some non-linear (a.k.a. lateral) thinking? Not familiar with lateral thinking? De Bono should be a required reading for software developers, and even more so for project managers and designers!

- is your new/target architecture aligned with your company strategy? If not, think twice!

OK, it's getting late again. See you soon, I hope, with some reflections on the concept of form (vs function) in the software realm.

Labels: , ,

Sunday, October 14, 2007 

Evolving (or rewriting) existing applications

I've been conspicuously absent from my blog in the last two weeks. Hectic life takes its toll :-), and it's not always possible to talk here about what I'm actually doing.
I'm often involved in very different activities, from weird bugs at the hardware/software interface, to coding tricky parts, to designing [small] application-specific frameworks, to making sense of nonsensical requirements. Recently, however, I've been helping a few customers make some rather fundamental decision about how to evolve (or rewrite) their applications.

Indeed, a significant subset of existing applications have been developed with now-obsolete technologies ("old" languages, framework, components, architectures), or even obsolete business concepts (e.g. a client-side application sold as a product, instead of a web application sold as a service).

Now, when you're planning the next generation of a successful application, you often end up trapped into a very natural, logical, linear thinking: we start from the core of the application, build a new/better/modern one, then we keep adding features till we have a complete application.
Sometimes, it's not even about starting at the core: you start with a large framework, and expect that you'll be able to build your applications in a snap (when the humongous framework is completed).

Now, this seems very logical, and if you look at it from a technology perspective, it makes a lot of sense. By dealing with the core first, you have the best chance to make a huge impact into the foundations, making them so much better. All those years on the market taught you a lot, so you know how to make the core better.
It's also easier this way: the new application will be entirely based on the new technology, from the bottom up. No need to mix old and new stuff.

As usual, a successful development strategy is completely context dependent! If your application is small, the natural strategy above is also the best overall strategy. In a relatively short time (since the application is small) you'll have a clean, brand-new application.
Unfortunately, the strategy does not scale at all to large applications. Let's see a few (well known, and often ignored) problems:

- The value delivery curve is pretty flat. Looks more like fig.1 in an old post of mine, as you can't really sell the new application till is finished.
People usually argue that by choosing the "right" core, they can start selling the core before the whole application has been ported. Yeah, sure, maybe in some alternative reality, but truth is, in many cases your existing customers won't downgrade to a less powerful, usually incompatible application (unless they're having major headaches from the old app).
New prospects won't be so exhilarated by a stripped-down core, either. Over the years, for a number of reasons, it' very likely that most innovation happened at the fringes of the old application, not at the core. You're now taking this stuff away for a potentially long time. Your old product will be competing with your new product, and guess what, lack of features tends to be rather visible.

- Although management may seem initially inclined to support your decision to rebuild everything from scratch, they won't stay silent as the market erodes. Soon they will (very reasonably) ask you to add some stuff to the existing application as well.
Now, by doing so, you'll slow down the development of the new application (resource contention) and you'll also create a backlog of features to be ported to the new application, once it is finished.

- All this conjures for very long development times. If you're working with intrinsically unstable technologies (like web applications), there is even a significant chance that your new application will be technologically obsolete before it gets to the market!

Let's try to model these (and a few more) issues using a diagram of effects

You may want to spend a little time looking at the different arrows, and especially at self-sustaining feedback loops. It's not hard to see that this is a recipe for having hard times. Yet companies routinely embark into this, because redoing the core is the most logical, most technologically sound thing to do. Unfortunately, it doesn't always make business sense
As usual, there are always other choices. As usual, they have their share of problems, and often requires better project management practices and skilled software designers. They also happen to be very context-dependent, as you have to find a better balance between your business, your current application, your new application.
Ok, time and space are up :-), I'll add a few details later.

Labels: , ,

Wednesday, March 15, 2006 

Organizational Structure

Tomorrow and the day after I'll be teaching my Software Project Management course. From previous discussions, I could foresee that there will be some interest on organizational structure (functional teams, project teams, matrix structure and the like), so I decided to add some custom material. However, the traditional literature on organizational strategy (I've got more than 20 books on the subject) is very theoretical, and definitely not grounded in software, which is an hallmark of my course. Therefore, I took a rather original road, organized around two major tenets.
1) The organizational structure must support the company strategy. Any misalignment will be a cause for friction and inefficiency. An effective, barebone classification of company strategy can be found in Michael Tracy, Fred Wiersema, "The Discipline of Marker Leaders". A couple of papers from Stan Rifkin ("Why Software Process Innovation Are Not Adopted", IEEE Software, July/August 2001 and "What Makes Measuring Software So Hard?", IEEE Software, May/June 2001) give some hints on the relevance of this work for software development.
2) The organizational structure must be aligned with the product architecture. This is a consequence of Conway's Law: "Any organization which designs a system will inevitably produce a design whose structure is a copy of the organization's communication structure" (Melvin E. Conway, "How Do Committees Invent?", Datamation, April, 1968). Convay's paper is one of those classics often quoted and never read, partially because they used to be hard to find. Fortunately, here is an online version. Luke Hohmann ("Journey of the Software Professional", Prentice-Hall) also had an interesting intuition, that is, the best organizational structure for a product is not necessarily fixed: it may change over time (this is probably an agile management frontier right now :-). Finally, some useful information on how the traditional organizational structures (functional, matrix, etc.) performs on software projects can be found in Jack T. Marchewka, "Information Technology Project Management", John Wiley & Sons.
Quite a mouthful, but being a good manager takes much more than technical skills...
For another interesting point of view on organizational structure, see my post An organization consists of conversations.

Labels: , , , ,

Saturday, July 16, 2005 

"An organization consists of conversations"

The relationship between software engineers and business (management, marketing, etc) has never been idyllic - for many reasons I won't touch yet. Still, at some point, many talented software engineers are asked to outgrow their role and move into management, or in other business roles.
This is never an easy step. It's not just about new concepts and skills to master - software engineers are usually quick to grasp new ideas; it's about learning new values, being able to shift into a different mindset, and then back when is needed. It takes time (and some effort), and most people don't take that time. They assume they will succeed solely on the basis of technical excellence. Good luck :-).
Meanwhile, I see a minority of excellent software engineers growing a sincere interest (and appreciation) for management and organizational issues. They may appreciate this pamphlet: Notes on the Role of Leadership and Language in Regenerating Organizations. I've stumbled into it about one year ago, and found it extremely interesting. Here is my favorite paragraph:
an organization consists of conversations:
who talks to whom, about what.
Each conversation
is recognized, selected, and amplified
(or ignored) by the system.
Decisions, actions, and a sense of valid purpose
grow out of these conversations.

Now, what is the conversational nature of your company? :-)

Labels: , ,