Carlo Pescio - blog

SmartFP™ paper (and tool) online

2008-07-22T14:27:00.000+02:00

As promised, I've uploaded a free, simple tool to calculate Function Points using a decision tree. I've also uploaded a (draft) paper describing the overall approach. The paper is still missing a case study, which would help, but I just wanted to put the whole thing online. I'll add the case study, and a few more details, before submitting the paper for publication.

The decision tree approach is quite simple, especially if you have some knowledge of function points. Although it may seem like a small change in perspective from the usual "counting" approach, the result is that we can save a lot of time doing a function point estimate, and in many cases we also get more robust results.

Experiences and feedback are welcome, as usual. You can find the whole thing on the SmartFP page.

Note: as I plan to make more tools and libraries freely available, I've also created a new "tools" page. So far, there is only a link to BetterEstimate and SmartFP, but more will come...

Almost there

2008-07-20T16:59:00.000+02:00

In a previois post I mentioned I'll soon add a new [draft] paper to my website, together with a free tool. It's almost done: I just have to add a short case study and review the text a little. Or maybe I'll leave the case study for the next revision, dunno yet.

The tool has been put to use for a while now, and it seems like it's doing its [simple] job. The icon is ugly but I can live with that :-).

So, everything should be in place in the next few days. Then I'll work on the paper a little more and see if it can get past the academic publishing gates :-).

Quote of the Day

2008-07-09T21:42:00.003+02:00

"Out of clutter find simplicity;
From discord find harmony;
In the middle of difficulty lies opportunity."
Albert Einstein

Note: I was teaching requirements engineering today, and I spent more time than usual talking about viewpoints, task descriptions, personas, the need for the analyst to invent requirements and so on.
For some reason, while walking to the hotel those words of wisdom came back to my mind. Indeed, those principles apply equally well to analysis, design, or coding, and in a sense, they are at the core of quality software development.
The pursuit of simplicity and harmony is a recurring theme in Einstein's writings, some of which I've quoted before .

More on Code Clones

2008-06-25T18:24:00.004+02:00

I've been talking about code clones before. It's a simple metric that I've used in several projects with encouraging results.

Till no long ago, however, I thought code clones detection was useful mostly to:

1) Assess and monitor an interesting quality aspect of a product
This requires that we constantly monitor code clones. If some code already exists, we can create a baseline and enforce a rule that things can only get better, not worse. I usually monitor several internal quality attributes at build time, because that's a fairly flexible moment, where most tools allow to insert some custom steps.

2) Identify candidates for refactoring, mostly in large, pre-existing projects.
This requires, of course, a certain willingness to act on your knowledge, that is, to actually go ahead and refactor duplicated code.

Sometimes, when the codebase is large, resources are scarce, or the company interest in software quality is mostly a marketing statement disconnected from reality, a commitment to refactor the code is never taken, or never taken seriously, which is about the same.

Here comes the third use of code clones. It is quite obvious, and I should have considered it earlier, but for some reason I didn't. I guess I was somehow blinded by the idea that if you care about quality, you must get in there and refactor the damn code. Strong beliefs are always detrimental to creativity :-).

Now: clones are bad because (in most cases) you have to keep them in synch during maintenance. If you don't, something bad is gonna happen (and yes, if you do, you waste a lot of time anyway, so you could as well refactor; but this is that strong belief rearing its head again :-).
So, if you don't want to use a code clones list to start a refactoring campaign, what else can you do? Use it to make sure you didn't forget to update a clone!

Unfortunately, with the tools I know, a large part of this process can't be easily automated. You would have to run a clone detection tool and keep the log somewhere. Then, whenever you change some portion of code, you'll have to check if that portion is cloned elsewhere (from the log). You then port your change in the other clones (and test everything). The clones list must be periodically updated, also to account for changes coming from different programmers.

Better tools can be easily conceived. Ideally, this could be integrated in your IDE: as I suggested in Listen to Your Tools and Materials, editors could provide unobtrusive backtalk, highlighting the fact that you're changing a portion of code that has been cloned elsewhere. From there, you could jump into the other files, or ask the editor to apply the same change automatically. In the end, that would make clones more tolerable; while this is arguably bad, it's still much better than leave them out of synch.

From that perspective, I would say that another interesting place in our toolchain where we would benefit from an open, customizable process is the version control system. Ideally, we may want to verify and enforce rules right at check-in time, without the need to delay checks until build time. Open source tools are an obvious opportunity to create a better breed of version control systems, which so far (leaving a few religious issues aside) have been more or less leveled in term of available features.

Note: I've been writing this post on a EEE PC (the Linux version), and I kinda like it. Although I'm not really into tech toys, and although the EEE looks and feels :-) like a toy, it's just great to carry around while traveling. The tiny keyboard is a little awkward to use, but I'll get used to it...

Elsewhere

2008-06-11T08:53:00.001+02:00

Once again, I can't find much time for blogging: nothing serious, just too much work (and a little fun too :-).

The bright side is that I'll soon finish a paper on function points, and that the paper will come with a free tool. I'll probably submit the paper to some journal, but the draft version will be available online from day 1. And yeah, I know, function points sound like a dull, overly explored subject, but remember the shirt folding idea: there is always something new to say!

On the concept of Form (3): the Force Field

2008-05-27T19:25:00.007+02:00

Warning: :-) this post is going to be somehow conceptual. I'll soon move to some real-world, software-based example, but I really need to introduce some concepts first.

The notion of force field might be unfamiliar to some, so I'll borrow a great example from Alexander himself. Consider your first "requirement" (for a system yet to be built) as a permanent magnet of some size and shape. If you place a flat glass over that magnet, and drop some iron filings on it, the iron will naturally dispose along the magnetic field lines. That gives us an image of [a section of] the force field. Now add another magnet: the shape of the field will change, as the magnets are interacting, thereby shaping a more complex force field.
We can change the [shape of the] field in many ways: moving magnets around, changing their shape, their magnetization, or even adding some shields around magnets.

The great thing about the magnetic field is that we can somehow observe its shape. Indeed, if our goal was to create a form that can be put into effortless contact with the field, we'll just have to replicate the same form that the magnetic field is giving to the iron filings. As Alexander says (NoTSoF, page 21), "once we have a diagram of forces [...] this will in essence also describe the form as a complementary diagram of forces".
In the real world, and even more so in the software world, we are never so lucky: the force field is invisible and tends also to be highly unstable.

Usually, the force field of a software project starts with Requirements. Requirements are often categorized in some way, like "functional" and "nonfunctional", or "user requirements" and "system requirements. However, requirements of any kind are just like magnets: they contribute to shape the overall field.

Requirements are just one kind of force, that is, they are not alone in shaping the field. Many technological choices we make, sometimes very (or too) early, are also shaping the force field.
Consider a simple business application. Once you decide that you'll build a web application, you have added quite a few powerful magnets. If you're familiar with JSP and EJB, you are naturally tempted to choose those technologies early on. That's like adding quite a few powerful magnets again. Or maybe it's like adding a magnetic shield: it really depends on context.

Sometimes, technology makes the field simpler: the right infrastructure should simplify the field, that is, it should act more like a shield than like a magnet. In this sense, infrastructure should be chosen when the dominant forces are known, unlike what happens in many projects, where infrastructure (usually a superstructure in disguise) is chosen too early, thereby making the overall field even more complex.

We also shape the field, so to speak, by choosing what to ignore and what to postpone in any given release. Anything we ignore, like anything we postpone, won't be allowed to shape the field right now.
This is fine, as long as the corresponding magnets will be placed somehow distant from the others (good modularity), possibly with some kind of magnetic shielding in between (stable interfaces). It's also fine if we can ignore it forever. Any attempt to temporarily ignore a strictly interacting force will wreak havoc later on, as our form will no longer match the resulting force field. Refactoring can accommodate minor misfits with the ideal form, but won't help much when the force field changes radically (see also my notes on refactoring here).

Here lies one of the architect's fundamental abilities: the intuitive understanding that something can be beneficially postponed, while something else must be dealt with immediately, because its influence on the force field is so strong that doing otherwise will shift us toward the wrong kind of form.

It is important to understand the role of choice in exploiting instability. Too often, software developers tend to see requirements as "fixed". They don't like to negotiate: it's much easier to fight the compiler than the marketing guys.
A good architect, however, can't miss the opportunity to simplify the field by moving some magnets around. That requires the ability to see the overall picture and the fine details at the same time. Here is Alexander again (page 18): "this ability to deal with several layers of form-context boundaries in concert is an important part of what we often refer to as the designer's sense of organization. The internal coherence of an ensemble depends on a whole net of such adaptations".
That ain't an easy feat. It requires an understanding of the business, the users, and the technology. And even more important, it requires a willingness to act on that knowledge. The power of choice extends to the infrastructure: sometimes, by willingly postponing a technological choice until the force fields takes shape, we can make a better, more "natural" choice.

This can be hard for some developers: they want certainty, and they want it now. In my experience, that goes in pair with the willingness to adopt a sub-optimal, but repetitive and context free solution for a wide class of problems, instead of adopting several optimal, but reasoned and context-dependent solution for smaller classes of problems.

Unfortunately, choosing the "wrong" technology is very much like choosing the wrong shape or orientation for a building. To quote Alexander once again (page 29): "Instead of orienting the house carefully for sun and wind, the builder conceives its organization without concern for orientation, and light, heat, and ventilation are taken care of by fans, lamps, and other kinds of peripheral devices. Bedrooms are not separated from living rooms in plan, but are placed next to one another and the walls between them stuffed with acoustic insulation".
I think we can easily see a parallel with software here: a misfit technology is chosen early on. As a consequence, you find yourself adding more and more technology (fans, lamps, insulation) to satisfy the end-user needs. "Modern" web applications seem to have taken this path: faced with a difficult field, they're layering one technology on top the other, desperately trying to overcome the problems of the previous layer.

Next time, in no particular order: agility, unstable requirements, early coding, TDD, "seeing" the field, internal and external representations, is UML any useful, order within chaos (dominant forces), constructive force field and systematic techniques, and whatever else will come to my mind :-).

Natural language

2008-05-13T15:52:00.004+02:00

Some (most :-) of my clients are challenging. Sometimes the challenge comes from the difficult technical problems they face. That's the best kind of challenge.
Sometimes the challenge comes from people: that's the worst kind of challenge, and one that right now is better left alone.
Sometimes the challenge comes from the organization, which means it also comes from people, but with a different twist. Challenges coming from the organization are always tough, but overcoming those challenges can really make a difference.

One of my challenging clients is a rather large company in the financial domain. They are definitely old-school, and although upper management can perfectly see how software is permeating and enabling their business, middle management tend to see software as a liability. In their eternal search for lower costs, they moved most of the development offshore, keeping only an handful of designers and all the analysts in-house. Most often, design is done offshore as well, for lack of available designers on this side of the world.

Analysts have a tough job there. On one side, they have to face the rest of the company, which is not software-friendly. On the other side, they have to communicate clear requirements to the offshore team, especially to the designers, who tend to be very technology-oriented.
To make things more complicated, the analysts often find themselves working on unfamiliar sub-domains, with precise regulations but also with large gray areas that must be somehow understood and communicated.
Icing on the cake: some of those financial instruments do not even exist in the local culture of the offshore team, making communication as difficult as ever.

Given this overall picture, I've often recommended analysts to spend some time creating a good domain model (usually, a UML class diagram, occasionally complemented by some activity diagrams).
The model, with unambiguous associations, dependencies, multiplicities, and so on, will force them to ask the right questions, and will make it easier for the offshore designer to acquaint himself with the problem. Over time, this suggestion has been quite helpful.
However, as I said, the organization is challenging. Some of the analysts complained that their boss is not satisfied by a few diagrams. He wants a lengthy, wordy explanation, so he can read it over and see if they got it right (well, that's his theory anyway). The poor analyst can't possibly do everything in the allotted time.

Now, I always keep an eye on software engineering research. I've seen countless attempts to create UML diagrams from natural language specifications. The results are usually unimpressive.
In this case, however, I would need exactly the opposite: a tool to generate a precise, yet verbose domain description out of a formal domain model. The problem is much easier to solve, especially because analysts can help the tool, by using the appropriate wording.

Guess what, the problem must be considered unworthy, because there is a dearth of works in that area. In practice, the only relevant paper I've been able to find is Generating Natural Language specifications from UML class diagrams by Farid Meziane, Nikos Athanasakis and Sophia Ananiadou. There is also Nikos' thesis online, with a few more details.
The downside is that (as usual) the tool they describe does not seem to be generally available. I've yet to contact the authors: I just hope it doesn't turn out to be one of those Re$earch Tool$ that never get to be used.

From the paper above, I've also learnt about ModelExplainer , a similar tool from a commercial company. Again, the tool doesn't seem to be generally available, but I'll get in touch with people there and see.

Overall, the problem doesn't seem so hard, especially if we accept the idea that the analyst will help the tool, choosing appropriate wording. An XMI-to-NL (Natural Language) would make for a perfect open source project. Any takers? :-)

Can AOP inform OOP (toward SOA, too? :-) [part 2]

2008-04-25T17:19:00.004+02:00

Aspect-oriented programming is still largely code-centric. This is not surprising, as OOP went through the same process: early emphasis was on coding, and it took quite a few years before OOD was ready for prime time. The truth about OOA is that it never really got its share (guess use cases just killed it).

This is not to say that nobody is thinking about the so-called early aspects. A notable work is the Theme Approach (there is also a good book about Theme). Please stay away from the depressing idea that use cases are aspects; as I said a long time ago, it's just too lame.

My personal view on early aspects is quite simple: right now, I mostly look for cross-cutting business rules as candidate aspects. I guess it's quite obvious that the whole "friend gift" concept is a business rule cutting through User and Subscription, and therefore a candidate aspect. Although I'm not saying that all early aspects are cross-cutting business rules (or vice-versa), so far this simple guideline has served me well in a number of cases.

It is interesting to see how early aspects tend to be cross-cutting (that is, they hook into more than one class) but not pervasive. An example of pervasive concern is the ubiquitous logging. Early aspects tend to cut through a few selected classes, and tend to be non-reusable (while a logging aspect can be made highly reusable).
This seems at odd with the idea that "AOP is not for singleton", but I've already expressed my doubt on the validity of this suggestion a long time ago. It seems to me that AOP is still in its infancy when it comes to good principles.

Which brings me to obliviousness. Obliviousness is an interesting concept, but just as it happened with inheritance in the early OOP days, people tend to get carried over.
Remember when white-box inheritance was applied without understanding (for instance) the fragile base class problem?
People may view inheritance as a way to "patch" a base class and change its behaviour in unexpected ways. But truth is, a base class must be designed to be extended, and extension can take place only through well-defined extensions hot-points. It is not a rare occurrence to refactor a base class to make extension safer.

Aspects are not really different. People may view aspects as a way to "patch" existing code and change its behaviour in unexpected ways. But truth is, when you move outside the safe realm of spectators (see my post above for more), your code needs to be designed for interception.
Consider, for instance, the initial idea of patching the User class through aspects, adding a data member, and adding a corresponding data into the database. Can your persistence logic be patched through an aspect? Well, it depends!
Existing AOP languages can't advise any given line: there is a fixed grammar for pointcuts, like method call, data member access, and so on. So if your persistence code was (trivially)

class User
  {
  void Save()
    {
    // open a transaction
    // do your SQL stuff
    // close the transaction
    }
  }

there would be no way to participate to the transaction from an aspect. You would have to refactor your code, e.g. by moving the SQL part in a separate method, taking the transaction as a parameter. Can we still call this obliviousness? That's highly debatable! I may not know the details of the advice, but I damn sure know I'm being advised, as I refactored my code to be pointcut-friendly.

Is this really different from exposing a CreateSubscription event? Yeah, well, it's more code to write. But in many data-oriented applications, a well-administered dose of events in the CRUD methods can take you a long way toward a more flexible architecture.

A closing remark on the SOA part. SOA is still much of a buzzword, and many good [design] ideas are still in the decontextualized stage, where people are expected to blindly follow some rule without understanding the impact of what they're doing.

In my view, a crucial step toward SOA is modularity. Modularity has to take place at all levels, even (this will sound like an heresy to some) at the database level. Ideally (this is not a constraint to be forced, but a force to be considered) every service will own its own tables. No more huge SQL statements traversing every tidbit in the database.

Therefore, if you consider the "friend gift" as a separate service, it is only natural to avoid tangling the User class, the Subscription class, and the User table with information that just doesn't belong there. In a nutshell, separating a cross-cutting business rule into an aspect-like class will bring you to a more modular architecture, and modularity is one of the keys to true SOA.

Can AOP inform OOP (toward SOA, too? :-) [part 1]

2008-04-24T16:05:00.007+02:00

Note: I began tinkering with this idea several months ago, inspired by some design work I was doing on a real-world project. Initially, I thought I could pimp up this stuff into a full-fledged article. Months came by and I didn't, so (in the spirit of my old concept of Blogging as Destructuring) I thought I could just as well say something here instead.

Consider a web site offering some kind of service to users. Users have to register (yeah :-), and they can then buy a subscription to several services. A simple class diagram can model this easily:

In most cases, the underlying database schema wouldn't be much different.

In a real case, there might be different kind of services, each requiring a derived class, but we can ignore this issue right now. Also, there might be several kind of subscription (time-based, pay-per-use, and so on), but again, let's ignore that issue right now, and just concentrate on time-base subscriptions.

A subscription can be renewed at any time. In the object model, this would translate into a Renew method in class Subscription. Renew could take a parameter, like the extension of the renewal. Most likely, it would add a new record to the Subscription table (to keep track of the whole subscription history for that user), and possibly create a new Subscription object. So far, so good.

Now the marketing guys come up with a nice idea: whenever you register, you can provide the email address of a friend who has already registered. If you do, everytime you renew a subscription your friend will get some kind of gift, like a free extension or whatever. This may generate a few more leads, meaning a little more business.

From a purely OO mindset, this may lead us to perform a little maintenance on the existing model. We can add a relationship from User to User to model the "friend" relationship, or we could just add a field like friendEmail (possibly left empty). At the database level, adding a field is probably easier.
We can also modify the Renew method to check for the presence of a friend in the subscribing user, and if so, invoke some Gift logic. I won't draw a modified diagram for this scenario: I guess it's just too obvious.

Now, this approach obviously works. However, you may recognize that the whole "friend gift" concept is a cross-cutting concern: it cuts through User (requiring new data) and through Subscription (requiring new logic). More on detecting cross-cutting concerns during analysis and design (and on the difference between cross-cutting and pervasive concerns) next time.

In the AOP world, we could approach the problem differently. We could define a FriendGift aspect. The aspect may add a new data member to the User class (the friendEmail), and intercept the persistence logic to save / load that data from the database. The aspect may also intercept Renew and perform the Gift logic if required.
Actually the aspect doesn't have to modify User (class and table); indeed, it would be better not to, as the persistence logic might not be easy to intercept (more on this next time). The aspect could just use a different database table to store the friend email.

Using an UML-like notation, we could model this approach as:

Interestingly, the database is probably different from the previous scenario: Friend would map to its own table.

What if we are not using an AOP-enabled language? For instance, we might be using plain old C#. Can we still borrow some ideas from the above? I believe so. In the same sense as OO thinking can inform traditional structured programming, AOP thinking can inform traditional object oriented programming.

If we don't have pointcuts, join points and aspects, we may still have events (in C#/.NET, or callback in other languages, and so on). Sure, we have to forego obliviousness (which might not be so bad: more on this next time). But we can come up with a more modular and decoupled design by reasoning along the AOP lines:

No rocket science here :-). Just a different form, same function. Different database, a more general Subscription class, unaffected by a business rule which may change at any time. Also the User class and table are left unaffected.

More on this, and a few comments on the SOA part, in just a few days (I hope :-).

Old Problems, new Solutions

2008-04-15T16:44:00.004+02:00

In a comment to my previous post, I mentioned an half-baked idea about a (truly) modern architecture for GUI applications.

In fact, I've been experimenting with different approaches to GUI construction for "classic" Windows applications, Web applications, Smart Clients calling web services, and even Web applications calling web services for quite a while now.

Some were dead ends. Others have been a successful intermediate step toward something better. I usually pass on the good ideas to my clients, and meanwhile, I keep trying different approaches on small scale, low-risk projects. Because the "traditional" way of building GUI applications sucks big time. So does the traditional thinking about "good" GUI constructions, that is, that you need a tightly integrated language / ide / library / operating system to do anything decent.

I also don't like the idea that I should buy-in a monolithic framework for GUI construction. Different problems are better solved in different ways: we should be free to mix and match different approaches in different forms (within the same application), or even within the single form, or component.

Over time, a few people told me not to bother: Microsoft, Sun, Apple (not to mention a thousand smart guys working for smaller companies) have been working at that for years. It's an old problem, and it's unlikely that anybody can come up with a radically different solution.

It's very poor thinking. There is no limit to human creativity. Take a look at How To Fold A T-Shirt In 2 Seconds. The problem is much older than GUI construction. Uncountable people have been faced with the same problem and they didn't come up with anything similar so far. Sure, there are pros and cons in this technique (as usual), but it's a radically different (and much faster) approach.

You may want to go through the explained version, as I did. I actually tried that a few times, and it works :-).

Quote of the day past

2008-04-13T14:14:00.003+02:00

	Life is either a daring adventure or nothing (Helen Keller).
	Good luck Ma, you've been strong and brave beyond my imagination. Have fun!

Asymmetry

2008-04-04T10:23:00.010+02:00

I'm working on an interesting project, trying to squeeze all the available information from sampled data and make that information useful for non-technical users. I can't provide details, but in the end it boils down to reading a log file from a device (amounting to about 1 hour of sampled data from multiple channels), do the usual statistics, noise filtering, whatever :-), calculate some pretty useful stuff, and create a report that makes all that accessible to a business expert.

The log file is (guess what :-) in XML format, meaning it's really huge. However, thanks to modern technology, we just generated a bunch of classes from the XSD and let .NET do the rest. Parsing is actually pretty fast, and took basically no time to write.
In the end, we just get a huge collection of SamplingPoint objects. Each Sampling point is basically a structure-like class, synthesized from the XSD:

class SamplingPoint
{
public DateTime Timestamp { // get, set }
public double V1 { // get, set }
// ...
public double Vn { // get, set }
}

each value (V1...Vn) is coming from a different channel and may have a different unit of measurement. They're synchronously sampled, so it made sense for whoever developed the data acquisition module to group them together and dump them together in a single SamplingPoint tag.

We extract many interesting facts from those data, but for each Vi (i=1...N) we also show some "traditional" statistics, like average, standard deviation and so on.
Reasoning about average and standard deviation is not for everyone: I usually consider an histogram of the distribution much easier to understand (and to compare with other histograms):

Here we see the distribution of V1 over time: for instance, V1 had a value between 8 and 9 for about 6% of the time. Histograms are easy to read, and users quickly asked to see histograms for each V1..Vn over time. Actually, since one of the Vj is monotonically increasing with time, they also asked to see the histogram of the remaining Vi against Vj too. So far, so good.

Now, sometimes I hate writing code :-). It usually happens when my language doesn't allow me to write beautiful code. Writing a function to calculate the histogram of (e.g.) V1 against time is trivial: you end up with a short piece of code taking an array of SamplingPoints and using the V1 and Timestamp properties to calculate the histogram. No big deal.

However, that function is not reusable, exactly because it's using V1 and Timestamp. You can deal with this in at least 3 unpleasant :-) ways:

1) you don't care: you just copy/paste the whole thing over and over. If N = 10, you get 19 almost-identical functions (10 for time, 9 for Vj).

2) you restructure your data before processing. Grouping all the sampled data at a given time in a single SamplingPoint structure makes a lot of sense from a producer point of view, but it's not very handy from a consumer point of view. Having a structure of arrays (of double) instead of an array of structures would make everything so much simpler.

3) you write an "accessor" interface and N "accessors" classes, one for each Vi. You write your algorithms using accessors. Passing the right accessors (e.g. for time and V1) will get you the right histogram.

All these options have some pros and cons. In the end, I guess most people would go with (2), because that brings us into the familiar realm of array-based algorithms.

However, stated like this, it seems more like a "data impedance" problem between two subsystems than a language problem. Why did I say it's the language fault? Because the language is forcing me to access data members with compile-time names, and does not (immediately) allow me to access data members using run-time names.

Don't get me wrong: I like static typing, and I like compiled languages. I know from experience that I tend to make little stupid mistakes, like typing the wrong variable name and stuff like that. Static typing and compiled languages catch most of those stupid mistakes, and that makes my life easier.

Still, the fact that I like something doesn't mean I want to use that thing all the time. I want to have options. Especially when those options would be so simple to provide.

In a heavily reflective environment like .NET, every class can be easily considered an associative container, from the property/data member names to property/data member values. So I shold be able to write (if I wanted):

SamplingPoint sp = ... ;
double d1 = sp[ "V1" ] ;

which should be equivalent to

double d1 = sp.V1 ;

Of course, that would make my histogram code instantly reusable: I'll just pass the run-time names of the two axes. You can consider this equivalent to built-in accessors.

Now, I could implement something like that on my own, using reflection. It's not really difficult: you just have to gracefully handle collections, nested objects, and so on. Unfortunately, C# (.NET) do not allow a nice implementation of the concept, mostly for a bunch or constraints they added to conversion operators: no generic conversion operators (unlike C++), no conversion to/from Object, and so on. In the end you may need a few more casts that you'd like to, but it can be done.

I'll also have to evaluate the performance implications for this kind of application, but I know it would make my life much easier in other applications (like binding smart widgets to a variety of classes, removing the need for braindead "controller" classes). It's just a pity that we don't have this as built-in language feature: it would be much easier to get this right (and efficient) at the language level, not at the library level (at least, given C# as it is now).

Which brings me to the concept of symmetry. A few months ago I stumbled upon a paper by Jim Coplien and Zhao Liping (Understanding Symmetry in Object-Oriented Languages, published in Journal of Object Technology, an interesting, free publications that's filling the void left by the demise of JOOP). Given my interest on the concept of form in software, the paper caught my attention, but I postponed further thinking till I could read more on the subject (there are several papers on symmetry in Cope's bibliography, but I need a little more time than I have). A week ago or so, I've also found (and read) another paper from Zhao in Communications of ACM, March 2008: Patterns, Symmetry, and Symmetry breaking.

Some of their concepts sound odd to me. The concept of symmetry is just fine, and I think it may help to unravel some issues in language design.
However, right now the idea that patterns are a way to break symmetry doesn't feel so good. I would say exactly the opposite, but I really have to read their more mathematically-inclined papers before I say more, because words can be misleading, while theorems usually are not :-).

Still, the inability to have built-in, natural access to fields and properties through run-time names struck me as a lack of symmetry in the language. In this sense, the Accessor would simply be a way to deal with that lack of symmetry. Therefore it seems to me that patterns are a way to bring back symmetry, not to break symmetry. In fact, I can think of many cases where patterns "expose" some semantic symmetry that was not accessible because of (merely) syntactic asymmetry.

More on this as I dig deeper :-).

Zen and the Art of Fixing

2008-03-27T23:18:00.000+01:00

A little known fact about me is that I like fixing things. Mostly TV sets, but over the years I've tried to repair (with various degrees of success :-) any electrical, electronic or mechanical device that needed some fixing. Indeed, when I was about 14 I made a small business out of rescuing old vacuum-tube radios, fixing and then selling them to collectors.
Even today, I occasionally talk about diagnosing and repairing a defective TV when teaching about modular design, modular reasoning, design for testability, and so on.

In fact, fixing a device has a lot in common with fixing a buggy program. Some devices (like old washing machines) are relatively simple. You just need to know some basic concepts, and apply some common sense. Sure, it may take some creativity to fix even the simplest device (especially if you can't find the right replacement parts), but overall, the problem is often self-evident, and you "only" need to resort to your ability, often just manual ability.

Electronic devices, however, can fail in complex, even puzzling ways. You need a better understanding of what's going on under the hood. You may need special tools (you can't get much far repairing a TV if you ain't got an oscilloscope) but sometimes it's just plain old intuition (or sheer luck :-). You need to know some tricks of the trade (like using a light bulb to discriminate problems in the power supply vs. horizontal deflection), but overall, what you need more is rational system thinking.

The same is true when reasoning about complex software failures. We often have a large numbers of parts (hardware, firmware, drivers, OS, libraries, our own code) which can fail for a large number of reasons. Your best allied is rational system thinking. Your worst enemy is the all-so-common uncircumstantiated certainty that the fault must lie exactly somewhere (usually outside our own code :-). Overall, I would say my experience fixing stuff has made me a better troubleshooter, helping me to find obscure bugs in systems I knew very little about.

Still, I don't like to fix computers. I guess I'm already spending too much time around computers, and besides, today the integration scale is so high that you can hardly fix a broken motherboard. However, there is always an exception, and this one is interesting enough to be worth telling.

A couple of years ago (yeah :-) I rescued a notebook before it was thrown away. After a fall, it wouldn't even turn on. The CD-ROM was visibly damaged, but the screen was intact. I took it home and left it alone for a few months, till I had some time to kill.

I'm usually lucky, and in fact, I discovered that by pushing the power plug a little heavier than usual, the notebook would indeed power on. That's usually just a broken joint; it took forever :-) to disassemble the notebook and expose the PCB, but after that, it was quite easy to fix. Now the computer would turn on, start booting XP, and die a few seconds later. I suspected the HD was damaged too, and replaced it with a spare one (with W2K installed). Similar story: it would start booting the OS, then dump a blue screen reporting that “the boot device was not accessible”.

That's usually due to a broken IDE controller, or a faulty RAM chip. I tried replacing the RAM, but still got the same problem. However, the IDE controller was somehow working, as the computer would indeed start booting. Weird :-).

I didn't give up and got another HD, with good old MS-DOS installed. The notebook booted like a charm, and all the applications seemed to work. Weird again: the IDE controller seemed fine. Again, it could be a faulty RAM chip, since MS-DOS only uses the first 640KB.

I connected a USB CD-ROM and tried an old version of Knoppix on CD: knoppix uses all the available RAM, so that was like a definitive RAM test. It worked fine, but as soon as I tried using the HD, it would simply lock up. So, RAM was fine, the IDE controller was fine under MS-DOS but failed under other operating systems.
My diagnosis was that DMA was at fault. I tried to disable DMA at the BIOS level with no luck. I also disabled DMA in that W2K HD (using another PC, of course), but it would still lock, just like Knoppix.

At that point, I contemplated using an external HD (via USB), and perhaps installing a tweaked version of XP which can boot from a USB device (details on the BartPE page). I could even rewire one of the USB ports and install the whole thing internally, since the missing CD-ROM left a lot of space. But it looked like a damn lot of work :-), so I did nothing and left the notebook around for more than one year.

A few days ago, having a little time to kill again, I tried a different experiment. I got a 1GB USB stick, downloaded the latest version of Knoppix, and put it on the stick (instead of a CD-ROM). I followed this tutorial to speed things up.
It worked fine, and booting from the USB stick was very quick. However, at that point I noticed (from the boot log) that this newer version of Knoppix ran with DMA disabled (I guess the old one I tried before didn't). Time to test my DMA controller theory!

I put an HD inside, booted Knoppix from the USB stick, and guess what, no hang-up! I could access the HD with not problem whatsoever. At that point, it was hard to resist: I had to try installing knoppix on the HD itself. Again, I followed a tutorial to speed things up: this one is about a previous version, but still valid. The proof of the pudding is in the eating: I removed the USB stick, tried booting and Lo!, it worked. The world is still somehow deterministic :-).
By the way: Knoppix takes forever to boot from the HD. Overall, it would be better to keep the USB stick, maybe rewiring a USB port to keep everything inside.

Now, in a perfect world, I would have found a way to really "fix" the notebook, that is, to repair the DMA controller. I suspect it's just another loose joint from the fall. Most likely, some pin of some surface-mounted chip suffered some mechanical stress. This world, however, ain’t perfect, so I did nothing else, leaving it as a working Knoppix notebook. Fun is over, time to get back to work :-)

(Simple) Metrics

2008-03-19T14:42:00.000+01:00

I've been using metrics for a long time (certainly more than 10 years now). I've been using metrics to control project quality (including my own stuff, of course), to define acceptance criteria for outsourced code, to understand the way people work, to "smell" large projects before attempting a refactoring activity, to help making an informed refactor / rewrite decision, to pinpoint functions or classes in need of a careful review, to estimate residual bugs, an so on.

Of course, I use different metrics for different purposes. I also combine metrics to get the right picture. In fact, you can now find several tools to calculate (e.g.) code metrics. You can also find many papers discussing (often with contradictory results) the correlation between any given metric and (e.g.) bug density. In most cases, those papers are misguided, as they look for correlation between a single metric and the target (like bug density). Reality is not that simple; it can be simplified, but not to that point.

Consider good old cyclomatic complexity. You can use it as-is, and it can be useful to calculate the minimum reasonable number of test cases you need for a single function. It's also known that functions with higher cyclomatic complexity tend to have more bugs. But it's also well known that (on average) there is a strong, positive correlation between cyclomatic complexity (CC) and lines of code (LOC). That's really natural: long functions tend to have a complex control flow. Many people have therefore discounted CC, as you can just look at the highly correlated (and easier to calculate) LOC. Simple reasoning, except it's wrong :-).

The problem with that, again, is trying to use just one number to understand something that's too complex to be represented by a single number. A better way is to get both CC and LOC for any function (or method) and then use quadrants.

Here is a real-world example, albeit from a very small program: a smart client invoking a few web services and dealing with some large XML files on the client side. It has been written in C# using Visual Studio, therefore some methods are generated by the IDE. Also, the XML parser is generated from the corresponding XSD. Since I'm concerned with code which is under the programmer's control, I've excluded all the generated files, resulting in about 20 classes. For each method, I gathered the LOC and CC count (more on "how" later). I used Excel to get the following picture:

As you can see, every method is just a dot in the chart, and the chart has been split in 4 quadrants. I'll discuss the thresholds later, as it's more important to understand the meaning of each quadrant first.

The lower-left quadrant is home for low-LOC, low-CC methods. These are the best methods around: short and simple. Most code ought to be there (as it is in this case).

Moving clockwise, the next you get (top-left) is for high LOC, low CC methods. Although most coding standards tend to somehow restrict the maximum length of any given method, it's pretty obvious that a long method with a small CC is not that bad. It's "linear" code, likely doing some initialization / configuration. No big deal.

The next quadrant (top-right) is for high LOC, high CC methods. Although this might seem the worst quadrant, it is not. High LOC means an opportunity for simple refactoring (extract method, create class, stuff like that). The code would benefit from changes, but those changes may require relatively little effort (especially if you can use refactoring tools).

The lower-right quadrant is the worst: short functions with high CC (there are none in this case). These are the puzzling functions which can pack a lot of alternative paths into just a few lines. In most cases, it's better to leave them alone (if working) or rewrite them from scratch (if broken). When outsourcing, I usually ask that no code falls in this quadrant.

For the project at hand, 3 classes were in quadrant 3, so candidate for refactoring. I took a look, and guess what, it was pretty obvious that those methods where dealing with business concerns inside the GUI. There were clearly 3 domain classes crying to be born (1 shared by the three methods, 1 shared by 2, one used by the remaining). Doing so brought to better code, with little effort. This is a rather ordinary experience: quadrants pinpoint problematic code, then it's up to the programmer/designer to find the best way to fix it (or decide to leave it as it is).

A few words on the thresholds: 10 is a rather generous, but somewhat commonly accepted threshold for CC. The threshold for LOC depends heavily on the overall project quality. I've been accepting a threshold of 100 in quality-challenged projects. As the quality improves (through refactoring / rewriting) we usually lower the threshold. Being a new development, I adopted 20 LOC as a rather reasonable threshold.

As I said, I use several different metrics. Some can be used in isolation (like code clones), but in most cases I combine them (for instance, code clones vs. code stability gives a better picture of the problem). Coupling and cohesion should also be considered as pairs, never as single numbers, and so on.
Quadrants are not necessarily the only tool: sometimes I also look at the distribution function of a single metric. This is way superior to what too many people tend to do (like looking at the "average CC", which is meaningless). As usual, a tool is useless if we can't use it effectively.

Speaking of tools, the project above was in C#, so I used Source Monitor, a good free tool working directly on C# sources. Many .NET tools work on the MSIL instead, and while that may seem like a good idea, in practice it doesn't help much when you want a meaningful LOC count :-).

Source Monitor can export in CSV and XML. Unfortunately, the CSV didn't contain the detailed data I wanted, so I had to use the XML. I wrote a short XSLT file to extract the data I needed in CSV format (I suggest you use the "save as" feature, as unwanted spacing / carriage returns added by browsers may cripple the result). Use it freely: I didn't put a license statement inside, but all [my] source code in this blog can be considered under the BSD license unless otherwise stated.

Problem frames and the DNC

2008-03-07T18:52:00.000+01:00

If you've read any of my posts before, you probably know I'm not a fan of pre-packaged, one-size-fits-all ideas. Methods, technology, languages, models, even specific values (design values; business values; etc.) must always be considered in context.

With this background, it's hardly surprising that I've always found the notion of a Domain Neutral Component quite uncomfortable. It really sounded like an attempt to shoehorn the world into a predefined model, while we should carefully look for the relevant portion of the world we want to represent into our model.

Still, in many cases (especially for a junior analyst) starting with the DNC might be better than starting with a blank page. How could this be? Does it work all the time? If not, when? Honestly, in the past years I haven't spent too much time trying to answer those questions. The DNC was part of my bag of tricks, but I didn't use it often.

Recently, however, I was thinking (once more :-) about colors and UML, and while looking into some of Peter Coad's works for a specific reference, I stumbled on the DNC again. So I thought, maybe I've learnt something in the past few years that could shed some light on the inner quality of the DNC, and its suitability in any (?) given context.

The DNC can be considered as an overengineered "standard" model representing something (an event / moment / interval) happening somewhere (a place) involving one party or more (originally an actor), usually exchanging or dealing with some good (a thing). The party plays a role, hence the later shift from actor to party + role. Indeed, you can start with a very simple model, and "derive" the DNC by following a very reasonable line of reasoning: see From Associations To Domain Neutral Component for the full story.

Of course, in many cases, the DNC might be overengineered. But you can always simplify the unnecessary parts. The real question, however, is when the DNC can give you a head start, and when it won't (context, context, context :-).

That's where Problem Frames Patterns can shed some light. I recommend that you keep the PFP paper at hand while reading what follows.

Consider, for instance, the Commanded Behavior problem frame. Shortly, the problem is stated as:
There is some part of the world whose behavior is to be controlled in accordance with commands issued by
an operator. The problem is to build a machine that will accept the operator's commands and impose the
control accordingly.
and the frame concerns are:
1. When the Operator issues a Command
2. AND the Machine rejects invalid Commands
3. AND the Machine either ignores it if unviable, OR issues Control Events
4. AND the Control Events change the Controlled Domain
5. ENSURE the changed state meets the Commanded Behavior in every case.
That's not at odds with the DNC. Control Events map nicely to moment-interval; moreover, the text above suggests that multiple events might be issued for a single command (MomentInterval-MomentIntervalDetails). The Control Events change the Controlled Domain. Therefore, they must describe external entities (each probably having a Role) that must somehow influence internal entities (Party, having a Role, or Places, having a Role).
Using the Email Client example, "Email Retrieval" is an event, composed of individual retrieval events (one for each email message). Each Message is a Thing, although with a dubious Role. Retrieval needs (at least) an Account, which is a Party playing a specific Role (Receiver). Retrieval takes place on a specific Server, playing a Role (POP3 or IMAP server). Not so bad.

What if we look into another problem frame? Let's try Transformation:
There are some given inputs which must be transformed to give certain required outputs. The output data
must be in a particular format, and it must be derived from the input data according to certain rules. The
problem is to build a machine that will produce the required outputs from the inputs.
Concerns:
1. BY traversing the input in sequence, and simultaneously traversing the outputs in sequence
2. AND finding values in the input domain, and creating values in the output domain
3. AND that the input values produce the correct output values
4. ENSURES the I/O relation is satisfied.
Hmmm. Doesn't map so nicely, but the text is really too abstract. Let's try the actual problem: an HTML email to be converted in plain text to be shown on a limited device (I added some context to the equally abstract problem described in the original paper). Well, there is hardly a dominant MomentInterval here. Hardly a party, place, thing triad gravitating around the central MI. Hardly any value in adopting the DNC as a starting point. What can be helpful here? Concepts from grammars, taxonomies, language theory. We're basically modeling a translator, and language theory will give you the head start.

So here it is. The DNC is an interesting concept because it maps nicely to some recurring problem frames (if you got time to spare, you may want to investigate which frames are a good fit for the DNC). Some problem frames, however, just don't match with the DNC. It's not just about individual problems: it's about a whole class of problems, all those within the mismatched problem frames.

For me, this is actually good news. Once we know which problem frames map nicely to the DNC, I would say the DNC itself becomes a more powerful tool, one that can be applied wisely and not blindly.

Winding down: since it all began with colors, it's interesting to see how people used the DNC to reason about more general issues. For instance, in Whole Part Relationships in Color Models, David J. Anderson starts with the DNC and ends up recommending that we avoid some whole-part colors, like a green whole with a yellow part. There is probably more to investigate along those lines. Next time I get back to my idea of coloring associations and dependencies, I'll give it a deeper thought.

Domain Neutral Component

2008-03-03T22:23:00.004+01:00

I mentioned the Domain Neutral Component when answering a comment to my post on the Cognitive Dimensions of Notations. That reminded me I've never been a fan of the DNC, exactly because of its context-independent ambitions. Recently, however, I've reconsidered the DNC in terms of Problem Frames.
It's kinda late, so more on this soon, I hope...

WebSite is (almost) back

2008-02-27T21:31:00.002+01:00

I changed hosting company overnight :-) and republished everything this morning. Email should be working fine too.

All the dynamic pages (forms, search, and so on) are not working yet. Guess the new company has different rules for PHP scripts.
It will take a few days before I can look into that, but I guess I'll probably scrap most of those pages and just adopt a different solution (like using google to search the website, and so on).

Gee, the website is more than 10 years old, and it shows :-).

WebSite Issues

2008-02-26T20:27:00.002+01:00

For a number of reasons, I'm moving my website to another hosting service. I'll try to minimize the inconvenience, but is very likely that there will be some interruption in email delivery and/or in http access.
If you want to send me email in the next few days please send it to my alternate address ( ). If you have sent me any message in the last few days, please send it again (thanks).

Bookmarking

2008-02-21T14:42:00.005+01:00

A few readers have asked me to make my blog more bookmarking-friendly. I started a few weeks ago by making the permalink more visible (moving away from the almost-invisible default proposed by Blogger, that is, a "#"), but I'm now beginning to add "buttons" to submit individual posts to bookmarking sites.
Effective now, I've added a stumble upon button. I plan to add Digg and del.icio.us soon. Technorati is not really about bookmarking pages, but better support is under way too.
Suggestions welcome!

Cognitive Dimensions of Notations

2008-02-17T17:16:00.004+01:00

I constantly hear that the blogsphere is highly self-referential, so what's better than starting a post by quoting yourself? Here is an excerpt from Listen to Your Tools and Materials:
Our material is knowledge, or information. We acquire, understand, filter, and structure information; and we encode it in a variety of formats: text, diagrams, code, and so on.[...]the diagram isn't a material, just a representation. We use a tool to represent the material, which is intangible knowledge. [...] What's peculiar with software is that, in many cases, the tools and the materials have the same nature.

Note that I'm using "tool" in a very broad sense there. UML is not our material, is a tool we use to encode our material (and yeah, we may also use a software tool to draw the UML). Source code is not our material, is a tool we use to encode our material.
The fact that we often confuse tools and materials, and that we can get along with that just fine, empirically proves that they are somehow similar in nature.

Now, If our tools and materials have similar nature, then perhaps by understanding better the properties of our tools we can also understand better the properties of our materials, and ultimately of our creations. Assuming, of course, that we can somehow define and classify some interesting properties of our tools (that is, notations).

Turns out that people have been doing so for quite a while now. There is an interesting body of knowledge about the so-called Cognitive Dimensions of notations. I'll quote some portions from a short paper (Cognitive Dimensions of Notations: Design Tools for Cognitive Technology), but if you're interested, I recommend you dig deeper by reading Cognitive Dimensions of Information Artefacts: a tutorial.

When we use a notation, we are both limited and enabled by its own peculiarities. Our own style further emphasizes some of those attributes. For instance, a class diagram is relatively easy to change: you can quickly reshape your design into a very different one. In the Cognitive Dimensions vocabulary, we would say that its viscosity is low. However, if you have also modeled some scenarios on that class diagram (using for instance a sequence diagram) you have to work much harder to make changes. Sequence diagram have high viscosity.

Now, I really like the choice of viscosity as an attribute. I have to confess: I like it because it's not nerdy. When we're dealing with intangible information, it's easy to find ourselves lacking the necessary words. In software development, there is then a wide tendency to adopt some weird, geeky term.
Perhaps that's why I've immediately appreciated the Cognitive Dimensions framework. They put some considerable effort in creating the vocabulary itself. Here is a quote from the aforementioned paper: We believe that this problem is best addressed by providing a vocabulary for discussing the design problems that might arise – a vocabulary informed by research in cognitive psychology, but oriented toward the understanding of a system developer. The Cognitive Dimensions of Notations are such a vocabulary. Bingo! :-)

If you spend some time familiarizing with the vocabulary, you'll see how useful it can be to settle down some long-standing debates, like code Vs. diagrams or even static typing Vs. dynamic typing. Consider, for instance:
Premature commitment: constraints on the order of doing things.
Hidden dependencies: important links between entities are not visible.
Error-proneness: the notation invites mistakes and the system gives little protection.
and so on. They provide a sound reference against which the pros and cons of different notations can be evaluated.

The real boon (for me) is: some of those concepts can be extended from the tool to the material! Actual code may or may not have Hidden Dependencies. Actual code may have different degrees of Viscosity. Actual code may or may not have Closeness of Mapping. And so on. Of course, the notation itself is providing a bottom line on some properties. But ultimately, the form you give to your material may move quite far from the bottom line.
Indeed, I believe the form itself will be heavily influenced by the process and notation you used when you conceived the form. If you conceive your form using diagrammatic reasoning, it will keep some of the inherent properties of that notation even when it's translated into another notation (like source code). If you start with source code, your form will be more heavily influenced by the properties of source code.

I also like some of the candidate dimensions. For instance, I'm especially fond of Creative Ambiguity. This, I would say, it's one of the properties that code is lacking most. It's also something academics are trying hard to remove :-) from UML. And in a sense, it's what makes some practices like TDD so limited.
Code doesn't afford much creative ambiguity, yet what we call "modeling" should probably be called "incubation". Shaping a great form ain't easy. We need low viscosity, a good degree of creative ambiguity, high provisionality, and so on. Code alone just won't cut it.
Of course, we might not be after great form, just run-of-the-mill form. Then anything sensible is gonna work: see also my recent post on old-fashioned architectures "designed" for mass adoption.

Do Less

2008-02-09T18:11:00.000+01:00

In many software projects lies some dreaded piece of code. Legacy code nobody wants to touch, dealing with a problem nobody wants to look at. In the past few days, I've been involved in a (mostly unpleasant :-) meeting, at the end of which I learnt that some "new" code was actually dealing with a dreaded problem by calling into legacy code. Dealing with that problem inside the "new" code was deemed risky, dirty, hard work.

Now, as I mentioned in my Work Smarter, Not Harder post, I don't believe in hard work. I believe in smart work.

After thinking for a while about the problem, I formulated an alternative solution. It didn't come out of nowhere: I could see the problem frame, and that was a language compiler/interpreter frame, although most of the participants didn't agree when I said that.
They had to agree, however, that my proposed solution was simple, effective, risk-free, and could be actually implemented in a few days of light :-) work. Which I take as an implicit confirmation that the problem frame was right.
I also had to mix some creativity, but not too much. So I guess the dreaded problem can now be solved by doing much less than expected.

This could explain the title of this post, but actually, it doesn't. More than 20 years ago, good ol' Gerald Weinberg wrote:
Question: How do you tell an old consultant from a new consultant?
Answer: The new consultant complains, "I need more business." The old consultant complains, "I need more time."

I guess I'm getting old :-), and I should probably take Seth Godin's advice. Seth is best known for "Purple Cow", a wonderful book that ought to be a mandatory reading for everyone involved in creating a new product. The advice I'm thinking of, though, comes from Do Less, a short free pamphlet he wrote a few years ago. There he writes:
What would happen if you fired half of your clients?

Well, the last few days really got me thinking :-)).

A few free tools

2008-02-03T17:27:00.000+01:00

A few free tools I've found myself using and/or recommending in the last few days:

Refactor!™ for C++
A refactoring plug-in for Visual Studio. Even if you only use Extract Method, you'll like it. Downside: your IDE may get a bit sluggish.

CScout
A different refactoring/code mining tool. If you work on a large project, you'll love the ability to discover "redundant" #include in your header files. Disclaimer: I haven't tried that feature on really sick include files :-). If you do, let me know...

StarUML
(thanks to Fulvio Esposito who told me about it)
A Rational Rose lookalike in Delphi (open source). One order of magnitude faster than the average java behemoth. Limited to UML 1.4, but easy to use, reasonably stable, does a reasonable job at importing Rose files, and the "auto layout" feature is better than average.

VirtualBox
(thanks to Roberto Rossi who told me about it)
An open-source virtual machine for Windows / Linux. Reasonably stable, faster than average. I wish I had more time to learn about the internals and the architecture!

As usual, guys, this is not an endorsement or whatever else. Try that stuff at your own risk :-).

Being 10 Years Behind (part 2)

2008-01-27T18:36:00.000+01:00

Do you remember "Windows DNA"? If you can't, don't blame yourself, because the MSDN doesn't remember either :-).
Indeed, it seems like Microsoft took good care of removing most of the material on Windows DNA from its developer-oriented website.
However, here comes the TechNet website to the rescue (well, at least till they realize it :-). As you see, the much touted "Distributed interNet Applications Architecture" was the usual 3 tier blurb. There is no date on that web page, but the "Windows DNA" stuff is about ten years old.

Sure, Windows DNA was all based on COM+ components, most likely implemented in Visual Basic, maybe glued to a presentation logic written in old-style ASP (VBScript all around). But look at that architecture again. Does it look familiar?

Let's take a look at some recent Microsoft-oriented paper on application architecture. For instance, in Microsoft .NET Pet Shop 3.x: Design Patterns and Architecture of the .NET Pet Shop the Microsoft-flavored "Data Access Layer" is introduced, along with a general architecture (see fig. 3) which looks absolutely identical to the old "Windows DNA" stuff.

Dig deeper (fig. 5, 6, 8, 9) and you'll also realize that the DAL structure is a mirror of the DB structure (that is, basically one class for each table). Looks really like the decade-old, fragile architecture I described in my previous post, except this paper is "just" 5 years old. Particularly dreadful is the "business entities" yellow box in fig. 8, spanning the 3 tiers with a set of hard-coded structures (which end up being a mirror of the database tables).

Fast forward to the present (sort of), and you get introductory papers like Creating a Data Access Layer where again the same basic architecture is rehashed under the .NET 2.0 newfangled classes and wizards.

And oh yeah, if you really wanna feel up-to-date, LINQ will take care of the DB, no more SQL, thank you. Except they've just embedded SQL in C#, thereby exposing your code to the same fragility WRT changes in the database schema.

Now, why is Microsoft pushing (through authors and evangelists) old stuff like that? I've partially answered in a comment to a previous post, but I'll add a little more. It's not that they're not smart enough to do better. It's that they think we are not smart enough to do better (Sun doesn't think much differently either).
Indeed, the architecture they're selling is easy to explain, easy to understand, easy to implement piecemeal, without much thinking. It's almost a Marketecture (short for Marketing Architecture, contrast with Technical Architecture).

Here are a few half-baked thoughts for those of you with a little time to spare :-) and a sincere interest about creating modern (or post-modern) architectures:

A) Information Hiding is about hiding likely changes. Likely changes in a database-oriented architecture are:
1) the database engine itself (oracle, sql server, etc). That includes the SQL-dialect of the database, so don't rely entirely on odbc, ado and the like.
2) the data access technology (remember odbc, rdo, dao, ado, ado.net, ado.net 2.0, linq, all have been sold as the ultimate technology, yet every 2 years or so we get a new one).
3) the database schema itself.
The old-style architecture may do something about 1 and 2, but precious nothing for 3 (which is going to consume most of your time anyway).

Now, the database schema may change for several reasons. Over time, you will:
- normalize
- denormalize
- add/drop fields
- add/drop tables
- re-route relationships
- change cardinality in relationships
You need to understand the most likely changes, as these are shaping your context (and therefore influence the best form)

B) The interface between a (well-designed) Data Layer and the Business Layer must be loose. It shouldn't break when the database schema changes because you added a field. Therefore, if the interface is based on strongly typed entities which mirror the database schema, you're doomed.

C) The interface between a (well-designed) Business Layer and the UI Layer or Service Layer must be loose. See above.

D) Don't lock the architecture on the worst case. We all know that a lot of code behind the UI is not that smart.
In many cases, given a robust validation layer, which can be designed to be very flexible and dynamic, the business layer won't do much except routing data to / from the data layer.
Don't make the business layer a necessary burden. Make it an important, yet optional component that kicks in only when important business logic is needed.

E) Reflection is the key to flexible DB applications.

F) You can only get so far with language-based reflection at the Data Layer level, because SQL is too old/primitive. Sooner or later, you'll need to attach more semantics to each field than your DB wants you to (especially if you don't want to tie yourself to a single DB vendor). Be creative :-), as this would take too much space for a single post.

G) Static typing is great inside each layer. It's also great at the interface level when the structure we're talking about is stable. It's truly bad when you want to expose a flexible or changing structure.
Remember why we conceived XML in the first place? Data are fluid!

Ok, there would be more to say about semistructured data, service-oriented architectures and the like, but that will have to wait.

I'll just repeat my caveat: be wary about buying an architecture from your vendor. Apply a good dose of critical thinking and look for the real value in your specific context.
You wouldn't buy the architectural blueprint of your house from a bricks or pipes vendor, no matter the quality of those bricks and pipes. You normally shouldn't buy your application architecture from your language, tools, or operating system vendor either.

Being 10 Years Behind (part 1)

2008-01-14T12:22:00.000+01:00

In the last two years I've been working quite closely with a company, designing the fourth generation of a successful product. Indeed, a few of my posts have been inspired by the work I did on that project.
What we have now is a small, flexible, fast web application where we definitely pushed the envelope using AOP-like techniques pervasively, although in .NET/C#.

Compared with the previous generation, our application has more features, is much easier to customize (a must), is much easier to use thanks to the task-oriented design of the HCI (the previous generation was more slanted toward the useless computer approach) and also about 10 times faster (thanks to better database design and a smarter business layer).
Guess what, the source code size is just about 1/30 of what we had before (yeap, 30 times less), excluding generated code to read/write some XML files. The previous application was written in a mixture of C++, VB6, Perl, Python, C#.

Now, the company is considering a strict partnership with an Asian corporation. They have a similar product, in ASP.NET / C# as well. It took them something like 30 times our man-months to write it, so the general feeling was that it should have been "more powerful". Time to look at the features, but hey, features are basically the same, although their product is not task-oriented. If anything, they lack a lot of our customization interface.

Time to look at the code, as the code never lies.
The code is probably 50 times bigger, with no flexibility whatsoever. If you need to attach one more field to a business concept, just to track some custom information, you probably have to change the database, 5-8 classes between the database and the GUI, and the GUI itself.
In most cases, in our application we just need to open the administration console, add the field, specify validation rules, and that's it.
If you have special needs, you write a custom class, decorate the class with attributes, and we take care of all the plumbing to instantiate / call your class in the critical moments (that's one of the several places where the AOP-like stuff kick in).

What struck me when I looked at that code was the (depressing :-) similarity with a lot of old-style Java code I've been seeing over the years, especially in banking environments.

There is an EntityDAO (data access object) package with basically one class for each business entity. That class is quite stupid, basically dealing with persistence (CRUD) and exposing properties. Those classes are used "at the bottom" of the system, that is, near to the database.

Then there is an Entity package where (again!) there is basically one class for each (business) entity. The class is completely stupid, offering only get/set methods. Those classes are used "at the top" of the system, that is, near to the GUI or external services.

There is a BusinessLogic package where Entities gets passed to various classes as parameters, and EntityDAO objects are used to carry out persistency-related tasks.
Actually, inside the BusinessLogic lies a lot of database-related code, sometimes with explicit manipulation of DataRow classes. The alternative would have been to create much more EntityDAO classes.

Here and there, the coupling between the BusinessLogic and the database must have seemed too strong, so an EntityReader package has been created, where more sophisticated (yet still stupid) entities (or collections) are built using EntityDAO classes.

Finally, you just need :-) something to call from your service or GUI layer. The ubiquitous BusinessFacade package is therefore introduced, implementing a very large, use-case driven interface (put in yet another package), taking Entity instances as parameters and using the BusinessLogic.

At that point, people invariably realize that services need much more logic than what is provided by the BusinessLogic package, and so go ahead and create a (very sad) BusinessHelper package, where they complement all the missing parts in the BusinessLogic, most often by direct database access.

Then we have other subsystems (cache, SQL, and so on) all built around XXXManager classes, which we can ignore.

Of course, in the end everything is so coupled with the database schema that just adding a field results in a nightmare. And you get a lot of code to maintain as well. Good luck. Meanwhile, the Ruby On Rails guys are creating (simple) applications faster than the other guys can spell BusinessHelper. Say good-bye to productivity.

We can blame it on static typing, but reality is much simpler. That architecture is wrong. Is at least 10 years behind from the state of practice, which means is probably 15 to 20 years behind from the state of the art.
The problem is, that ancient architecture was popularized years ago mostly by language and tools vendors, or by people who thinks Architects don't have to understand code or the real problem being solved, just to replicate a trivial-yet-humonguous structure everywhere. It's basically a decontextualized architecture (more on this next time).

Indeed, if you look at the Java literature, you can find good books dating back to the early decade (like "EJB Design Patterns" by Floyd Marinescu, year 2002), where the pros and cons of several choices adopted in that overblown yet fragile architectural model are discussed. When a Patterns book appears, a few years of practice are gone by. That was 2002; now, ten years are gone, and yet developers still fall into the same trap.

It gets worse. While even the most outdated Java applications are gradually moving away from that model (see Untangling Enterprise Java by Chris Richardson for a few ideas), Microsoft evangelists are so excited about it. They happily go around (re)selling an architecture that is remarkably similar to the 10-years-behind behemoth above.

Which brings me to "Being 10 Years Behind (part 2)". Stay tuned :-).

Happy New Year!!!!

2007-12-31T11:16:00.000+01:00

Wish you all a wonderful 2008!