Tuesday, July 22, 2008 

SmartFP™ paper (and tool) online

As promised, I've uploaded a free, simple tool to calculate Function Points using a decision tree. I've also uploaded a (draft) paper describing the overall approach. The paper is still missing a case study, which would help, but I just wanted to put the whole thing online. I'll add the case study, and a few more details, before submitting the paper for publication.

The decision tree approach is quite simple, especially if you have some knowledge of function points. Although it may seem like a small change in perspective from the usual "counting" approach, the result is that we can save a lot of time doing a function point estimate, and in many cases we also get more robust results.

Experiences and feedback are welcome, as usual. You can find the whole thing on the SmartFP page.

Note: as I plan to make more tools and libraries freely available, I've also created a new "tools" page. So far, there is only a link to BetterEstimate and SmartFP, but more will come...

Labels: , ,

Tuesday, May 27, 2008 

On the concept of Form (3): the Force Field

Warning: :-) this post is going to be somehow conceptual. I'll soon move to some real-world, software-based example, but I really need to introduce some concepts first.

The notion of force field might be unfamiliar to some, so I'll borrow a great example from Alexander himself. Consider your first "requirement" (for a system yet to be built) as a permanent magnet of some size and shape. If you place a flat glass over that magnet, and drop some iron filings on it, the iron will naturally dispose along the magnetic field lines. That gives us an image of [a section of] the force field. Now add another magnet: the shape of the field will change, as the magnets are interacting, thereby shaping a more complex force field.
We can change the [shape of the] field in many ways: moving magnets around, changing their shape, their magnetization, or even adding some shields around magnets.

The great thing about the magnetic field is that we can somehow observe its shape. Indeed, if our goal was to create a form that can be put into effortless contact with the field, we'll just have to replicate the same form that the magnetic field is giving to the iron filings. As Alexander says (NoTSoF, page 21), "once we have a diagram of forces [...] this will in essence also describe the form as a complementary diagram of forces".
In the real world, and even more so in the software world, we are never so lucky: the force field is invisible and tends also to be highly unstable.

Usually, the force field of a software project starts with Requirements. Requirements are often categorized in some way, like "functional" and "nonfunctional", or "user requirements" and "system requirements. However, requirements of any kind are just like magnets: they contribute to shape the overall field.

Requirements are just one kind of force, that is, they are not alone in shaping the field. Many technological choices we make, sometimes very (or too) early, are also shaping the force field.
Consider a simple business application. Once you decide that you'll build a web application, you have added quite a few powerful magnets. If you're familiar with JSP and EJB, you are naturally tempted to choose those technologies early on. That's like adding quite a few powerful magnets again. Or maybe it's like adding a magnetic shield: it really depends on context.

Sometimes, technology makes the field simpler: the right infrastructure should simplify the field, that is, it should act more like a shield than like a magnet. In this sense, infrastructure should be chosen when the dominant forces are known, unlike what happens in many projects, where infrastructure (usually a superstructure in disguise) is chosen too early, thereby making the overall field even more complex.

We also shape the field, so to speak, by choosing what to ignore and what to postpone in any given release. Anything we ignore, like anything we postpone, won't be allowed to shape the field right now.
This is fine, as long as the corresponding magnets will be placed somehow distant from the others (good modularity), possibly with some kind of magnetic shielding in between (stable interfaces). It's also fine if we can ignore it forever. Any attempt to temporarily ignore a strictly interacting force will wreak havoc later on, as our form will no longer match the resulting force field. Refactoring can accommodate minor misfits with the ideal form, but won't help much when the force field changes radically (see also my notes on refactoring here).

Here lies one of the architect's fundamental abilities: the intuitive understanding that something can be beneficially postponed, while something else must be dealt with immediately, because its influence on the force field is so strong that doing otherwise will shift us toward the wrong kind of form.

It is important to understand the role of choice in exploiting instability. Too often, software developers tend to see requirements as "fixed". They don't like to negotiate: it's much easier to fight the compiler than the marketing guys.
A good architect, however, can't miss the opportunity to simplify the field by moving some magnets around. That requires the ability to see the overall picture and the fine details at the same time. Here is Alexander again (page 18): "this ability to deal with several layers of form-context boundaries in concert is an important part of what we often refer to as the designer's sense of organization. The internal coherence of an ensemble depends on a whole net of such adaptations".
That ain't an easy feat. It requires an understanding of the business, the users, and the technology. And even more important, it requires a willingness to act on that knowledge. The power of choice extends to the infrastructure: sometimes, by willingly postponing a technological choice until the force fields takes shape, we can make a better, more "natural" choice.

This can be hard for some developers: they want certainty, and they want it now. In my experience, that goes in pair with the willingness to adopt a sub-optimal, but repetitive and context free solution for a wide class of problems, instead of adopting several optimal, but reasoned and context-dependent solution for smaller classes of problems.

Unfortunately, choosing the "wrong" technology is very much like choosing the wrong shape or orientation for a building. To quote Alexander once again (page 29): "Instead of orienting the house carefully for sun and wind, the builder conceives its organization without concern for orientation, and light, heat, and ventilation are taken care of by fans, lamps, and other kinds of peripheral devices. Bedrooms are not separated from living rooms in plan, but are placed next to one another and the walls between them stuffed with acoustic insulation".
I think we can easily see a parallel with software here: a misfit technology is chosen early on. As a consequence, you find yourself adding more and more technology (fans, lamps, insulation) to satisfy the end-user needs. "Modern" web applications seem to have taken this path: faced with a difficult field, they're layering one technology on top the other, desperately trying to overcome the problems of the previous layer.

Next time, in no particular order: agility, unstable requirements, early coding, TDD, "seeing" the field, internal and external representations, is UML any useful, order within chaos (dominant forces), constructive force field and systematic techniques, and whatever else will come to my mind :-).

Labels: , , ,

Tuesday, May 13, 2008 

Natural language

Some (most :-) of my clients are challenging. Sometimes the challenge comes from the difficult technical problems they face. That's the best kind of challenge.
Sometimes the challenge comes from people: that's the worst kind of challenge, and one that right now is better left alone.
Sometimes the challenge comes from the organization, which means it also comes from people, but with a different twist. Challenges coming from the organization are always tough, but overcoming those challenges can really make a difference.

One of my challenging clients is a rather large company in the financial domain. They are definitely old-school, and although upper management can perfectly see how software is permeating and enabling their business, middle management tend to see software as a liability. In their eternal search for lower costs, they moved most of the development offshore, keeping only an handful of designers and all the analysts in-house. Most often, design is done offshore as well, for lack of available designers on this side of the world.

Analysts have a tough job there. On one side, they have to face the rest of the company, which is not software-friendly. On the other side, they have to communicate clear requirements to the offshore team, especially to the designers, who tend to be very technology-oriented.
To make things more complicated, the analysts often find themselves working on unfamiliar sub-domains, with precise regulations but also with large gray areas that must be somehow understood and communicated.
Icing on the cake: some of those financial instruments do not even exist in the local culture of the offshore team, making communication as difficult as ever.

Given this overall picture, I've often recommended analysts to spend some time creating a good domain model (usually, a UML class diagram, occasionally complemented by some activity diagrams).
The model, with unambiguous associations, dependencies, multiplicities, and so on, will force them to ask the right questions, and will make it easier for the offshore designer to acquaint himself with the problem. Over time, this suggestion has been quite helpful.
However, as I said, the organization is challenging. Some of the analysts complained that their boss is not satisfied by a few diagrams. He wants a lengthy, wordy explanation, so he can read it over and see if they got it right (well, that's his theory anyway). The poor analyst can't possibly do everything in the allotted time.

Now, I always keep an eye on software engineering research. I've seen countless attempts to create UML diagrams from natural language specifications. The results are usually unimpressive.
In this case, however, I would need exactly the opposite: a tool to generate a precise, yet verbose domain description out of a formal domain model. The problem is much easier to solve, especially because analysts can help the tool, by using the appropriate wording.

Guess what, the problem must be considered unworthy, because there is a dearth of works in that area. In practice, the only relevant paper I've been able to find is Generating Natural Language specifications from UML class diagrams by Farid Meziane, Nikos Athanasakis and Sophia Ananiadou. There is also Nikos' thesis online, with a few more details.
The downside is that (as usual) the tool they describe does not seem to be generally available. I've yet to contact the authors: I just hope it doesn't turn out to be one of those Re$earch Tool$ that never get to be used.

From the paper above, I've also learnt about ModelExplainer , a similar tool from a commercial company. Again, the tool doesn't seem to be generally available, but I'll get in touch with people there and see.

Overall, the problem doesn't seem so hard, especially if we accept the idea that the analyst will help the tool, choosing appropriate wording. An XMI-to-NL (Natural Language) would make for a perfect open source project. Any takers? :-)

Labels: , , , ,

Tuesday, December 18, 2007 

Problem Frames

I became aware of problem frames in the late 90s, while reading an excellent book from Michael Jackson (Software Requirements and Specifications; my Italian readers may want to read my review of the book, straight from 1998).

I found the concept interesting, and a few years later I decided to dig deeper by reading another book from Jackson (Problem Frames: Analyzing and structuring software development problems). Quite interesting, although definitely not a light reading. I cannot say it changed my life or heavily influenced the way I work while I analyze requirements, but it added a few interesting concepts to my bag of tricks.

Lately, I've found an interesting paper by Rebecca Wirfs-Brock, Paul Taylor and James Noble: Problem Frame Patterns: An Exploration of Patterns in the Problem Space. I encourage you to read the paper, even if you're not familiar with the concept of problem frame: in fact, it's probably the best introduction on the subject you can get anywhere.

In the final section (Assessment and Conclusions) the authors compare Problem Frames and Patterns. I'll quote a few lines here:

A problem frame is a template that arranges and describes
phenomena in the problem space, whereas a pattern maps forces to a solution in the solution space.

Patterns are about designing things. The fact that we put problem frames into pattern form demonstrates
that when people write specifications, they are designing too—they are designing the overall system, not its
internal structure. And while problem frames are firmly rooted in the problem space, to us they also suggest

If you read that in light of what I've discussed in my latest post on Form, you may recognize that Problem Frames are about structuring and discovering context, while Design Patterns helps us structure a fitting form.
When Problem Frames suggests solutions, there is a good chance that they're helping us in the elusive game of (to quote Alexander again) bringing harmony between two intangibles: a form which we have not yet designed, and a context which we cannot properly describe.

Back to the concept of Problem Frames, I certainly hope that restating them in a pattern form will foster their adoption. Indeed, the paper above describes what is probably the closest thing to true Analysis Patterns, and may help analysts look more closely at the problem before jumping into use cases and describe the external behaviour of the system.

Labels: , , , , ,

Sunday, December 16, 2007 

On the concept of Form (2)

In the Notes, Alexander defines form in a rather unique and interesting way: "The form is the part of the world over which we have control, and which we decide to shape while leaving the rest of the world as it is. The context is that part of the world which puts demands on this form; anything in the world that makes demands on the form is context. [...] We want to put the context and the form into effortless contact or frictionless coexistence" (pg. 18-19).

Alexander's definition goes well beyond the old-school, classical dichotomy between form and function, and also beyond the old-school dichotomy between functional and non-functional requirements. In fact, function intended as purpose is part of the context: functional requirements are obviously putting demands on form. Non-functional requirements are part of the context as well, and they should be considered first-class citizens too.

It is important to understand the human role in the design process. It is first an act of selection, and then an act of shaping.
We decide to shape a part of the world and leave the rest as it is. These are powerful words. I've long contended that in most case we have a much higher degree of control over requirements (context) than we're inclined to believe.
I've also contended for a very long time (mostly inspired by Tom Peters, I guess) that efficiency demands requirements (that is, context) to be developed jointly by developers and marketing.

Move away from this, and you get an inefficient context: a set of requirements (mostly functional) which may put strenuous demands on the form, without a corresponding premium in the resulting value. If you wanna win the game, you have to set the right rules.
In software development, more than in any other discipline, we can choose the right context before we step up to build our product. Of course, it is much easier to get a "fixed" set of requirements, complain about marketing lack of understanding, and start coding. But that's just not efficient. Also, keeping the context fixed while we shape the form is inefficient, as we wouldn't be profiting from newly discovered opportunities.

Building form, therefore, is an incremental process of discovery, as we don't know the exact context, and even if we do, we don't know if that's the context we wanna play with (or against, if you're the competitive kind ;-).
Indeed, Alexander reminds us that "In a real design project [...] we are searching for some kind of harmony between two intangibles: a form which we have not yet designed, and a context which we cannot properly describe" (pg. 26). It is not hard to see a deep connection with the notion of "software development as knowledge acquisition" popularized by Armour.

It is also important to understand a simple, yet deep consequence of Alexander's definition of form. As we create a product, even a software product, we are shaping its form. Even if we ignore design altogether, or even if we just concentrate on making it work, we are shaping some kind of form.
In most piecemeal development cases, that form will be determined by a small subset of forces: functional requirements, and sometimes the viscosity of previously developed software as well. The resulting structure won't be in frictionless contact with the true context, and the resulting friction will make our product brittle.

There is still a lot to be said about the subtle relationship between the above and agility, about the concept of shaping software, the [constructive] force field, and about the ubiquitous real options as well. Some more software-oriented examples are probably badly due too. And I should really say something about inventing requirements! So many things, so little time :-). Stay tuned!

Labels: , ,

Sunday, December 02, 2007 

More synchronicity

While reading the same issue of IEEE Software that I mentioned in my previous post, I came across a paper from Ed Yourdon, "Celebrating Peopleware's 20th Anniversary". I mentioned Peopleware while answering a comment to Process as the company's homeostatic system, but so far, it's just a coincidence.

It got more interesting as I went further, as Ed says:
"my colleague Larry Constantine and I had borrowed an even earlier collection of Alexander's ideas from his 1964 book Notes on the Synthesis of Form as the basis for the structured-design concepts of coupling and cohesion."
Oh, look, Notes on the Synthesis of Form, that's another interesting coincidence :-).

Speaking of cohesion, I should note that the process described by Alexander (modeling relationships between requirements as a weighted graph) has strong resemblances to the process adopted in KAOS (which I've mentioned in several posts now). The purpose is different however, as Alexander aims to derive clusters of highly cohesive requirements mechanically, while KAOS is leaning more on the soft side, allowing people to "see" the interplay between requirements.

Funny enough, in the same issue there is a paper by Simon Buckingham Shum ("There's Nothing Like a Good Argument..."), which describes a tool (Compendium) "providing a flexible visual interface for managing the connections between information and ideas" (from the compendium website).
I haven't tried it out yet, but from the screenshots, it seems to embody everything we need to apply a KAOS-like technique to requirements analysis, and also keep track of major design decisions.

Gee, everything seems so interconnected these days :-)

Labels: , , ,

Monday, October 31, 2005 

Teaching SOFTWARE Project Management

Over the years, a number of customers asked me to teach project management techniques to their team leaders and project/product managers. The reason is quite simple: traditional project management techniques don't work so well for software projects. Sure, you are better off if you know how to use PERT and GANTT charts, and you may still benefit from some traditional lecturing on risk management, but software is different. The best reason for software to be different comes from Armour: software development is a knowledge acquisition process, not a product manufacturing process. That's a huge difference.
Despite the large number of requests, I've never committed myself to create a set of PM slides. For more than a few years, I've been firm into telling that I knew enough to run a successful project, and even enough to advise on how to run a specific project, but not enough to teach how to do it in general (which is what I would expect from a PM course).
In the last year, I've spent more and more time thinking on what I could actually teach - valuable, modern, software-specific techniques that I've tried in the real world and that I can trust to work. It turned out that I knew more than I thought, but also that I couldn't teach those techniques without first teaching some (even more fundamental) conceptual tools, like Armour Ignorance Orders or Project Portfolios or Option Thinking, and so on.
In these days I'm polishing the slides I've created, and trying to create a natural bridge between those slides and some of my material on Requirements Analysis. This is probably a good chance to review that material as well, along the lines I've envisioned a few months ago. So, very soon I'll have a new, short, hopefully fun course on PM appearing on my course catalogue.

Labels: , ,