Carlo Pescio - blog

Warning: :-) this post is going to be somehow conceptual. I'll soon move to some real-world, software-based example, but I really need to introduce some concepts first.

The notion of force field might be unfamiliar to some, so I'll borrow a great example from Alexander himself. Consider your first "requirement" (for a system yet to be built) as a permanent magnet of some size and shape. If you place a flat glass over that magnet, and drop some iron filings on it, the iron will naturally dispose along the magnetic field lines. That gives us an image of [a section of] the force field. Now add another magnet: the shape of the field will change, as the magnets are interacting, thereby shaping a more complex force field.
We can change the [shape of the] field in many ways: moving magnets around, changing their shape, their magnetization, or even adding some shields around magnets.

The great thing about the magnetic field is that we can somehow observe its shape. Indeed, if our goal was to create a form that can be put into effortless contact with the field, we'll just have to replicate the same form that the magnetic field is giving to the iron filings. As Alexander says (NoTSoF, page 21), "once we have a diagram of forces [...] this will in essence also describe the form as a complementary diagram of forces".
In the real world, and even more so in the software world, we are never so lucky: the force field is invisible and tends also to be highly unstable.

Usually, the force field of a software project starts with Requirements. Requirements are often categorized in some way, like "functional" and "nonfunctional", or "user requirements" and "system requirements. However, requirements of any kind are just like magnets: they contribute to shape the overall field.

Requirements are just one kind of force, that is, they are not alone in shaping the field. Many technological choices we make, sometimes very (or too) early, are also shaping the force field.
Consider a simple business application. Once you decide that you'll build a web application, you have added quite a few powerful magnets. If you're familiar with JSP and EJB, you are naturally tempted to choose those technologies early on. That's like adding quite a few powerful magnets again. Or maybe it's like adding a magnetic shield: it really depends on context.

Sometimes, technology makes the field simpler: the right infrastructure should simplify the field, that is, it should act more like a shield than like a magnet. In this sense, infrastructure should be chosen when the dominant forces are known, unlike what happens in many projects, where infrastructure (usually a superstructure in disguise) is chosen too early, thereby making the overall field even more complex.

We also shape the field, so to speak, by choosing what to ignore and what to postpone in any given release. Anything we ignore, like anything we postpone, won't be allowed to shape the field right now.
This is fine, as long as the corresponding magnets will be placed somehow distant from the others (good modularity), possibly with some kind of magnetic shielding in between (stable interfaces). It's also fine if we can ignore it forever. Any attempt to temporarily ignore a strictly interacting force will wreak havoc later on, as our form will no longer match the resulting force field. Refactoring can accommodate minor misfits with the ideal form, but won't help much when the force field changes radically (see also my notes on refactoring here).

Here lies one of the architect's fundamental abilities: the intuitive understanding that something can be beneficially postponed, while something else must be dealt with immediately, because its influence on the force field is so strong that doing otherwise will shift us toward the wrong kind of form.

It is important to understand the role of choice in exploiting instability. Too often, software developers tend to see requirements as "fixed". They don't like to negotiate: it's much easier to fight the compiler than the marketing guys.
A good architect, however, can't miss the opportunity to simplify the field by moving some magnets around. That requires the ability to see the overall picture and the fine details at the same time. Here is Alexander again (page 18): "this ability to deal with several layers of form-context boundaries in concert is an important part of what we often refer to as the designer's sense of organization. The internal coherence of an ensemble depends on a whole net of such adaptations".
That ain't an easy feat. It requires an understanding of the business, the users, and the technology. And even more important, it requires a willingness to act on that knowledge. The power of choice extends to the infrastructure: sometimes, by willingly postponing a technological choice until the force fields takes shape, we can make a better, more "natural" choice.

This can be hard for some developers: they want certainty, and they want it now. In my experience, that goes in pair with the willingness to adopt a sub-optimal, but repetitive and context free solution for a wide class of problems, instead of adopting several optimal, but reasoned and context-dependent solution for smaller classes of problems.

Unfortunately, choosing the "wrong" technology is very much like choosing the wrong shape or orientation for a building. To quote Alexander once again (page 29): "Instead of orienting the house carefully for sun and wind, the builder conceives its organization without concern for orientation, and light, heat, and ventilation are taken care of by fans, lamps, and other kinds of peripheral devices. Bedrooms are not separated from living rooms in plan, but are placed next to one another and the walls between them stuffed with acoustic insulation".
I think we can easily see a parallel with software here: a misfit technology is chosen early on. As a consequence, you find yourself adding more and more technology (fans, lamps, insulation) to satisfy the end-user needs. "Modern" web applications seem to have taken this path: faced with a difficult field, they're layering one technology on top the other, desperately trying to overcome the problems of the previous layer.

Next time, in no particular order: agility, unstable requirements, early coding, TDD, "seeing" the field, internal and external representations, is UML any useful, order within chaos (dominant forces), constructive force field and systematic techniques, and whatever else will come to my mind :-).

Labels: architecture, design, form, requirements

Aspect-oriented programming is still largely code-centric. This is not surprising, as OOP went through the same process: early emphasis was on coding, and it took quite a few years before OOD was ready for prime time. The truth about OOA is that it never really got its share (guess use cases just killed it).

This is not to say that nobody is thinking about the so-called early aspects. A notable work is the Theme Approach (there is also a good book about Theme). Please stay away from the depressing idea that use cases are aspects; as I said a long time ago, it's just too lame.

My personal view on early aspects is quite simple: right now, I mostly look for cross-cutting business rules as candidate aspects. I guess it's quite obvious that the whole "friend gift" concept is a business rule cutting through User and Subscription, and therefore a candidate aspect. Although I'm not saying that all early aspects are cross-cutting business rules (or vice-versa), so far this simple guideline has served me well in a number of cases.

It is interesting to see how early aspects tend to be cross-cutting (that is, they hook into more than one class) but not pervasive. An example of pervasive concern is the ubiquitous logging. Early aspects tend to cut through a few selected classes, and tend to be non-reusable (while a logging aspect can be made highly reusable).
This seems at odd with the idea that "AOP is not for singleton", but I've already expressed my doubt on the validity of this suggestion a long time ago. It seems to me that AOP is still in its infancy when it comes to good principles.

Which brings me to obliviousness. Obliviousness is an interesting concept, but just as it happened with inheritance in the early OOP days, people tend to get carried over.
Remember when white-box inheritance was applied without understanding (for instance) the fragile base class problem?
People may view inheritance as a way to "patch" a base class and change its behaviour in unexpected ways. But truth is, a base class must be designed to be extended, and extension can take place only through well-defined extensions hot-points. It is not a rare occurrence to refactor a base class to make extension safer.

Aspects are not really different. People may view aspects as a way to "patch" existing code and change its behaviour in unexpected ways. But truth is, when you move outside the safe realm of spectators (see my post above for more), your code needs to be designed for interception.
Consider, for instance, the initial idea of patching the User class through aspects, adding a data member, and adding a corresponding data into the database. Can your persistence logic be patched through an aspect? Well, it depends!
Existing AOP languages can't advise any given line: there is a fixed grammar for pointcuts, like method call, data member access, and so on. So if your persistence code was (trivially)

class User
  {
  void Save()
    {
    // open a transaction
    // do your SQL stuff
    // close the transaction
    }
  }

there would be no way to participate to the transaction from an aspect. You would have to refactor your code, e.g. by moving the SQL part in a separate method, taking the transaction as a parameter. Can we still call this obliviousness? That's highly debatable! I may not know the details of the advice, but I damn sure know I'm being advised, as I refactored my code to be pointcut-friendly.

Is this really different from exposing a CreateSubscription event? Yeah, well, it's more code to write. But in many data-oriented applications, a well-administered dose of events in the CRUD methods can take you a long way toward a more flexible architecture.

A closing remark on the SOA part. SOA is still much of a buzzword, and many good [design] ideas are still in the decontextualized stage, where people are expected to blindly follow some rule without understanding the impact of what they're doing.

In my view, a crucial step toward SOA is modularity. Modularity has to take place at all levels, even (this will sound like an heresy to some) at the database level. Ideally (this is not a constraint to be forced, but a force to be considered) every service will own its own tables. No more huge SQL statements traversing every tidbit in the database.

Therefore, if you consider the "friend gift" as a separate service, it is only natural to avoid tangling the User class, the Subscription class, and the User table with information that just doesn't belong there. In a nutshell, separating a cross-cutting business rule into an aspect-like class will bring you to a more modular architecture, and modularity is one of the keys to true SOA.

Labels: AOP, article reference, book reference, design

Note: I began tinkering with this idea several months ago, inspired by some design work I was doing on a real-world project. Initially, I thought I could pimp up this stuff into a full-fledged article. Months came by and I didn't, so (in the spirit of my old concept of Blogging as Destructuring) I thought I could just as well say something here instead.

Consider a web site offering some kind of service to users. Users have to register (yeah :-), and they can then buy a subscription to several services. A simple class diagram can model this easily:



In most cases, the underlying database schema wouldn't be much different.

In a real case, there might be different kind of services, each requiring a derived class, but we can ignore this issue right now. Also, there might be several kind of subscription (time-based, pay-per-use, and so on), but again, let's ignore that issue right now, and just concentrate on time-base subscriptions.

A subscription can be renewed at any time. In the object model, this would translate into a Renew method in class Subscription. Renew could take a parameter, like the extension of the renewal. Most likely, it would add a new record to the Subscription table (to keep track of the whole subscription history for that user), and possibly create a new Subscription object. So far, so good.

Now the marketing guys come up with a nice idea: whenever you register, you can provide the email address of a friend who has already registered. If you do, everytime you renew a subscription your friend will get some kind of gift, like a free extension or whatever. This may generate a few more leads, meaning a little more business.

From a purely OO mindset, this may lead us to perform a little maintenance on the existing model. We can add a relationship from User to User to model the "friend" relationship, or we could just add a field like friendEmail (possibly left empty). At the database level, adding a field is probably easier.
We can also modify the Renew method to check for the presence of a friend in the subscribing user, and if so, invoke some Gift logic. I won't draw a modified diagram for this scenario: I guess it's just too obvious.

Now, this approach obviously works. However, you may recognize that the whole "friend gift" concept is a cross-cutting concern: it cuts through User (requiring new data) and through Subscription (requiring new logic). More on detecting cross-cutting concerns during analysis and design (and on the difference between cross-cutting and pervasive concerns) next time.

In the AOP world, we could approach the problem differently. We could define a FriendGift aspect. The aspect may add a new data member to the User class (the friendEmail), and intercept the persistence logic to save / load that data from the database. The aspect may also intercept Renew and perform the Gift logic if required.
Actually the aspect doesn't have to modify User (class and table); indeed, it would be better not to, as the persistence logic might not be easy to intercept (more on this next time). The aspect could just use a different database table to store the friend email.

Using an UML-like notation, we could model this approach as:



Interestingly, the database is probably different from the previous scenario: Friend would map to its own table.

What if we are not using an AOP-enabled language? For instance, we might be using plain old C#. Can we still borrow some ideas from the above? I believe so. In the same sense as OO thinking can inform traditional structured programming, AOP thinking can inform traditional object oriented programming.

If we don't have pointcuts, join points and aspects, we may still have events (in C#/.NET, or callback in other languages, and so on). Sure, we have to forego obliviousness (which might not be so bad: more on this next time). But we can come up with a more modular and decoupled design by reasoning along the AOP lines:



No rocket science here :-). Just a different form, same function. Different database, a more general Subscription class, unaffected by a business rule which may change at any time. Also the User class and table are left unaffected.

More on this, and a few comments on the SOA part, in just a few days (I hope :-).

Labels: AOP, design

In a comment to my previous post, I mentioned an half-baked idea about a (truly) modern architecture for GUI applications.

In fact, I've been experimenting with different approaches to GUI construction for "classic" Windows applications, Web applications, Smart Clients calling web services, and even Web applications calling web services for quite a while now.

Some were dead ends. Others have been a successful intermediate step toward something better. I usually pass on the good ideas to my clients, and meanwhile, I keep trying different approaches on small scale, low-risk projects. Because the "traditional" way of building GUI applications sucks big time. So does the traditional thinking about "good" GUI constructions, that is, that you need a tightly integrated language / ide / library / operating system to do anything decent.

I also don't like the idea that I should buy-in a monolithic framework for GUI construction. Different problems are better solved in different ways: we should be free to mix and match different approaches in different forms (within the same application), or even within the single form, or component.

Over time, a few people told me not to bother: Microsoft, Sun, Apple (not to mention a thousand smart guys working for smaller companies) have been working at that for years. It's an old problem, and it's unlikely that anybody can come up with a radically different solution.

It's very poor thinking. There is no limit to human creativity. Take a look at How To Fold A T-Shirt In 2 Seconds. The problem is much older than GUI construction. Uncountable people have been faced with the same problem and they didn't come up with anything similar so far. Sure, there are pros and cons in this technique (as usual), but it's a radically different (and much faster) approach.

You may want to go through the explained version, as I did. I actually tried that a few times, and it works :-).

Labels: design, GUI, link

I'm working on an interesting project, trying to squeeze all the available information from sampled data and make that information useful for non-technical users. I can't provide details, but in the end it boils down to reading a log file from a device (amounting to about 1 hour of sampled data from multiple channels), do the usual statistics, noise filtering, whatever :-), calculate some pretty useful stuff, and create a report that makes all that accessible to a business expert.

The log file is (guess what :-) in XML format, meaning it's really huge. However, thanks to modern technology, we just generated a bunch of classes from the XSD and let .NET do the rest. Parsing is actually pretty fast, and took basically no time to write.
In the end, we just get a huge collection of SamplingPoint objects. Each Sampling point is basically a structure-like class, synthesized from the XSD:

class SamplingPoint
{
public DateTime Timestamp { // get, set }
public double V1 { // get, set }
// ...
public double Vn { // get, set }
}

each value (V1...Vn) is coming from a different channel and may have a different unit of measurement. They're synchronously sampled, so it made sense for whoever developed the data acquisition module to group them together and dump them together in a single SamplingPoint tag.

We extract many interesting facts from those data, but for each Vi (i=1...N) we also show some "traditional" statistics, like average, standard deviation and so on.
Reasoning about average and standard deviation is not for everyone: I usually consider an histogram of the distribution much easier to understand (and to compare with other histograms):



Here we see the distribution of V1 over time: for instance, V1 had a value between 8 and 9 for about 6% of the time. Histograms are easy to read, and users quickly asked to see histograms for each V1..Vn over time. Actually, since one of the Vj is monotonically increasing with time, they also asked to see the histogram of the remaining Vi against Vj too. So far, so good.

Now, sometimes I hate writing code :-). It usually happens when my language doesn't allow me to write beautiful code. Writing a function to calculate the histogram of (e.g.) V1 against time is trivial: you end up with a short piece of code taking an array of SamplingPoints and using the V1 and Timestamp properties to calculate the histogram. No big deal.

However, that function is not reusable, exactly because it's using V1 and Timestamp. You can deal with this in at least 3 unpleasant :-) ways:

1) you don't care: you just copy/paste the whole thing over and over. If N = 10, you get 19 almost-identical functions (10 for time, 9 for Vj).

2) you restructure your data before processing. Grouping all the sampled data at a given time in a single SamplingPoint structure makes a lot of sense from a producer point of view, but it's not very handy from a consumer point of view. Having a structure of arrays (of double) instead of an array of structures would make everything so much simpler.

3) you write an "accessor" interface and N "accessors" classes, one for each Vi. You write your algorithms using accessors. Passing the right accessors (e.g. for time and V1) will get you the right histogram.

All these options have some pros and cons. In the end, I guess most people would go with (2), because that brings us into the familiar realm of array-based algorithms.

However, stated like this, it seems more like a "data impedance" problem between two subsystems than a language problem. Why did I say it's the language fault? Because the language is forcing me to access data members with compile-time names, and does not (immediately) allow me to access data members using run-time names.

Don't get me wrong: I like static typing, and I like compiled languages. I know from experience that I tend to make little stupid mistakes, like typing the wrong variable name and stuff like that. Static typing and compiled languages catch most of those stupid mistakes, and that makes my life easier.

Still, the fact that I like something doesn't mean I want to use that thing all the time. I want to have options. Especially when those options would be so simple to provide.

In a heavily reflective environment like .NET, every class can be easily considered an associative container, from the property/data member names to property/data member values. So I shold be able to write (if I wanted):

SamplingPoint sp = ... ;
double d1 = sp[ "V1" ] ;

which should be equivalent to

double d1 = sp.V1 ;

Of course, that would make my histogram code instantly reusable: I'll just pass the run-time names of the two axes. You can consider this equivalent to built-in accessors.

Now, I could implement something like that on my own, using reflection. It's not really difficult: you just have to gracefully handle collections, nested objects, and so on. Unfortunately, C# (.NET) do not allow a nice implementation of the concept, mostly for a bunch or constraints they added to conversion operators: no generic conversion operators (unlike C++), no conversion to/from Object, and so on. In the end you may need a few more casts that you'd like to, but it can be done.

I'll also have to evaluate the performance implications for this kind of application, but I know it would make my life much easier in other applications (like binding smart widgets to a variety of classes, removing the need for braindead "controller" classes). It's just a pity that we don't have this as built-in language feature: it would be much easier to get this right (and efficient) at the language level, not at the library level (at least, given C# as it is now).

Which brings me to the concept of symmetry. A few months ago I stumbled upon a paper by Jim Coplien and Zhao Liping (Understanding Symmetry in Object-Oriented Languages, published in Journal of Object Technology, an interesting, free publications that's filling the void left by the demise of JOOP). Given my interest on the concept of form in software, the paper caught my attention, but I postponed further thinking till I could read more on the subject (there are several papers on symmetry in Cope's bibliography, but I need a little more time than I have). A week ago or so, I've also found (and read) another paper from Zhao in Communications of ACM, March 2008: Patterns, Symmetry, and Symmetry breaking.

Some of their concepts sound odd to me. The concept of symmetry is just fine, and I think it may help to unravel some issues in language design.
However, right now the idea that patterns are a way to break symmetry doesn't feel so good. I would say exactly the opposite, but I really have to read their more mathematically-inclined papers before I say more, because words can be misleading, while theorems usually are not :-).

Still, the inability to have built-in, natural access to fields and properties through run-time names struck me as a lack of symmetry in the language. In this sense, the Accessor would simply be a way to deal with that lack of symmetry. Therefore it seems to me that patterns are a way to bring back symmetry, not to break symmetry. In fact, I can think of many cases where patterns "expose" some semantic symmetry that was not accessible because of (merely) syntactic asymmetry.

More on this as I dig deeper :-).

Labels: article reference, design, form, language design, link

I mentioned the Domain Neutral Component when answering a comment to my post on the Cognitive Dimensions of Notations. That reminded me I've never been a fan of the DNC, exactly because of its context-independent ambitions. Recently, however, I've reconsidered the DNC in terms of Problem Frames.
It's kinda late, so more on this soon, I hope...

Labels: analysis, design

I constantly hear that the blogsphere is highly self-referential, so what's better than starting a post by quoting yourself? Here is an excerpt from Listen to Your Tools and Materials:
Our material is knowledge, or information. We acquire, understand, filter, and structure information; and we encode it in a variety of formats: text, diagrams, code, and so on.[...]the diagram isn't a material, just a representation. We use a tool to represent the material, which is intangible knowledge. [...] What's peculiar with software is that, in many cases, the tools and the materials have the same nature.

Note that I'm using "tool" in a very broad sense there. UML is not our material, is a tool we use to encode our material (and yeah, we may also use a software tool to draw the UML). Source code is not our material, is a tool we use to encode our material.
The fact that we often confuse tools and materials, and that we can get along with that just fine, empirically proves that they are somehow similar in nature.

Now, If our tools and materials have similar nature, then perhaps by understanding better the properties of our tools we can also understand better the properties of our materials, and ultimately of our creations. Assuming, of course, that we can somehow define and classify some interesting properties of our tools (that is, notations).

Turns out that people have been doing so for quite a while now. There is an interesting body of knowledge about the so-called Cognitive Dimensions of notations. I'll quote some portions from a short paper (Cognitive Dimensions of Notations: Design Tools for Cognitive Technology), but if you're interested, I recommend you dig deeper by reading Cognitive Dimensions of Information Artefacts: a tutorial.

When we use a notation, we are both limited and enabled by its own peculiarities. Our own style further emphasizes some of those attributes. For instance, a class diagram is relatively easy to change: you can quickly reshape your design into a very different one. In the Cognitive Dimensions vocabulary, we would say that its viscosity is low. However, if you have also modeled some scenarios on that class diagram (using for instance a sequence diagram) you have to work much harder to make changes. Sequence diagram have high viscosity.

Now, I really like the choice of viscosity as an attribute. I have to confess: I like it because it's not nerdy. When we're dealing with intangible information, it's easy to find ourselves lacking the necessary words. In software development, there is then a wide tendency to adopt some weird, geeky term.
Perhaps that's why I've immediately appreciated the Cognitive Dimensions framework. They put some considerable effort in creating the vocabulary itself. Here is a quote from the aforementioned paper: We believe that this problem is best addressed by providing a vocabulary for discussing the design problems that might arise – a vocabulary informed by research in cognitive psychology, but oriented toward the understanding of a system developer. The Cognitive Dimensions of Notations are such a vocabulary. Bingo! :-)

If you spend some time familiarizing with the vocabulary, you'll see how useful it can be to settle down some long-standing debates, like code Vs. diagrams or even static typing Vs. dynamic typing. Consider, for instance:
Premature commitment: constraints on the order of doing things.
Hidden dependencies: important links between entities are not visible.
Error-proneness: the notation invites mistakes and the system gives little protection.
and so on. They provide a sound reference against which the pros and cons of different notations can be evaluated.

The real boon (for me) is: some of those concepts can be extended from the tool to the material! Actual code may or may not have Hidden Dependencies. Actual code may have different degrees of Viscosity. Actual code may or may not have Closeness of Mapping. And so on. Of course, the notation itself is providing a bottom line on some properties. But ultimately, the form you give to your material may move quite far from the bottom line.
Indeed, I believe the form itself will be heavily influenced by the process and notation you used when you conceived the form. If you conceive your form using diagrammatic reasoning, it will keep some of the inherent properties of that notation even when it's translated into another notation (like source code). If you start with source code, your form will be more heavily influenced by the properties of source code.

I also like some of the candidate dimensions. For instance, I'm especially fond of Creative Ambiguity. This, I would say, it's one of the properties that code is lacking most. It's also something academics are trying hard to remove :-) from UML. And in a sense, it's what makes some practices like TDD so limited.
Code doesn't afford much creative ambiguity, yet what we call "modeling" should probably be called "incubation". Shaping a great form ain't easy. We need low viscosity, a good degree of creative ambiguity, high provisionality, and so on. Code alone just won't cut it.
Of course, we might not be after great form, just run-of-the-mill form. Then anything sensible is gonna work: see also my recent post on old-fashioned architectures "designed" for mass adoption.

Labels: article reference, design, form

posted by Carlo.Pescio @ 5:16 PM <permalink>  6 comments
bookmarking:    

My paper Listen to Your Tools and Materials (published in IEEE Software, September/October 2006) is now freely available on my website.

Labels: announce, article reference, design

In the Notes, Alexander defines form in a rather unique and interesting way: "The form is the part of the world over which we have control, and which we decide to shape while leaving the rest of the world as it is. The context is that part of the world which puts demands on this form; anything in the world that makes demands on the form is context. [...] We want to put the context and the form into effortless contact or frictionless coexistence" (pg. 18-19).

Alexander's definition goes well beyond the old-school, classical dichotomy between form and function, and also beyond the old-school dichotomy between functional and non-functional requirements. In fact, function intended as purpose is part of the context: functional requirements are obviously putting demands on form. Non-functional requirements are part of the context as well, and they should be considered first-class citizens too.

It is important to understand the human role in the design process. It is first an act of selection, and then an act of shaping.
We decide to shape a part of the world and leave the rest as it is. These are powerful words. I've long contended that in most case we have a much higher degree of control over requirements (context) than we're inclined to believe.
I've also contended for a very long time (mostly inspired by Tom Peters, I guess) that efficiency demands requirements (that is, context) to be developed jointly by developers and marketing.

Move away from this, and you get an inefficient context: a set of requirements (mostly functional) which may put strenuous demands on the form, without a corresponding premium in the resulting value. If you wanna win the game, you have to set the right rules.
In software development, more than in any other discipline, we can choose the right context before we step up to build our product. Of course, it is much easier to get a "fixed" set of requirements, complain about marketing lack of understanding, and start coding. But that's just not efficient. Also, keeping the context fixed while we shape the form is inefficient, as we wouldn't be profiting from newly discovered opportunities.

Building form, therefore, is an incremental process of discovery, as we don't know the exact context, and even if we do, we don't know if that's the context we wanna play with (or against, if you're the competitive kind ;-).
Indeed, Alexander reminds us that "In a real design project [...] we are searching for some kind of harmony between two intangibles: a form which we have not yet designed, and a context which we cannot properly describe" (pg. 26). It is not hard to see a deep connection with the notion of "software development as knowledge acquisition" popularized by Armour.

It is also important to understand a simple, yet deep consequence of Alexander's definition of form. As we create a product, even a software product, we are shaping its form. Even if we ignore design altogether, or even if we just concentrate on making it work, we are shaping some kind of form.
In most piecemeal development cases, that form will be determined by a small subset of forces: functional requirements, and sometimes the viscosity of previously developed software as well. The resulting structure won't be in frictionless contact with the true context, and the resulting friction will make our product brittle.

There is still a lot to be said about the subtle relationship between the above and agility, about the concept of shaping software, the [constructive] force field, and about the ubiquitous real options as well. Some more software-oriented examples are probably badly due too. And I should really say something about inventing requirements! So many things, so little time :-). Stay tuned!

Labels: design, form, requirements

posted by Carlo.Pescio @ 5:49 PM <permalink>  5 comments
bookmarking:    

I postponed writing on Form (vs. Function) for a while now, mostly because I was struggling to impose a rational structure on my thoughts. Form is a complex subject, barely discussed in the software design literature, with a few exception I'll quote when relevant, so I'm sort of charting new waters here.

The point is, doing so would take too much [free] time, delaying my writings for several more weeks or even months. This is exactly the opposite of what I want to do with this blog (see my post blogging as destructuring), so I'll commit the ultimate sin :-) and talk about form in a rather unstructured way. I'll start a random sequence of posting on form, writing down what I think is relevant, but without any particular order. I won't even define the concept of form in this first instalment.

Throughout these posts, I'll borrow extensively from a very interesting (and warmly suggested to any software designer) book from Christopher Alexander. Alexander is better known for his work on patterns, which ultimately inspired software designers and created the pattern movement. However, his most releavant work on form is Notes on the Synthesis of Form. When quoting, I'll try to remember page numbers, in which case, I'll refer to the paperback edition, which is easier to find.

Alexander defines two processes through which form can be obtained: the unselfconscious and the selfconscious process. I'll get back to these two concepts, but in a few words, the unselfconscious process is the way some ancient cultures proceeded, without an explicit set of principles, but relying instead on rigid tradition mediated by immediate, small scale adaptation upon failure. It's more complex than that, but let's keep it simple right now.

Tradition provides viscosity. Without tradition, and without explicit design principles, the byproducts of the unselfconscious process will quickly degenerate. However, merely repeating a form won't keep up with a changing environment. Yet change happens, maybe shortly after the form has been built. Here is where we need immediate action, correcting any failure using the materials at hand. Of course, small scale changes are sustainable only when the rate of change (or rate of failure) is slow.

Drawing parallels to software is easy, although subjective. Think of quick, continuous, small scale adaptation. The immediate software counterpart is, very naturally, refactoring. As soon as a bad smell emerge, you fix it. Refactoring is usually small scale, using the "materials at hand" (which I could roughly translate into "changing only a small fraction of code"). Refactoring, by definition, does not change the function of code. Therefore, it can only change its form.

Now, although some people in the XP/agile camp might disagree, refactoring is a viable solution only when the desired rate of change is slow, and only when the gap to fill is small. In other words, only when the overall architecture (or plain structure) is not challenged: maybe it's dictated by the J2EE way of doing things, or by the Company One True Way of doing things, or by the Model View Controller police, and so on. Truth is, without an overall architecture resisting change, a neverending sequence of small-scale refactoring may even have a negative large-scale impact.

I've recently said that we can't reasonably turn an old-style, client-server application into a modern web application by applying a sequence of small-scale changes. It would be, if not unfeasible, hardly economic, and the final architecture might be far from optimal. The gap is too big. We're expecting to complete a big change in a comparatively short time, hence the rate of change is too big. The viscosity of the previous solution will fight that change and prevent it from happening. We need to apply change at an higher granularity level, as the dynamics in the small are not the dynamics in the large.

Curiously enough (or maybe not :-), I'll be talking about refactoring the next two days. As usual, I'll try to strike a balance, and get often back to good design princples. After all, as we'll see, when the rate of change grows, and/or when the solution space grows, the unselfconscious process must be replaced by the selfconscious process.

Labels: book reference, design, form, profession

posted by Carlo.Pescio @ 7:25 PM <permalink>  8 comments
bookmarking:    

A fascinating topic in software development is the non-linear relationship between requirements and effort. A slight change in requirements can quickly unravel into a large, unexpected implementation effort. This is well-known among developers, and sadly unfamiliar to anyone who has not spent a significant time in the trenches of software development (including, unfortunately, a disproportionate part of upper management).

Early detection of decision points, where an innocent-looking choice can significantly alter the development schedule (or the probability of getting the software right) is even more fascinating. Even if the final decision rules in favor of the additional complexity (hopefully in exchange for greater benefits), it is important to understand the consequences of our choices.

Let's see a real-world example, although on a small scale.
A few weeks ago I was working on a top secret :-)) application, basically a rich client invoking web services on a remote server.
The users must authenticate before they can use the rich client, and it was decided that they will use their email address as the user id (instantly removing collisions), and a PIN instead of a user-chosen password.
The PIN is assigned by a server-side procedure when the user is registered, and users cannot change their PIN. This simplifies a few administrative issues, and also prevents the widely spread problem of password reuse (see, for instance, "The domino effect of password reuse" by Ives, Walsh, and Schneider, Communications of the ACM, April 2004).
At this point, the login screen could look somewhat like this:



Of course, assuming there is some support on the server side, this is so trivial to implement on the client side that you don't need to design anything. There is just no benefit; you can simply code your way through it. Any half-decent widget library natively supports a "password style" edit box which won't show what you're typing, so we're talking about a few lines of code here.

Now, the problem with a machine-generated PIN (everything has a downside!) is that it's harder to remember than a password chosen by the user. However, if they can't remember the PIN, users will write it down, compromising security again. We could argue that, instead of having people write it down, we could as well have the client computer write it down. The "remember me on this computer" checkmark of fame was then proposed (to be used only on computers considered "safe"):



All right, no need to design anything yet. It's just a checkmark, and when it's on, you save the email and PIN. Just remember to clear the storage if it's turned off. Well, ok, there is some chance to put a bug in here, but it's still quite simple, and most people, I guess, will just add some logic inside the form (or "login component", if they feel fancy about reuse), try a few test cases, and declare success.

Of course, storing a human-readable PIN on the client computer is quite lame. It's not much better than a post-it on the user's desk. Can we make the computer a little more helpful? Sure :-). We can store an encrypted PIN on the client computer. The problem is, the encryption key must be inside the rich-client, and this makes it vulnerable. Indeed, if the client computer can decode the encrypted PIN, it's just slightly harder (even for a naive hacker) to get hold of the decrypted PIN itself. Since the same PIN was used for authentication purposes on a related web site, that was deemed too risky.

Technology often comes to the rescue: we decided to use asymmetric encryption. The public key will be hard-coded inside the rich-client, but only the private key (deployed in our secure server), can decrypt the PIN. Good. That means we have to send the encrypted PIN to the server, whether we get it from the edit box (originally unencrypted) or from the local storage (already encrypted).

Now, this may seem like a tiny change upon storing the unencrypted PIN, especially when public key encryption is already available in standard libraries, so it seems that we just need to add a few function calls, and that would be it.
Well, if you think so, think twice :-), as you should catch the glimpse of a non-linear increase in complexity. Let's see why.

When the program opens the authentication screen it must now check if you have the email and [encrypted] PIN on local storage. If you do, it will display the email and... well, it can't just set the encrypted PIN in the corresponding text box, for several reasons, like:
- the encrypted PIN has a different length than the unencrypted PIN
- you usually get the unencrypted PIN from the textbox and crypt it, and you don't want to encrypt it twice.
So you probably want to set a dummy string there.

Now, the user may (for instance) turn off and back on the "remember me" checkmark, at which point you still have to use the encrypted PIN on file. But if the user changes PIN, while leaving the checkmark on (possibly turning it on and off any number of times), you have to read an unencrypted PIN from the textbox, encrypt it, authenticate, and save the new email + encrypted PIN in a local storage. If the checkmark is just turned off, but it was initially on, you have to use the encrypted PIN on file, but clear the local storage. But not if it was initially off. And so on: there are a quite a few different scenarios. Which leads to the second part of this post: modeling and correctness by design.

When faced with a sudden increase in complexity, you can basically follow three paths:
1) [recommended] Negotiate this complexity away. In this case, we didn't want to, as it provided a significant benefit to the user.
2) [suicidal] Ignore it: just work as you would do if that additional complexity wasn't there. In many cases, that means: just code your way through it.
3) [professional] Deal with it. Make it manageable, remove any accidental complexity (as Brooks would say), leave only the essential complexity on the table, shape and represent your problem in a way that is amenable to reasoning.

Now, I know from experience that, faced with the above, quite a few people would go for (2). Maybe they don't see the complexity surge. Maybe they think code is the only important artifact. Maybe they just don't know any better. But they will just add code inside that form, test a little, find bugs, patch it, find more bugs, and so on. In the end, they will release a faulty program, because they didn't try all the scenarios, and by focusing on single cases they missed the big picture.

I'm really curious about how the TDD guys would approach this problem (more on this later), but I can tell you how I did. There is obviously a state machine behind the problem. So my best shot was to define the precise behavior of that state machine. It's not a trivial on-off machine, so I needed a representation which helps me think. I could use a state/event table, or I could use a diagram. I choose a diagram, for reasons I'll discuss shortly.

Now, I must admit that it took me more than I expected to get it right on paper. I can blame my tool (the CASE tool we use on this project is based on UML 1.4, and we had to tweak it more than a bit to adopt some useful UML 2 modeling concepts) but it was largely a learning process. When I started, I felt like in this specific case superstates were the key to keep the diagram simple, so I sketched a model based on superstates. It didn't look nice (read: obviously correct, easy to understand and to implement).
I started moving some concepts from superstates to concurrent (orthogonal) states. In the end I killed the superstates, paying a small price in increased transitions, but the final model was reasonably simple. At the very least, it shows the real complexity of the problem. Here it is: overall, it took me something like two and a half hours to nail it down in this form.



You may want to think a little about the problem and take a look at the diagram (hey, you might even find a bug :-). You'll see that the model is relatively abstract - it doesn't deal with the GUI side at all, and it shouldn't. There is no attempt to capture keystrokes or mouse clicks. What is important at this abstraction level is that the PIN or email has been changed, or that the checkmark has been turned on or off. Everything else belongs elsewhere (in the GUI components, in the authentication form). Creating a model at the right abstraction level is fundamental, as it allows you to concentrate on the important issues, while at the same time shielding the model itself from irrelevant changes (more on this later).

Part of those 2.5 hours went into shaping the diagram. Shaping is not the same as drawing. In a sense, shaping is to drawing as architecture is to structure. Shaping is an intentional activity, directed at revealing the fundamental nature of what we're modeling. Indeed, the biggest difference between a visual model and a state/transition table is that a table won't give you any opportunity for shaping.

Now, look at that diagram again. Can you see the symmetry in shape and the antisymmetry in meaning/behavior? Can you relate that to the problem we're trying to solve? Do the crossing of transitions 6 and 7 further reinforce your understanding of the strict relationship between antisymmetry and behavior? Do you see how information is lost as you move left-to-right in transitions 6 and 7, so that you cannot get back, while it is preserved in transitions 4-5 and 8-9, so you can always get back? Also, see that anomaly on the bottom left side? The shape is no longer symmetric there. Why? Is that right? Can this be avoided? At what cost?
And so on. There is a lot of reasoning that can be suggested and simplified by the right shape. In the end, when the shape of the model closely resembles the shape of the problem, you can basically see that the model is right.
Remember: code is a model too, with the extremely useful property of being executable. However, you just can't apply the same kind of reasoning to code. It is not the right model to think about the global picture.

Ok, time to move to coding. I must confess I usually don't code simple stuff like this. I would usually leave this to someone else, someone who as been working with me on the model. However, having spent so much time on the model, I wanted to see first-hand if my feeling was right: coding should now take very little time, yield relatively few lines, and no bugs, as it was designed for correctness. So I implemented this thing myself.

Now, this may surprise you, but the problem didn't suggest me any fancy implementation. No State pattern, no table-driven state machine implementation. I used only one class, which I named Credentials (no, AuthenticationManager is not a good name :-), so no class diagram this time.
Internally, Credentials uses three enum types to model the orthogonal states. I simply coded the state-dependent methods with an infamous switch/case.
Why? Quite simple! I do not expect the kind of fine-grained changes that more sophisticated implementations can handle gracefully. The State pattern is about extendibility of states. A table-driven implementation makes event re-routing easy. But I do not foresee any need for this.
My estimate is that this code will stay untouched with a 99% probability. The remaining 1% represents a potential for disruptive change (like, we use some authentication device), at which point, in this case, no sophisticated implementation would shield this code from scrapping. Anyway, it's a very small class, so no big deal (excluding braces, empty lines and a few comments, it took me 50 lines of code, including the hard-coded public key and a call to the encryption library). More on this later.

Being so short, it took very little to implement. I didn't keep track, but I would say less than half an hour, including some initial testing. I still managed to put a bug in it. Well, actually not in the state machine itself, but in the glue code inside the form: I called an event when I should have called another (ok ok I'm pretty dumb). Of course, the bug wasn't subtle, and I caught it immediately . No more bugs. Game over.

Now, take a look at the state machine again. All those numbers in red are not strictly part of UML, but they're extremely useful to define scenarios and test cases. For instance, a scenario where the user starts up the application for the first time, types the email and pin, sets the "remember me" checkmark and authenticates successfully can be simply represented as "0-2-1-9-15-22". If, next time, he turns the checkmark off and then back on, but doesn't change a thing and still authenticates successfully, the scenario would be "0-3-5-4-10-20", and so on.
Which leads us to the next subject: testing.

How many interesting test cases do we have here? Quite a few, and you can quickly derive a list of significant case by "navigating" the diagram. Remember to include cases where the user makes some mistake, fails to authenticate, and then corrects himself (or cancel). That is, something like "0-2-13-14-13-23" to get a short one. If you look carefully, you'll find at least a couple of dozen interesting scenarios, possibly more, depending on how much testing you think is enough.

Now, what is the TDD guy supposed to do? Simple: shun the diagram, think about a scenario, make it fail, make it work. Think about the next scenario, and so on. All code-centered. Chance to get it right and clean? Sorry, I think it's quite small. See my previous posts on the XP episode for further evidence.
Time for another confession:I didn't test all the scenarios myself. We have professional testers on this project. Time ago, I trained them on testing state-based software. They can derive a sensible test plan from the state diagram (plus their domain knowledge and understanding of the problem, of course), execute it, report bugs. Sure, it is my responsibility to give them working code, which I did :-), as they didn't find any problem.
We didn't write automatic tests. It could be done - especially because I have a Credentials class decoupled from GUI code. But again, there would be little value on those automatic tests, for the very same reasons that brought me to choose a simple switch/case implementation. Which leads to the final point: change management.

The common tenets of code-centric practices is that if you really want to sketch a diagram you're free to do it (thanks guys :-), but once you get your code working, you must scrap the diagram, as maintenance costs would raise if you don't. Context-unaware suggestions like this are always short-sighted.
You basically touch your code for one of these 3 reasons:
1) you have a bug to fix
2) design (but not requirements) change - let's simplify this to refactoring.
3) requirements change
Let's consider each one in context, as true agility cannot be achieved by applying cookbook recipes.

Say you have a bug to fix. Some scenario is not behaving as intended. Would you rather:
a) play the faulty scenario on the diagram above, understand if it's truly a defect, understand if it is a design defect (the diagram is wrong) or a coding defect (the diagram is right but the code is not behaving properly), at which point you probably know exactly which transition or state is misbehaving, go there and fix it, without losing sight of the global picture.
b) start your debugger and just sift through the code, fix it locally, and rely on a hopefully large set of test cases to guarantee you haven't broken anything.
Dunno about you, but I'll choose (a) in this case - I've seen way too many "simple" systems like this fail for lack of global understanding.
Note that if you expect bugs, you may want to invest on a testing suite. I didn't, so I didn't.

Say you want to refactor your code. Maybe you decided, for some pervert :-) reason, to adopt the State pattern. Fine. Guess what, I choose to draw only a state (not class) diagram exactly because the structural side was too simple to benefit from modeling. Refactor all you want, the state chart above will stay valid. You could even squeeze a little more value out of it, by using it to reason about the new structure you're going to impose on your software - how will you handle concurrent states, and so on. No need to scrap the diagram.

Say requirements change. Good. The change can be of a rather large magnitude, like you're no longer using a PIN but an authentication device. The diagram is now largely useless - although you might want to take some inspiration as you draw a new one. Truth is, considering the abstraction level of this diagram, if the diagram gets useless, you ain't gonna save a single line of code either. So if you scrap the diagram, you scrap the code as well.
But maybe the change is on a smaller scale, like: "when the checkmark is on, you cannot edit fields. You have to turn it off before you can change the email or PIN" (this might even make sense, to prevent accidental modifications). Again, what would you rather do:
a) understand the impact of the change on the diagram; change it to follow the new requirements. Understand correctness on the diagram, as I suggested above. See which states and transitions need to change. Go to your code and change it. Test again.
b) open your editor, look at your test code, change it to follow new requirements, hope you got enough test cases (and removed the ones no longer valid), start patching the old code, refactor as you go to keep it in a decent shape.
Again, dunno about you, but I'll choose (a) in this case - if you benefit from diagrammatic reasoning when you first think about the system, you'll benefit from diagrammatic reasoning when any significant change arise.

Ok, this is a very long post, so I'll cut it short. You can create software expecting bugs, or expecting correctness. Just be careful about the self-fulfilling expectation you choose :-).

Labels: article reference, design

posted by Carlo.Pescio @ 3:39 PM <permalink>  12 comments
bookmarking:    

The past two weeks have been hectic to say the least, and I had no time to conclude this quadrilogy. Fear not :-), however, because here I am. Anyway,I hope you used the extra time to read comments and answers to the previous posts, especially Zibibbo's comments; his contribution has been extremely valuable.
Indeed, I've left quite a few interesting topics hanging. I'll deal with the small stuff first.

Code [and quality]:
I already mentioned that I'm not always so line-conscious when I write code. Among other things, in real life I would have used assertions more liberally.
For instance, there is an implicit contract between Frame and Game. Frame is assuming that Game won't call its Throw method if the frame IsComplete. Of course, Game is currently respecting the contract. Still, it would be good to make the contract explicit in code. An assertion would do just fine.
I remember someone saying that with TDD, you don't need assertions anymore. I find this concept naive. Assertions like the above would help to locate defects (not just detect them) during development, or to highlight maintenance changes that have violated some internal assumption of existing code. Test cases aren't always the most efficient way to document the system. We have several tools, we should use them wisely, not religiously.

When is visual modeling not helpful?
In my experience, there are times when a model just won't cut it.
The most frequent case is when you're dealing with untried (or even unstable) technology. In this context, the wise thing to do is write code first. Code that is intended to familiarize yourself with the technology, even to break it, so that you know what you can safely do, and what you shouldn't do at all.
I consider this learning phase part of the design itself, because the ultimate result (unless you're just hacking your way through it) is just abstract knowledge, not executable knowledge: the code is probably too crappy to be kept, at least by my standards.
Still, the knowledge you acquired will shape the ultimate design. Indeed, once you possess enough knowledge, you can start a more intentional design process. Here modeling may have a role, or not, depending on the scale of what you're doing.
There are certainly many other cases where starting with a UML model is not the right thing to do. For instance, if you're under severe resource constraints (memory, timing, etc), you better do your homework and understand what you need at the code level before you move up to abstract modeling. If you are unsure about how your system will scale under severe loading, you can still use some modeling techniques (see, for instance, my More on Quantitative Design Methods post), but you need some realistic code to start with, and I'm not talking about UML models anyway.
So, again, the process you choose must be a good fit for your problem, your knowledge, your people. Restraining yourself to a small set of code-centric practices doesn't look that smart.

What about a Bowling Framework?
This is not really something that can be discussed briefly, so I'll have to postpone some more elaborated thoughts for another post. Guess I'll merge them with all the "Form Vs. Function " stuff I've been hinting to all this time.
Leaving the Ball and Lane aside for a while, a small but significant improvement would be to use Factory Method on Game, basically making Game an abstract class. That would allow (e.g.) regular bowling to create 9 Frames and 1 FinalFrame, 3-6-9 bowling to change the 3rd, 6th and 9th frame to instances of a (new) StrikeFrame class, and so on.
Note that this is a simple refactoring of the existing code/design (in this sense, the existing design has been consciously underengineered, to the point where making it better is simple). Inded, there is an important, trivial and at the same time deep reason why this refactoring is easy. I'll cover that while talking about options.
If you try to implement 3-6-9 bowling, you might also find it useful to change
if( frames[ currentFrame ].IsComplete() )
   ++currentFrame;
into
while( frames[ currentFrame ].IsComplete() )
   ++currentFrame;
but this is not strictly necessary, depending on your specific implementation.
There are also a few more changes that would make the framework better: for instance, as I mentioned in my previous post, moving MaxBalls from a constant to a virtual function would allow the base class to instantiate the thrownBalls array even if the creational responsibility for frames were moved to the derived class (using Factory Method).
Finally, the good Zibibbo posted also a solution based on a state machine concept. Now, it's pretty obvious that bowling is indeed based on a state machine, but curiously enough, I wouldn't try to base the framework on a generalized state machine. I'll have to save this for another post (again, I suspect, Form Vs. Function), but shortly, in this specific case (and in quite a few more) I'd try to deal with variations by providing a structure where variations can be plugged in, instead of playing the functional abstraction card. More on this another time.

Procedural complexity Vs. Structural complexity
Several people have been (and still are) taught to program procedurally. In some disciplines, like scientific and engineering computing, this is often still considered the way to go. You get your matrix code right :-), and above that, it's naked algorithms all around.
When you program this way (been there, done that) you develop the ability to pour through long functions, calling other functions, calling other functions, and overall "see" what is going on. You learn to understand the dynamics of mutually recursive function. You learn to keep track of side effects on global variables. You learn to pay attention to side effects on the parameters you're passing around. And so on.
In short, you learn to deal with procedural complexity. Procedural complexity is best dealt with at the code level, as you need to master several tiny details that can only be faithfully represented in code.
People who have truly absorbed what objects are about tend to move some of this complexity on the structural side. They use shape to simplify procedures.
While the only shape you can give to procedural code is basically a tree (with the exception of [mutually] recursive functions, and ignoring function pointers), OO software is a different material, which can be shaped in a more complex collaboration graph.
This graph is best dealt with visually, because you're not interested in the tiny details of the small methods, but in getting the collaboration right, the shape right (where "right" is, again, highly contextual), even the holes right (that is, what is not in the diagram can be important as well).
Dealing with structural complexity requires a different set of skills and tools. Note that people can be good at dealing with procedural and with structural complexity; it's just a matter of learning.
As usual, good design is about balance between this two forms of complexity. More on this another time :-).

Balls, Real Options, and some Big Stuff.
Real Options are still cool in project management, and in the past few years software development has taken notice. Unfortunately, most papers on software and real options tend to fall in one of these two categories:
- the mathematically heavy with little practical advice.
- the agile advocacy with the rather narrow view that options are just about waiting.
Now, in a paper I've referenced elsewhere, Avi Kamara put it right in a few words. The problem with the old-fashioned economic theory is the assumption that the investment is an all or nothing, now or never, project. Real options challenge that view, by becoming aware of the costs and benefits of doing or not doing things, build flexibility and take advantage of opportunities over time (italics are quotes from Kamara).
Now, this seems to be like a recipe for agile development. And indeed it is, once we break the (wrong) equation agile = code centric, or agile = YAGNI. Let's look a little deeper.
Options don't come out of nowhere. They came either from the outside, or from the inside. Options coming from the outside are new market opportunities, emerging users's requests or feedback, new technologies, and so on. Of course, sticking to a plan (or a Big Upfront Design) and ignoring those options wouldn't be smart (it's not really a matter of being agile, but to be economically competent or not).
Options come also from the inside. If you build your software upon platform-specific technologies, you won't have an option to expand into a different platform. If you don't build an option for growth inside your software, you just won't have the option to grow. Indeed, it is widely acknowledged in the (good) literature that options have a cost (the option premium). You pay that cost because it provides you with an option. You don't want to make the full investment now (the "all" in "all or nothing") but if you just do nothing, you won't get the option either.
The key, of course, is that the option premium should be small. Also, the exercise cost should be reasonable as well (exercise price, or strike price, is what you pay to actually exercise your option). Note: for those interested in these ideas, the best book I've found so far on real option is "Real options analysis" by Johnathan Mun, Wiley finance series)

Now, we can begin to see the role of careful design in building the right options inside software, and why YAGNI is an oversimplified strategy.
In a previous post I mentioned how a class Lane could be a useful abstraction for a more realistic bowling scorer. The lane would have a sensor in the foul line and in the gutters. This could be useful for some variation of bowling, like Low Ball (look under "Special Games"). So, should I invest into a Lane class I don't really need right now, because it might be useful in the future? Wait, there is more :-).
If you consider 5 pin Bowling, you'll see that different pins are awarded different scores. Yeap. You can no longer assume "number of pins = score". If you think about it, there is just no natural place in the XP episode code to put this kind of knowledge. They went for then "nothing" side of the investment. Bad luck? Well guys, that's just YAGNI in the real world. Zero premium, but potentially high exercise cost.
Now, consider this: a well-placed, under-engineered class can be seen as an option with a very low premium and very low exercise cost. Consider my Ball class. As it stands now, it does precious nothing (therefore, the premium cost was small). However, the Ball class can easily be turned into a full-fledged calculator of the Ball score. It could easily talk to a Lane class if that's useful - the power of modularity allows the rest of my design to blissfully ignore the way Ball gets to know the score. I might even have a hierarchy of Ball classes for different games (well, most likely, I would).
Of course, there would be some small changes here and there. I would have to break the Game interface, that takes a score and builds a Ball. I used that trick to keep the interface compatible with the XP episode code and harvest their test cases, so I don't really mind :-). I would also have to turn the public hitPins member into something else, but here is the power of names: they make it easier to replace a concept, especially in a refactoring-aware IDE.
Bottom line: by introducing a seemingly useless Ball class, I paid a tiny premium cost to secure myself a very low exercise cost, in case I needed to support different kinds of bowling. I didn't go for the "all" investment (a full-blown bowling framework), but I didn't go for the "nothing" either. That's applied real option theory; no babbling about being agile will get you there. The right amount of under-engineering, just like the right amount of over-engineering, comes only from reasoning, not from blindly applying oversimplified concepts like YAGNI.
One more thing about options: in the Bowling Framework section, I mentioned that refactoring my design by applying Factory Method to Game was a trivial refactoring. The reason it was trivial, of course, is that Game is a named entity. You find it, refactor it, and it's done. Unnamed entities (like literals) are more troublesome. Here is the scoop: primitive types are almost like unnamed entities. If you keep your score into an integer, and you have integers everywhere (indexes, counters, whatever), you can't easily refactor your code, because not any integer is a score. That was the idea behind Information Hiding. Some people got it, some didn't. Choose your teacher wisely :-).

A few conclusions
As I said, there are definitely times when modeling makes little sense. There are also times when it's really useful. Along the same lines, there are times when requirements are relatively unknown, or when nobody can define what "success" really means before they see it. There are also times when requirements are largely well-known and (dare I say it!) largely stable.
If you were asked to implement an automatic scoring system for 10 pin bowling and 3-6-9 bowling and 9 pin bowling, would you really start coding the way they did in the XP episode, and then slowly change it to something better?
Would you go YAGNI, even though you perfectly know that you ARE gonna need extensibility? Or would you rather spend a few more minutes on the drawing board, and come up with something like I did?
Do you really believe you can release a working 3-6-9 and a working 9-pin faster using the XP episode code than mine? Is it smart to behave as if requirements were unknown when they're, in fact, largely known? Is it smart not to exploit knowledge you already have? What do you consider more agile, that is, adaptive to the environment?
Now, just to provoke a reaction from a friend of mine (who, most likely, is somewhere having fun and not reading my blog anyway :-). You might remember Jurassik Park, and how Malcom explained chaos theory to Ellie by dropping some water on her hand (if you don't, here is the script, look for "38" inside; and no, I'm not a fan :-).
The drops took completely different paths, because of small scale changes (although not really small scale for a drop, but anyway). Some people contend that requirements are just as chaotic, and that minor changes in the environment will have a gigantic impact on your code anyway, so why bother with upfront design.
Well, I could contend that my upfront design led to code better equipped to deal with change than the XP episode did, but I won't. Because you know, the drops did take a completely different path because of small scale changes. But a very predictable thing happened: they both went down. Gravity didn't behave chaotically.
Good designers learn to see the underlying order inside chaos, or more exactly, predominant forces that won't be affected by environmental conditions. It's not a matter of reading a crystal ball. It's a matter of learning, experience, and well, talent. Of course, I'm not saying that there are always predominant, stable forces; just that, in several cases, there are. Context is the key. Just as ignoring gravity ain't safe, ignoring context ain't agile.

Labels: agile, book reference, design, real options

posted by Carlo.Pescio @ 6:24 PM <permalink>  1 comments
bookmarking:    

In the last few months I've been repeating a mantra (also in this blog): design is always highly contextual.
So, before taking a look at what I did, it's important to consider why I did it. It's also important to conside how I did it, as to put everything in perspective.

Why
- I use models quite often, but I'm not a visual modeling zealot. I know, from experience, that visual modeling can be extremely valuable in some cases, and totally useless in others (more on this next time). I can usually tell when visual modeling can be useful, using a mixture of experience, intuition, and probably some systematic rules I never took the time to write down.
When I first read the XP episode, I've been negatively impressed by the final results, and intuitively felt that I could get something better by not rushing into coding. So a first reason was to prove that point (to myself first :-).

- I aimed to remove all the bad smells I've identified in their code, without introducing others. I also wanted to prove a point: that a careful designed solution didn't have to be a code monster, as implied elsewhere. I wanted the final result to have basically the same number of rows as they had. Maybe less. Therefore, I wrote the code in a line-conscious way.

- I wanted the comparison to be fair. Therefore, I did not aim for the super-duper, ultimate bowling framework. I just aimed for a better design and better code, through a different process. I'll talk a little about what it would take to turn my design into a true "bowling framework" in my next post.

- Actually, I found the idea of not writing the ultimate bowling framework extremely useful. There has been a lot of babbling about Real Options in software development in the last few years, and I even mention a few concepts in my project management course.
However, I lacked a simple example where I could explain a few concepts better. By (consciously :-) under-engineering some parts of the design, I got just the example I needed (more on this next time).

- I also wanted to check how many bugs I could possibly put in my code without using TDD. I would just write the code, run all their tests, and see how many bugs I had to fix before I got a clean run.

How
I must confess I procrastinated this stuff for quite a while. In some way, it didn't feel totally right to criticize other people's work like I had to do. But after all, by writing this I'll open my work to critics as well. So one evening, at about 10 PM (yeap :-), I printed out the wikipedia page on bowling, fired up a CASE tool (just for drawing), and got busy.

Unfortunately, I can't offer you a realistic transcript of what I thought all along. When you really use visual modeling, you're engaged in a reflective conversation with your material (see my paper, "Listen to your tools and materials"; I should really get a pre-publishing version online). You're in the flow, and some reasoning that will take a while to explain here can take just a few seconds in real time.

Anyway, I can offer a few distinct reflections that took place as I was reading the wikipedia page and (simultaneously) designing my solution. Note that I entirely skipped domain modeling; the domain was quite simple, and the wikipedia page precise enough. Also, note that I felt more comfortable (coming from 0-knowledge of bowling) reading wikipedia, not reverse engineering requirements from the XP episode. That might be my own personal bias.

I immediately saw quite a few candidate classes: the game itself, the player, the pins, the lane, the frame, the ball. Bertrand Meyer said that classes "are here for picking", and in this case he was quite right.
I never considered the throw as a class though. Actually, "throw" can be seen as a noun or as a verb. In this context, it looked more like a verb: you throw the ball and hit pins. Hmmm, guess what, they didn't look at it this way in the XP episode; they sketched a diagram with a Throw class and rushed into coding.

Rejecting classes is not easy without requirements, even informal ones. For instance, if I have to design a real automatic scoring system, there will be a sensor in the lane, to detect the player crossing the foul line. There will also be the concept of foul ball, which was ignored in the XP episode. So (see above, "fair comparison") I decided that my "requirements" were to create a system with the same capabilities as they did. No connection to sensors, no foul ball. But, I wanted my design to be easily extended to include that stuff. So I left out the Lane class (which could also encapsulate any additional sensor, like one in the gutter). But I wanted a Ball class; more on this later.

Game is an obvious class: we need a starting point, and also a sort of container for the Frames (see below) and for Balls. The Game has a need to know when the current frame is completed (all allowed balls thrown, or strike), so it can move to the next frame. But while advancing to the next frame is clearly its own responsibility, knowledge of frame completion is not (see below).

The Frame looked like a very promising abstraction. If you look at the throwing and scoring rules, it's pretty obvious that in 10-pin bowling you have 9 "normal" frames and a 10th "non standard" frame, where up to 3 balls can be thrown, as there is the concept of bonus ball (hence the imprecise domain model in the XP episode, where they just wrote 0..3 throws in a frame). Also in 3-6-9 bowling you have "special" frames, and so on.
Does this distinction between standard and nonstandard frames ring the bell of inheritance? Good. Now if you want the frame to be polymorphic, you gotta put some behaviour in it. Of course you can do that by reasoning upon the diagram: you don't need to write test code to discover that hey, you dunno what method to put in a Frame class.
Anyway, they may not know, but I do :-). The Frame can then tell the Game if it is completed, shielding the Game from the difference between a standard or last frame (or "strike frames" in 3-6-9 bowling, for instance). To do so, the Frame must know about any Ball that has been thrown in that frame, so it may also have a Throw method. The Frame should also calculate its own score. Wait... that's something the Frame can't do on its own.

In fact, the two authors have been smart enough to pick a problem where trivial encapsulation won't do it. The Frame needs to know about balls that might have been thrown in other frames to calculate its score. Now, there are several ways to let it know about them, and I did consider a few, but for brevity, here is what I did (the simplest thing): the Score method in class Frame takes a list of Balls as a parameter; the Game keeps a list of Balls and passes it to Frame. The Frame just keeps track of the first Ball thrown in the frame itself. From that knowledge, it's trivial to calculate the score. The LastFrame class is just slightly different, as the completion criteria changes to account for bonus balls; I also have to redefine the Throw method, again to account for bonus balls.

So what is the real need for a Ball class? Well, right now, it keeps track of how many pins were hit, and its own index in the Balls list. However, it's also a cheap option for growth. More on this next time, where I'll talk about the difference between having even a simple, non-encapsulated class like Ball, and using primitive types like integers.

What
In the end I got something like this (except I had no Status: in the beginning, I put a couple of booleans for Strike and Spare).



At that point, I decided to move to coding. In the XP episode the selected language is Java, but I don't like the letter J much :-) so I used C#. Given the similarity of the languages, the difference is irrelevant anyway. I didn't even want to wait for Visual Studio to come up, so I just started my friendly Notepad and typed the code without much help from the editor :-). Classes and methods were just a few lines long anyway. At that point it was 11 PM, and I called it a day. (So all this stuff took about 1 hour, between BUFD :->> and coding).

Next morning I was traveling, so I started Visual Studio, imported the code, corrected quite a few typos to make it compile, then imported the test cases from the XP episode, converted them to the NUnit syntax, and hit the run button. Quite a few failed. I had a stupid bug in LastFrame.Throw. I fixed the bug. All tests succeeded. While playing with the code, I saw an opportunity to make it shorter by adding a class (an enum actually) that could model the frame state. I did so in the code, as the change was simple and neat. I also brought back the change in the diagram, which took me like 10 seconds (too much for the XP guys, I guess). I run the tests, and they failed, exactly like before :-).
I could blame the editor and the autocompletion, but in LastFrame.Throw I had something like

if( status != Status.Spare && status != Status.Spare )

instead of

if( status != Status.Strike && status != Status.Spare )

Ok, I'm pretty dumb :-), but anyway, I fixed the bug and got a green bar again.

That was it. Well actually it wasn't. In my initial design I had a MaxBalls virtual method in Frame. That allowed for some more polymorphism in Game. In the end I opted for a constant to make the code a few line shorter, since the agile guys seem so concerned about LOC (again, more on this next time). I'm not really so line-conscious in my everyday programming, so I would have usually left the virtual method there (no YAGNI here; or maybe I could restate is You ARE Gonna Need It :-)).

So, here is the code. Guess what, if you remove empty lines, it's a few line shorter than their code. If you remove also brace-only lines, it's a few lines longer. So we could call it even. Except it's just much better :-).

Ok, this is like the longest post ever. Next time I'll talk a little about the readability of the code, give some examples of how this design/code could be easily changed to (e.g.) 3-6-9 bowling, digress on structural Vs. procedural complexity, tinker a little with real options and under/over engineering, and overall draw some conclusions.


internal class Ball
  {
  public Ball( int pins, int index )
    {
    hitPins = pins;
    indexInGame = index;
    }

  public int hitPins;
  public int indexInGame;
  }

internal class Frame
  {
  public const int MaxBalls = 2;

  public virtual bool IsComplete()
    {
    return status != Status.Incomplete ;
    }

  public int Score( Ball[] thrownBalls )
    {
    int score = 0;
    if( IsComplete() )
      {
      score = firstBall.hitPins + thrownBalls[ firstBall.indexInGame + 1 ].hitPins;
      if( status == Status.Strike || status == Status.Spare )
        score += thrownBalls[ firstBall.indexInGame + 2 ].hitPins;
      }
    return score;
    }

  public virtual void Throw( Ball b )
    {
    const int totalPins = 10;
    if( firstBall == null )
      firstBall = b;
    if( b == firstBall && b.hitPins == totalPins )
      status = Status.Strike;
    else if( b != firstBall && firstBall.hitPins + b.hitPins == totalPins )
      status = Status.Spare;
    else if( b != firstBall )
      status = Status.Complete;
    }

  private Ball firstBall;
  protected Status status;
  protected enum Status { Incomplete, Spare, Strike, Complete } ;
  }

internal class LastFrame : Frame
  {
  public new const int MaxBalls = Frame.MaxBalls + 1;

  public override bool IsComplete()
    {
      return ( status == Status.Strike && bonusBalls == 2 ) ||
             ( status == Status.Spare && bonusBalls == 1 ) ||
             ( status == Status.Complete );
    }

  public override void Throw( Ball b )
    {
    if( status != Status.Strike && status != Status.Spare )
      base.Throw( b );
    else
      ++bonusBalls;
    }

  private int bonusBalls;
  }

public class Game
  {
  public Game()
    {
    const int MaxFrames = 10;
    frames = new Frame[ MaxFrames ];
    for( int i = 0; i < MaxFrames - 1; ++i )
      frames[ i ] = new Frame();
    frames[ MaxFrames - 1 ] = new LastFrame();
    int maxBalls = (MaxFrames-1)*Frame.MaxBalls + LastFrame.MaxBalls;
    thrownBalls = new Ball[ maxBalls ];
    }

  public void Throw( int pins )
    {
    if( frames[ currentFrame ].IsComplete() )
      ++currentFrame;
    Ball b = new Ball( pins, currentBall );
    frames[ currentFrame ].Throw( b );
    thrownBalls[ currentBall++ ] = b;
    }

  public int Score()
    {
    int score = 0;
    foreach( Frame f in frames )
      score += f.Score( thrownBalls );
    return score;
    }

  private Ball[] thrownBalls;
  private Frame[] frames;
  private int currentFrame;
  private int currentBall;
  }

Labels: agile, design

posted by Carlo.Pescio @ 5:46 PM <permalink>  18 comments
bookmarking:    

I hope you had some time to read the XP episode paper and ponder a little on what they managed to get in the end. As I've anticipated, I do I have a few issues with the results. It's not a matter of correctness: they pass all the tests. It's a matter of quality.
The design is questionable, and even the code is questionable. No big deal: after all, it's a little more than a toy example. It gets slightly worse when you consider all this stuff in perspective (at the end of the XP game, code is all you get; I'll get back to this later). Right now, here are the main flaws I see:

- overabundance of numerical literals.
This is programming 101: literals are bad (although they keep your code a few line shorter). Go over the code. You'll see numbers like 10 just about everywhere. Unfortunately, in bowling you got 10 pins and also 10 frames. You always have to read carefully to understand which is which. Now go back to the wikipedia page on bowling. See the mention of 5-pin bowling, 9-pin bowling, etc? Just go ahead and change the code to use 9 pins and 10 frames(9-pin bowling ain't that simple, but anyway). Guess what, none of the test cases will help, and it's not a one-line change as it ought to be. There is even a 21 in that code, being 10 * 2 + 1. I'll let you figure what that 10, 2, and 1 are :-).

- programming against an implementation, not against the problem
Some portion of the code are indeed quite good:

 public int scoreForFrame(int theFrame)
  {
    ball = 0;
    int score=0;
    for (int currentFrame = 0; 
         currentFrame < theFrame; 
         currentFrame++)
    {<