Sunday, March 25, 2007
Design: lean, but upfront
What is remarkable about this project is that the design model bears very little resemblance to the analysis model. This should not come as a surprise: the analysis and design models are created for different reasons, with different goals, and shaped by different forces.
The analysis model (or, more appropriately in this case, the problem domain model) is created to encode our understanding of the problem domain. Therefore, major goals are the accuracy, a sufficient completeness, close adoption of problem domain terminology. We need a strong focus on capturing essential complexity and on leaving outside any accidental complexity; as part of this, we need to focus on keeping any element of possible solutions out of the way.
The design model is created to reason about a "good" solution, where "good" can be informally defined as "solving the problem while maintaining the right balance among all the non-functional requirements, the various design principles, all the non-technical issues like who's gonna implement the different portions of the system, and so on".
In this case, we had some large uncertainties, as some business rules were not crystal-clear, some subsystems with which we should integrate were not finished yet, and so on (business as usual). However, we agreed that the underlying domain was relatively stable, so we expect that after an initial instability, our system will be relatively stable. Therefore, there was no need to push the envelope on extendibility.
However, there was a lot to gain by moving to a meta-level description of some domain objects. A lot of structural properties could be easily captured that way, avoiding the creation of several custom classes to represent domain object. Also, by trying to move to a meta-level description, we gained a better understanding of some non-trivial, history-related issues (it's not unusual for design to clarify some analysis problems). We didn't try to capture behaviour at meta-level: instead, we defined a few hot-spots where behaviour can be easily plugged. The nice part is that in the end most classes will be very small, behavioral plug-ins will have a single concern and will be easy to develop and test without having a complete knowledge of the system (this will certainly help as the implementation will be offshored).
Now, what about those analysis models? Has the time spent on those diagram been wasted? I believe the time spent on those model has been very well spent. They helped the project leader keeping track of his understanding of the problem. More than that, the knowledge encoded in those diagram can be quickly shared with others. I didn't do any analysis: I've been given the models, and I've been asked to participate into the design. Sure, I knew some of those concepts from previous experience, but reading the models helped a lot, in a short time. I expect anyone new to the project will get acquainted more easily by reading models that are very close to reality (while the design models are, in this case, more about a meta-reality).
Looking at the design models, it's hard to say that you could get there starting with a trivial design (most likely, closely inspired by the analysis models, or even by a small portion of them, informally captured as nouns in some use case or user's story) and working your way out by refactoring. The architectural difference is too big. With a strong focus on code, it's hard to see the forest for the trees. You'll end up with an accidental architecture (or accidental structure, as I like to call it) instead of an intentional architecture. But that's the usual debate. What is interesting in this case is that we did very little overengineering to protect ourselves from change. Actually, we did almost nothing: the same behavioral hot-spots that we used to complete the meta-description are also perfect for extending or changing behaviour at a fine granularity. We designed the system for simplicity, high decoupling between classes, good separation of concerns, adequate performance, simple piecemeal implementation. Several other good properties just followed. So this is not the dreaded "big upfront design out of fear of change". Indeed the investment in design has been a small fraction of the total development effort: but as any good investment, I think it's gonna pay off big dividends.
Wednesday, March 14, 2007
Web Services and the unfriendly appdomain
Sending frame-oriented data was the easy part. We just used UDP: light, connectionless, and since we don't mind occasionally losing some data (if we lose one frame, we'll display the next) that was just the best fit.
The palmtop, however, must also send commands to (and receive answers from) the legacy applications. They already have a (legacy) interprocess communication protocol, currently based on named pipes, and we definitely didn't want to integrate the palmtop into that (as we want to scrap that subsystem as soon as possible).
This sounds like a good opportunities for decoupling everything through a web service: the palmtop will talk to the web service (with all the advantages of having the proxy code generated), the web service will talk with the legacy system[s]. For a number of reasons, we decided to develop the palmtop software in C#, a significant portion of the web service in C# as well, and the glue between the web service and the legacy applications in C++/CLI (which is an ideal candidate for gluing together the managed and unmanaged worlds).
We had a [legacy] DLL exposing a set of functions of the underlying protocol: it was basically a publish/subscribe event model. You have to register a set of callback functions, that the legacy DLL will call on specific events (that's where C++/CLI becomes handy: the DLL assumed regular C pointers everywhere).
Everything was smooth and fine, except that (as known) ASP.NET is using a different appdomain for each hosted web service (or application). That won't be a problem, except that we soon discovered that all the callbacks from the unmanaged world got rooted into a separate appdomain. As appdomain are isolated, communicating data back and forth between the two is problematic.
This is usually no big deal: when you are in control of appdomain creation, you can have some portion of code (which creates the appdomains) storing serializable or (like it would be our case) MarshalByRefObject references into the different appdomains, through SetData. You can then retrieve the data through (guess what :-) GetData inside each appdomain. In our case, we could simply store a reference to AppDomain #2 (the one executing the web service) into AppDomain #1 (the one where the callback takes place). AppDomain derives from MarshalByRefObject, so this was just fine. Inside the callback, we could have used the DoCallback method of the AppDomain class to execute some code inside AppDomain #2 (as we needed to use data allocated as part of the web service logic).
The problem is, ASP.NET is creating the appdomains, and the .NET framework doesn't provide a simple way to obtain a reference to other appdomains (short of hosting the run-time yourself, or implementing a AppDomainManager, which I've considered at some point, but discarded as too ugly).
At that point, we started trying a variety of different approaches, including bypassing the whole appdomain isolation through C++/CLI and __declspec( process ) and all the exotic stuff, just to get a fancy variety of undefined behaviours.
In the end, a good guy working with us suggested that we just used a different approach:
a) create a new thread as part of the web service initialization, so in AppDomain #2.
b) have that thread wait on a Win32 kernel object (an auto-reset named event)
c) have the unmanaged callback in AppDomain #1 copy the data in a shared unmanaged space and signal the event
d) that will wake up our thread in AppDomain #2, and since unmanaged memory is, well :-), unmanaged, we will have no isolation problems.
That worked like a charm.
Now, we still have some issues when the appdomain gets unloaded (IIS can unload/reload your appdomain for a number of poorly specified reasons). By forcing a reload through a manual change to web.config, we're experiencing some problems with the IPC libraries, although we have carefully derived a critical class from CriticalFinalizerObject. There is some interplay between deferred finalization, new appdomain creation, unmanaged threads and so on that is misbehaving. We'll solve this too, but it's gonna cost more than expected.
Long story, short conclusion: it's ok for technology to go to some length to protect programmers from themselves. However, it is not ok to introduce a technology that cannot be controlled easily by programmers. It's the usual Visual Basic syndrome, where easy things get easier and hard things get harder, and I definitely don't like that...
Tuesday, March 06, 2007
Fighting the Useless Computer
As usual, the most difficult part is keeping the system out of the way while we try to understand the real users' needs. As I've argued before, there is often a difference between what the users are asking for and their realneeds. Like everybody else, users have an hard time separating their current practice from their fundamental tasks (or problems to be solved). Blindly implementing users' requests often results in a sub-optimal solution to their real problems, to the point that many have argued that IT does not really improve productivity.
There is, in fact, a very common syndrome, that I call "the useless computer". It's when a computer is used as a file cabinet on steroids. Indeed, the most common sign of the useless computer syndrome is a GUI centered around a multi-column grid where information has to be selected, filtered, ordered by the user. That usually happens because the computer doesn't know the last thing about what the user wants to do with that information, so it can't help at all.
Interestingly, the grid is a popular shortcut among programmers. I've heard countless stories about how users are actually requesting a grid. In facts, I believe the grid is a popular compromise based on mutual frustration: users who have given up on the idea that someone could really care about their needs, software developers who find users unavailable, illogical, fuzzy, contradictory, never satisfied.
Unfortunately, the tools we use are not helping us much either. I've never been a big fan of use cases, because they bring in the system way too soon. By analyzing and documenting "requirements" as a conversation between the user and the system, the focus is soon shifted away from the fundamental problem and toward a possible solution. Sure, rushing into implementation can be even worse, but use cases are a very unbalanced tool. Task descriptions, although quite similar, can be more effective. The KAOS approach, with its concept of goal and softgoal, can be quite effective too. Unfortunately, use cases have marginalized most other approaches: they're relatively easy for the software guys yet give almost-decent results. Most companies are fine with that, as they don't pursue excellence in solving the real users' needs.
As usual, it could be useful to take whatever it's widely used and improve it just a little bit, maybe by transplanting some idea from better approaches. A simple, neat idea that I've recently read about in a paper by Neil Maiden is to decorate the use case prose with attributes like [ENABLE] (or, I would say, [REINFORCE], and so on) followed by the name/id of a fundamental users' goal. This is very similar to what we do in KAOS, except that in KAOS the focus is on the goal themselves, while they play a marginal role here. Still, it's simple, is effective, and can be a small step in the right direction.
