I'll be in Fiji for about 10 days, no internet, no phone, no nerdy stuff :-).
See you all when I'm back :-).

I've put together an incomplete implementation of the Wise Pointers concept. Although a few features and optimizations are missing, there is enough substance to do a first performance comparison. See my previous posting, Benchmarking Smart Pointers in C++, for more details on the benchmark.
In their first run, with N = 10,000, Wise Pointers didn't improve much over the best smart pointer arond: 44,000 cycles Vs. 47,000. Quite unimpressive. With N = 100,000 things got more interesting: 660,000 Vs. 1,000,000, much closer to the raw pointers performance (450,000).
The problem is that the copy constructor, destructor, and assignment operator for Wise Pointers are too big (around 160 bytes of machine code each), so they don't get inlined. If I force them to be inlined with __forceinline under Visual C++, I get much better performance: 36,000 cycles in the first case (closer to the 26,000 score of raw pointers) and 570,000 cycles in the second case, again very close to raw pointers.
To be fair, I tried a __forceinline on my "regular" smart pointers (using two parallel pointers), and as expected their performance didn't change a bit, because they are already been inlined. I didn't try that on Boost and Loki implementation (yet).
Bottom line: with __forceinline, the current implementation is interesting, because the overhead of using smart pointers has gone down from 80% - 120% (respectively, for the two best implementation for N = 10,000 and N = 100,000) to 38% - 27% respectively. Of course, the code gets bigger (hardly a problem these days, but anyway). I'm still unhappy with the idea of using __forceinline, so I'm gonna think about the code a little bit more, as soon as I get some time to spare...

Before proceeding with the implementation of my Wise Pointers, I decided to create a small benchmark and test some known implementation of reference counted pointer against it. It is far from comprehensive (indeed, is extremely small) but still revealing.
The small program basically does this:
- Creates a std::vector of reference counted pointers to int
- reserves space for N items (see below)
- uses push_back to fill the vector with N items dynamically allocated and pointing to random integers
- calls std::sort to sort the vector, using a dereferencing functor to sort by pointee
- calls std::reverse on the vector
- calls std::random_shuffle on the vector
- sorts the vector again
- calls std::max_element on the vector, using a dereferencing functor to get the max pointee
I then tested the program against:
1) regular (raw) pointers. In this case, the program also loops and delete the dynamically allocated ints at the end
2) my own reference counted smart pointers (implemented using two parallel pointers)
3) boost::shared_ptr (implemented using two nested pointers)
4) Loki::SmartPtr, with the RefCounted policy (implemented using two parallel pointers and a custom small object allocator for the reference count storage)
5) Loki::SmartPtr, with the RefLinked policy (therefore implemented with reference linking)
I run the program with N = 10,000 (ten thousand) and N = 100,000 on a Pentium IV 3.2 GHz, using Visual C++ 7.1 with all the optimization turned on for maximum speed.
Here are the results in clock cycles, very roughly approximated right now:

N = 10,000
1) 26,000
2) 47,000
3) 53,000
4) 47,000
5) 61,000

First interesting conclusions: there is a very significant overhead (although that is not diminishing my love for smart pointers and RAII :-). Parallel pointers are faster than nested pointers as I fully expected :-). The small object allocator in Loki is very disappointing (or, the default allocator for VC 7.1 is very good :-). There is, indeed, a small difference between 2 and 3, but not at this approximation level. Reference linking is slower, as Alexandrescu expected in his "Modern C++ Design" (I have never used that technique in real programs).

N = 100,000
1) 450,000
2) 1,100,000
3) 1,300,000
4) crashes in custom allocator
5) 1,000,000

Relative overhead gets even bigger for 2) and 3), with parallel pointers still a winner. The custom allocator crashes, telling us to keep it simple :-). The performance of the reference linked implementation is quite interesting: it is relatively smaller, although not by much.

Well, as soon as I get some time to implement the Wise Pointers, I'll repeat the benchmark and post the results here.

In the past few days I've been thinking on how developers tend to organize themselves (as groups). For reasons I'm not sure I completely understand, it seems to me that they tend to follow a tribal organization, whereby they often have some kind of totem (e.g. a language), a shared set of beliefs and rituals, and a strong sense of who's part of the tribe and who's not.
Yesterday I was skimming through a rather boring article (sorry :-) on Communications of ACM, January 2003, "Barriers to Effective Use of Knowledge Management Systems in Software Engineering", by Kevin Desouza. At a point, he says: "Software engineering knowledge is highly tacit in nature, much of which cannot be articulated well or put in explicit format". This reminds me of the Feigenbaum bottleneck (see my Principles Versus Patterns paper for a few relationship with software engineering). But it also reminds me, for some tacit reason which cannot be articulated well :-))), of tribal knowledge, with an oral tradition that seldom gets outside the tribe itself.
Oh well, time to get back to PDM, Web Services and the like, overall much easier stuff to deal with :-)).

In 1998, while preparing the slides for an advanced course in C++, I envisioned a different kind of reference-counted smart pointer, which ought to be significantly faster than the then-existing alternatives. I called them "Wise Pointers" because they were... wiser :-) than the average smart pointer :-).
The subject seemed a little too advanced :-) for the course and didn't make it into the slides: actually, I never took the time to implement the concept and try it out. I guess I forgot about it shortly after; that's the problem of having too many ideas and too little time :-).
A few months ago, I read an interesting paper on the distribution of incoming and outgoing references in Java and C++ programs ("Scale-Free Geometry in OO Programs", by Alex Potanin, James Nobile, Marcus Frean and Robert Biddle, Communications of ACM, May 2005). It reminded me of the wise pointers idea, indeed providing an experimental background for the usefulness of the concept. Still, somehow I put the idea on hold and again forgot about it.
Today, I was reviewing my slides on template metaprogramming, as part of the customization of an advanced C++ course for embedded / real-time programming. The idea popped out again, and this time I didn't send it back. I did a reasonably thorough check and it seems like nobody has came up with something similar in the past 7 years :-), so it's time for me to write some code, try it out, and if it works as well as I think it will, write a short article about it (yes, full source code will be included :-)).

I subscribe to (and try to read :-) a number of magazines, both on paper and online. One of those is ACM Queue, that I usually just skim. Unlikely most refereed publications, it is not unusual to find some strongly opinionated article on Queue. Yesterday I read a new paper, a critics of overloading in programming language under the funny title of Syntactic Heroin.
The author is strongly against the concept of overloading, and he tries to explain why by using C++ as an example. Unfortunately, there are a number of flaws and omissions in the article. I posted a short comment on what I consider the major fault directly on the publisher's site, but here is the long version.

- Not seeing the forest for the tree
Near the beginning of his arguments, the author says The best you can say for programmer-defined overloading is that it is only syntactic sugar - that is, it makes things look nicer, but it adds no significant programming capabilities at all.
Now, we have to agree that everything more than a Turing machine is syntactic sugar at some level. There is no need for data abstraction, functional abstraction, symbolic names for functions and variables, etc. All the languages are basically equivalent in their functional capability (this is known as the Church's Thesis-M). However, we usually consider several factors as "useful" or "significant". Among those, fostering reusability is normally considered beneficial.
Now, overloading per se does not foster reusability. However, overloading + templates encourage reusability. The major fault of the author here is to consider a language feature in isolation, and not as part of an organic language. Since the article seems particularly concerned with C++ bashing, here is some C++ code:

#include < vector >

template< class T > T Sum( const std::vector< T >& v )
  {
  T sum = T() ;
  for( std::vector< T >::const_iterator i = v.begin(); i != v.end(); ++i )
    sum += *i ;
  return sum ;
  }

This will sum all the items into a vector< T >, for whatever type T with a += operator defined. Of course, further generalization is possible (there is no need for v to be a std::vector, any container with a forward iterator would do). Try that without overloading, and you end up passing a function object, making Sum harder to write, call, and read at both sites. Obviously, this is just a small example, but the technique in itself is quite useful, and indeed widely used (just look at the C++ standard library).
So, Overloading + Templates = Reuse. How's that for "adding no significant programming capabilities at all"? :-)).

- Confusing features with languages
As other readers pointed out, there is a constant confusion in the article between overloading and the intricacies of built-in conversions and promotions in C++. Note that unlike other readers, I do not consider C++ to be an ugly language because of that. And here comes another flaw:

- Not recognizing the forces and rationale behind a language
C++ inherits most of the built-in conversions from C. We may or may not agree with Stroustrup's decision to be as compatible with C as possible, but criticizing C++ without an appreciation of this issue is pointless.

- Intentionally (?) making the picture worst than it is
The author goes on to say "the constructors with one parameter, will usually have been written for another purpose (constructing), and so are likely to get into the overload resolution stew inadvertently". He probably knows (being a "language lawyer" as he defines himself) that in C++ we have the explicit keyword exactly to prevent that, but he choose to forget.

- Doomsayer without statistics
Overall, the author talks a lot about the woes of overloading without providing any reasonable statistics of errors that we can trace back to overloading. Not even the usual experiment with graduate students. He goes on and on saying stuff like "Language designers, compiler writers, developers, and users all suffer" but provides no experimental evidence for his claiming.

- Using scary numbers without examples and explanations
Those 29 built-in conversions are not so pervasive as he want the readers to believe. They apply between base type (e.g. integral an pointers) and are usually quite harmless and very natural, as an int can be promoted to a double, a pointer to a const pointer, and stuff like that. Providing clear examples of how this can harm the programmer would have made the article much better.

- Using scary formulas without a context
The author says If there are n of these, there are (n*(n-1))/2 pairs but fails to say that usually you don't have all those viable functions! In most cases you have 1 viable function, so that formula gives you 0 pairs (not so scary anymore, right? :-). Say that we have 3. We get 3 pairs. So what?

- Evangelist in disguise
All that C++ bashing seems more and more like Java evangelism (Java does not have overloading) when you consider the closing statement "Those with real courage will break the addiction by refusing to use programming languages that push this drug" immediately followed by a bibliography where (out of nowhere) we find a reference to the Java Language Specification...

I could go ahead - but there is no need to. Actually, the first fault is probably so huge that nothing more was needed anyway. Instead, it is interesting to see how other readers kept criticizing C++, which is obviously not so "pure" as most people (apparently) want languages to be. In my old Interview with Bjarne Stroustrup, he said C++ has the advantage that its use scales to real world problems in many diverse application areas. Much of the ease of learning cleaner/newer languages comes by simplifications that force its users to abandon the language when they hit an application outside the domain where the "clean" language is a reasonable choice. Try to use Java for scientific computing, and see how nice is to have a Quaternion, Vector, Matrix class without operator + defined :-).
The comment on COBOL is also obviously misleaded, as in C++ you cannot change the meaning of existing operators for existing types, but only to overload them on new types. There is only one exception, that is, overloading operator ",", which in fact Stroustrup now thinks he shouldn't have permitted. I would say this is a statistically insignificant problem, but hey, I don't have experimental data to back this up :-)))).

... until they try them out.
This comes from a senior manager at one of the companies I consult with, an electronic engineer with a deep incompatibility :-) with anything which can be labeled as software (including, in some sense, software engineers :-).
He pointed out how programmers in his department kept spitting on Windows as being slow and unreliable, praising QNX and VxWorks as rock-solid, rocket-fast. However, when they actually had to implement a system on those operating systems, they found a number of flaws, so now they're spitting on them too. Now someone is dreaming of good ol' days when MS-DOS let you do whatever you wanted (which, for some applications, is indeed the right thing).
Sad but true, I'm afraid a lot of programmers are spitting on < you name it > just because it's the system they mostly work with...

We are often asked for estimates, even more often for "quick estimates". Managing estimates is not trivial, and it starts by realising that you have to manage estimates :-), that is, that an estimate is not a static promise. It doesn't really matter if you used a formal approach like function points and COCOMO or you just concocted some numbers based on your experience: you may need to review your estimates later, when you obtain new knowledge, requirements change, etc. This is somehow recognized by agile methods, that in many cases simply avoid the project estimation part, and concentrate on planning only the next iteration / user story.
Anyway, our estimates are often given in man/hour or man/weeks, that is, as an effort. We translate this effort in a schedule by assigning resources to the project. This is reasonable but somehow simplistic: it assumes that the effort is fixed, and the schedule is adaptable. We know that's not true: we can't compress schedule indefinitely by just adding resources. The effort is not constant: it is a function of the schedule as well!
Phillip G. Armour elaborates on the nonlinear relationship between effort and schedule in a very nice (and short :-) article in Communications of ACM, June 2004: "Real Work, Necessary Friction, Optional Chaos". Worth reading :-).

Although it is not widely recognized, object oriented programming brought several concepts from artificial intelligence, in particular from [definitional] semantic networks.
I'm old enough to have taken an Artificial Intelligence course :-) back at the university, where we studied semantic networks among other things (truth maintenance systems, non-monotonic reasoning, frames, logic programming, etc). Overall, quite interesting stuff, but it didn't deliver. One of the reasons, in my opinion, was the huge waste of energy spent in the pursue of completeness, not to say perfection. Semantic networks, for instance, had a concept of inheritance, and lot of time was spent to accommodate misclassification, like the birds that can't fly. Robert Pirsig (author of "Zen and the Art of Motorcycle Maintenance" and "Lila") calls those hard classification problems the platypuses of a theory, and AI spent too much time tinkering with platypuses.
When OOP came along with the notion of inheritance, the whole misclassification problem was largely ignored (leaving aside a few academic papers). The OOP community basically took the useful stuff, said "who cares" about the rest, went on and delivered. They moved the problem of misclassification from the concerns of the system to the concerns of the developer. We can violate the LSP at our own risk. We do our best to avoid that, and meanwhile we build useful systems. OOP does not pretend to be a complete, perfect system; that's why it can do useful work.
Today, Aspect Oriented Programming is an interesting paradigm that seems to be spending a little too much time with its own platypuses. It's just a matter of time before someone will come along, grab the low fruits, and deliver. .NET context attributes (plus transparent and real proxies) and java refection proxies, are not powerful enough, but they're a first step away from all the aspect weaving blurb and toward a simpler interception model, and in a few cases they can actually deliver. We just need something easier and more powerful :-), and that would be well within our reach, if only simplicity was a little more valued.
To my completeness-and-perfection-oriented readers: remember that Goedel, in 1931, proved that in any consistent formal system, which is adequate for arithmetic, there are propositions that cannot be proved or disproved within the axioms of the system (that's the famous Goedel Incompleteness Theorem, informally). So all our beautiful math, all our revered logic, is not complete either. That's why it can do useful work.

Over the years I've studied and used several different languages. Like many programmers, I ended up liking some, disliking some others. I've also collected a number of ideas on how my "ideal language" should look like. Some of them are really trivial: for instance, I like the C/C++ syntax more than the verbose Pascal or Basic syntax. Others are much deeper. I'd like the language to be extremely small, so most of the language would end up in the library. I want a language so small that things like "for" and "while" should be in the library. This would allow the [smart] programmers to add new concepts working at "level 1", while usually working at an higher abstraction level. Method interception and rich relationship semantics, as well as deterministic destruction, are also fundamental ingredients of my ideal language.
More on this in the future, but no, I'm not seriously considering the idea of creating a new language :-).

Yesterday evening I picked a journal among the too many waiting to be read :-) and found a short, interesting paper about all-pairs (more generally, n-tuple) testing:
Software Fault Interactions and Implications for Software Testing by D. Richard Kuhn, Dolores R. Wallace and Alberg M. Gallo Jr, IEEE Transactions on Software Engineering, June 2004. See also my previous post Brushing up slides on Testing for Testers for more.
The authors analyze a few real-world systems, and their results confirm the theory that testing all-pairs, and maybe all-triples, may be enough to find a large majority of errors.
However, we are still talking about a lot of test cases here! The implications on ideas like "write test cases first" and "I can safely refactor because I've created an adequate unit test suite" are not so trivial... :-)

A comment to the previous post reminded me that many programmers would be happy to gain direct access to hardware under Windows without writing a device driver. Now, I'm not sure how much damage I'm going to do by saying this (I mean, it is not exactly a secret anyway :-) but it's actually possible to access physical memory without a driver, or to open up direct access to I/O ports for a regular user mode program.
Physical Memory is accessible by opening \Device\PhysicalMemory. Unfortunately you can't do that through the Win32 API, because it will filter out your request. You have to go one step below and use the (officially undocumented) Native API. You can find several docs on the Native API on the internet, but also in books like "Windows NT/2000 Native API Reference" by Gary Nebbet (the book is little more than a listing of functions, parameters, purpose: is not a tutorial). Complete source code for physical memory access is available at the SysInternals web site. That code uses the Native API even when the regular API is enough (map/unmap views on memory mapped files) but hey, you can fix that yourself :-).
Accessing I/O from user mode is not so straight. The best idea I've seen so far is pretty old (1996) but still working. You write (once and forever) a device driver that can modify the IOPM for a range of ports and for a specific process. Then, the process can use inp and outp to read/write from ports, undisturbed and much faster than it could go by asking a driver through IOCTL. So if you want a user-mode process directly peeking on the UART state :-), that's the fastest way to go.
You can find all the details in "Direct Port I/O and Windows NT" by Dale Roberts, Dr. Dobb's Journal, May 1996. Again, lot of drivers inspired from that article are available on the internet; some of them are free, but I'm not sure if they come with source code (if you find some, you may want to post the link here!).
Finally, you may also want to trigger some user-mode code when some interrupt fires. Again, you can write, once and forever, a driver that connects to the interrupt via IoConnectInterrupt, and then reacts by setting a named event (that can be opened from user mode as well). The user mode code will be waiting on the kernel event. Of course, you'll get a huge latency, so this is definitely not recommended in many cases.
Be aware: these are pretty much toy techniques. I've used them in a few cases, simply to avoid the creation of a custom driver under very controlled circumstances. I've also find them useful when porting old code from 16 bits Windows, where direct access to [unprotected] I/O and physical memory was possible. I don't recommend that you bet your projects on this stuff :-). Still, these toys can be very useful at times...