At the superficial level, pure breakage is caused by a lack of testing. The IT team sets up something new, but doesn’t test the new setup to the extent that the software team will exercise it. So we end up with an emergency when someone realizes they don’t have access to their important files from their PC. Or because our IT is outsourced, the IT team doesn’t realize their changes are not working a day later because they’re not using the system.

But both breakage and performance are classic software engineering problems. One well accepted solution for both types of issues is to build and run tests. Monitoring systems (cactus, monix, hobbit, etc) that provide automated alerts of problems are really an implementation of common unit tests.

Previously, I had not thought of IT as an environment that’s suitable for unit tests. After all, unit tests are for code. Why write a test for a problem that you’re going to Google for a solution? In my pragmatic view, IT is much more do it once and forget about it until it breaks next year.

But the more time I spend tracking down IT problems, reporting them and waiting for the fix, the more I want to just automate it all. So far, setting up Hobbit is our first step at a monitoring system that will detect first-level problems like, “the web server is down.”

Using the monitoring system to measure performance is a little trickier. I don’t have much experience with that yet, so I’ll have to save that for a later post.

Outsourcing IT allows my company to get cost-efficient service and skills without hiring dedicated employees, but it means that we really don’t have 9-5 service. When something goes down, there’s usually a delay until someone can look into it. Sometimes there will be an IT person on site, but generally if a machine goes down, the right person isn’t around.

There’s also a significant cost to “Emergency Service.” I don’t know what that cost is in dollars, but every time I’ve filed an emergency ticket, I’ve been asked by our accounting department whether or not it’s really an emergency. Now, according to the time and attention people (and I’m not saying that they’re bad people), putting this cost on a work order that’s going to interrupt somebody is actually a good thing. But the real-life effect is that I end up asking myself, “Is this something I can repair in 15 minutes?” So, there’s some set of critical repairs that displace my real work.

Another major consequence of outsourced IT is that the people doing the IT are not using the systems that they maintain everyday. This means that they don’t notice intermittent problems (here today, gone tomorrow) and they can’t react quickly to obvious outages.

These factors have driven a couple changes in the way I approach IT, which I’ll detail in future posts:

• I am effectively responsible for project management of IT projects; doing the planning, scoping and figuring out the details of how infrastructure projects should be implemented.
• Individual issues need to be prioritized relative to tasks for longer term projects, or else little progress is made on the long term projects because there are always minor issues to tackle.
• Automated monitoring is extremely important. Update: Monitoring is unit testing for IT.

On the plus side, we’ve got a decent team of people that are (for the most part) on call. On the minus side, I have to devote some of my time to IT work and I have to do much more of the planning and scoping of IT-related tasks. And there’s a lot of project definition that needs to be done to avoid IT disasters.

Anyway, as a command line geek, I wanted a system where I could build and modify my own software. As most command line geeks know, running through the ‘./configure && make && make install’ steps for each piece of software that you want to play with gets old pretty quickly. So originally, I started using Fink because at the time it was the most mature system.

There are lots of good reasons for using a packaging system, but having a system for automatically fetching and installing dependencies is fantastic if you want to experiment with random code that someone else developed and released. Because chances are, that random code depends on some other random library written by someone else.

There are a number of things that I like about Fink:

• it worked really well for me — generally, when I found some new software that I was eager to try, the software (or at least its dependencies) was often already ported in Fink
• the addition of binary packages was a great thing
• the community was helpful

And some that are kind of stifling, like the rigidness of the Fink motto, “there’s only one way to build a package.” Of course, I understand the argument for it, and that there were various (unsupported) ways to get around it. For a volunteer organization, it’s hard to allow too much flexibility.

When I upgraded to Leopard, I decided to try out MacPorts because of the feeling I was getting from Fink that its support was stagnating. It’s hard to tell where the momentum is without doing some comparison of the traffic / quality of the developer lists. At this point, I feel more comfortable relying on MacPort for my software.

There was an adjustment period. But it was mostly a matter to translating the commands I was used to from fink into MacPorts syntax. At first, it was a little confusing to get thrown into variants coming from Fink, but they make sense. I wasn’t used to checking out the various options before installing a new package. Overall though, the transition was pretty easy.

I started writing my own Portfile [link] recently, so that’s another sign that MacPorts is growing on me.

However, after reading some recent email on the MacPorts developer list, I see the same themes that came up periodically in Fink.

• what about binary packages (vs. what about security)
• too many ports and updates, not enough maintainers

I can’t yet tell where these themes are going to go. It really depends on how much the community gets involved in making improvements. From what I can tell, it’s a matter of making contributions (time). The community is definitely open to receive them.

For me, the migration was worth it. I was able to install the software I wanted to use — such as svn, gnupg, git, imagemagick, ragel, scons. It allowed me to prune a bunch of old cruft that I never use anymore. It allowed me to experiment with a different OS X packaging system.

We’ll just have to see what happens next. I’m happy with MacPorts now.

I recently read (and promptly re-read) Steve Yegge’s emacs tips, which is full of good info.

I use Emacs all the time when I’m on Linux, but I rarely hack elisp. I can get around well enough, but there’s plenty of stuff that I know I could be doing better (faster, or more efficiently, or without dropping into the shell) if I had a genuine emacs master looking over my shoulder.

I’m looking forward to exploring some “new-to-me” features.

class Foo {
public:
Foo(bool in);
...
};

...
Foo f = new Foo(true);
...

Yikes. Didn’t expect that to get by the compiler. But it did. Without even a warning. The compiler was happy to turn “new Foo” into a boolean value and instantiate f, silently letting the memory leak away.

This helps:

class Foo {
public:
explicit Foo(bool in);
...
};

But still. Ug. Single argument constructors in C++ are just evil.

Read all about my son, Caden, on the Bacher Blog.

Watch on blip.tv

via tedtalks in Democracy.

Anyway, we took a quick trip to Sedona for a long weekend. Here are some photos from the Grand Canyon.

More can be found in the Bacher gallery: http://daveandkelli.info/gallery/

Move or Quit: “I’ve been working from home for nearly 7 years now. In 2000 I asked my boss if I could work part-time from Corvallis because Mary was moving there. He agreed, and I’ve done work ever since.

The mother organization (12000 employees) has made the decision that people working remotely is not to be allowed. I must either move up to Portland (Santa Clara, CA and Hudson, MA are also allowed) or quit.”

I can imagine that there are a variety of reasons why this big company would want to institute such a policy: security issues, to reduce slack-iness, etc. But rather than forcing people (like my friend Trey) to move, wouldn’t it be better to address the specific security or productivity issues that gave rise to this new “no-tolerance” policy? Or perhaps, in the long run, it’s better for big companies to drive away good people.

(Via Trey.)