Suppose -- arbitrarily and entirely theoretical, of course -- that you wish to replace the currently used /lib/libc.so file. Considering that you might be using -- further absurdely and entirely theoretical, of course -- cp(1), itself dynamically linked. It might well happen, that during the copy process, you loose, as cp(1) might fail and thus won't finish the copy. In theory.

Now should this happen, you need to get a proper libc back into your filesystem. You can do this remotely (if you have remote console access) on your trusty old NetBSD system by booting into single user mode. Or can you? As you boot, you will find that the dynamically linked /sbin/init will also fail...

Ask the bootloader to prompt you for the path to init by passing '-a' to the boot command (so together with single-user mode this would be 'boot -as'). Then pass it /rescue/init as the path to init and /rescue/sh as the shell. From there, you can fix your system via the other commands found statically linked in /rescue. Which is exactly why they call it that.

Mac OS X packages and software updates in the form of a .dmg disk image can be installed remotely via the hdiutil(1) and installer(8) commands. For example:

% hdiutil mount image.dmg
% cd /Volumes/image
% installer -pkg pkgname.pkg -target /
% cd /
% hdiutil unmount /Volumes/image

Moving our cluster to a netbooting setup, we replaced the originally purchased Broadcom Gigabit Ethernet devices (which were unable to netboot) with Intel's i82540EM Gigabit Ethernet cards. While installing these cards, a number of them appeared to misbehave: upon bootup, they would display a link light, but as soon as the card was configured by the firmware, the link light turned off and the card would claim that no network link was found.

Some of the cards would maintain the link-light, but refuse to netboot -- they would not be able to get a DHCP lease, even though the server was configured to hand out the right lease to the devices. Some cards, seemed to work out of the box, without any problems whatsoever, but the majority would always loose the linklight soon after the card's firmeware kicked in.

In the end, the problem turned out to be related to the network cables: some of the cables used were regular CAT5 cables, while others were CAT5e cables. After replacing all cables with CAT6 cables, everything worked out fine and all cards were able to netboot.

Note that even though the CAT5e should work fine, some of the cards showed this unexpected and rather frustrating behaviour even with these cables. It seems that these network cards are very particular about the used cables.

After deciding to reinstall our Apple XServe, I faced an odd problem: the CDs would always be ejected. No matter what I did, I could not get the CD to stay in the tray, even though it was a verified good disk (check on other machines).

I was unable to boot into the firmware using either the keyboard shortcut, nor via the power-button sequence.

I was unable to boot from CD, obviously, but I was able to boot into single-user mode via the keyboard shortcut. Once in single-user mode, the keyboard stopped working. The next time around, booting into single-user mode worked again, and the keyboard worked as well, but still no luck with the CD.

Ready to take advantage of our AppleCare contract, I was going to ship back the machine, until the helpdesk at Apple asked whether I had made sure that the machine was completely unlocked. Unlocked. Not locked.

Well, go figure. I go and unlock the drives and the machine behaves normally (so far as I can tell): I can boot into firmware, I can keep a CD in the tray, I can boot from CD etc. etc.

So... lesson learned: You always need to unlock your XServe if you want to perform any such maintenance.

Go buy a Minneola, a crossing between a tangerine and a grapefruit. Apparently Mineolas are a type of tangelo. And rather tasty.