Signs of Triviality

May 06, 2014

I got patches on the patches
On my old blue jeans
Well they used to be blue
When they used to be new
When they used to be clean

Everybody knows what the internet looks like, right?

Before Clown^WCloud Computing became the buzzword du jour, countless diagrams have been drawn on corporate whiteboards, in schools and universities representing the internet as a cloud. Cute. Fluffy. Non-descriptive. Just the way you'd want something to look that nobody really needs to understand but has to trust anyway.

"Ah," you say, "but in reality, the internet looks like this:"

Network connections made at peering points. A series of tubes, if you will. Another visual, perhaps more accurate and the better to represent at least some technical aspects. That's nice. Ordered. Stable. Just the way you'd want something to look that fuels such an enormous part of the world's economy.

Here's what I think the internet looks like.

It's been around for a while. It's got a few scratches, and it has been taped over countless time. But it keeps running. Doesn't look like something you'd want to get on at all. And yet you do. Every day.

Having looked a bit under the hood, I can't even give you a more reassuring report of what's going on on the inside. It's a miracle that anything works at all on the internet. It's not turtles all the way down, it's patches upon patches. It's brand-new software unleashed onto the world with minimal testing upon old, rickety libraries and frameworks that were unleashed onto the world with minimal testing and that nobody dares to manipulate and few even pretend to understand.

Almost all software out there is basically a prototype that escaped into production. It wasn't meant to be relied on, to take all this traffic, to run for so long. You don't think so? Here's an example:

There's a reason for this, and it has to do with deadlines, a willful ignorance of how software projects develop a life of their own, and a fairly good chunk of hubris on our side.

I love deadlines. I like the whooshing sound they make as they fly by.
-- Douglas Adams

When was the last time you delivered a project on time? The way it was planned, without compromises, with full reliable end-to-end testing, and with complete and accurate documentation? I thought so.

We suck at estimating software projects. Take your most conservative estimate, multiply it by five, and you'll still have trouble meeting that deadline, even after having applied this rule in the initial estimate. Setting a software project schedule basically means choosing an arbitrary date as the launch date, and then making compromises until you ship whatever you have at that time.

The reasons for our poor software engineering estimates are well understood -- Brooks's Law has been known for over 30 years, for example -- but we choose to ignore them.

Dunning-Kruger necessarily far outweighs impostor syndrome in the internet industry. The projects we work on are new ("You'll work on something that nobody's ever done before!"), but we believe that similar past experiences will make this project a walk in the park. We're smart, we've done this before. Well, not that exact same thing, but something similar. It had to do with computers, anyway, so this shouldn't be a problem.

The reality is that nobody knows what they're doing, we're all just playing adults.

And in the highly unlikely event that the software we write is not perfect (gasp!), we have no doubt that changing it later will be easy anyway. After all, software is infinitely malleable. We can rebuild it. We have the technology. But we don't have to think about that right now, we're on a schedule.

Unfortunately, changing software after it has been released becomes increasingly more difficult. And laborsome. We'd have to retain backwards compatibility -- often times for bugs others have come to rely on -- or decide to break other software, clients, or consumer expectations. This gets tedious quickly. You know what's much more exciting? That new project over there, where all the smartest engineers in the company are working on this ground breaking new product for make benefit glorious ego of self. Shiny!

In the mean time, somewhere on the internet, a software engineer encounters our product -- bugs, inconsistencies, warts and all. Scoffs and snickers: "Why didn't they just (use NodeJS, YAML, use 64bit ints, write this whole thing the way I would have)?" So yeah, they go and write their own. It'll be so much better than this piece of crap! Witness the birth of another web server, itself further evidence of Sturgeon's law.

Somewhere else an SRE, or perhaps a System Administrator, accepts our products' problems and works around them, because that's what we do when we can't write our own funamjing. They probably used a shell script wrapper around a perl script, or perhaps a custom, self-extracting zip archive "binary" of some python code that can only be built on this one guy's laptop.

And that is how the internet is held together. Dirty Deeds Done Dirt Cheap.

May 06, 2014

Related:

• less bug, more /dev/null
• Defining "Operations"
• Writing (system) tools
• Software and Systems Engineers - How We See Ourselves
• Writing Shell Scripts
• Writing Consistent Tools

• Programming Sucks
• In Praise of Evolvable Systems
• Everything is broken
• The Internet is Burning
• The Internet Is Held Together With Bubble Gum And Baling Wire