The More they Overthink the Plumbing…

There are many lines in Star Trek which have stuck with me over the years; recently, it's been this line from Scotty when explaining how he sabotaged a ship that was trying to chase them: The more they overthink the plumbing, the easier it is to gum up the pipes.

TL;DR for the lazy: XP good, make code as simple as possible, avoid 3rd party libs and APIs, here's why and how we got here.

History

This feels very pertinent to software engineering, where we have been building ever taller towers of abstraction, many of which make things worse.

The very beginning

At the base, computers are electronics, complex circuits with voltages propagating through them. They're not even really digital electronics — transistors are fundamentally analog devices, that can be used as amplifiers, it's just that if you abstract things in the right way, you can treat it as "close enough" to discrete on-or-off to perform Boolean logic. This is genuinely useful, because while it is possible to perform computations in the analog domain, it's really hard to do this with sufficient reliability for anything complex… but even so, we may well find that very large neural networks are somewhat resilient to these imperfections just as our messy wet chemical brains are somewhat resilient to all the things going on in and around them.

Next level up, Boolean logic. On-vs.-off, 1-vs.-0, isn't enough, so we bunch several together to make base-2 numbers. That's great, and lets us represent natural integers, which themselves can be treated as if they were letters or symbols by way of a simple look-up table (which is basically how ASCII and its early competitors happened), or instructions for the CPU to follow. A few extensions also allow you to do negative numbers or floating point and decimal numbers (related but subtly different). All good stuff, though you do sometimes surprise people with 1/10th being a recurring fraction in base-2, so you shouldn't use this system for anything involving money.

Structures

Next level up, structures. Arrays are one of the foundational examples — you have a sequence of some length, call it n, that you're treating conceptually as a single thing. They might be numbers (income in each day) or they might be letters (to form a string of text). There were two common ways to do this when I was a kid: (1) to begin with a number saying how long this list is, followed by that many items; (2) to begin with the content, and to assume the list has a subsequent item until you encounter a special symbol that says "stop". Given my experience in Mac OS classic, the former was called a "Pascal string" and the latter a "C string", named after what the Pascal and C languages did.

Each of these approaches had weaknesses: C strings often didn't get their final "terminating" character created or checked leading to corruption of system memory; while Pascal strings had to have their size known in advance and thus led to using too much memory (by 1980s and 1990s standards where my first two home computers came with 64 kB and 8 MB of RAM respectively), and/or leaving them unable to cope with long bits of text (which is still a problem for some combinations of names and forms of national ID). There are ways around this with more complex structures such as linked lists and trees — I'm not actually sure how any given modern language handles strings internally, in part because they may have more than one string-like data structure and each may do things differently to give different performance trade-offs.

The structures can get more complex. Before I go down that path too far, I will also say that simple concepts like "time" and "a string of letters" can each be far more complex than it first seems: time comes with time zones, which can vary with little warning, and leap-seconds due to the rotation of the planet varying very slightly from one year to the next; and at the speed current computers operate at, I've even seen people care about light cones (to determine if a stock transaction on an exchange was suspicious do to it being created before information reached it from some other exchange) and GPS famously requires accounting for time dilation due to time dilation from both special relativity (speed-related) and general relativity (gravitational potential related). Strings aren't much better, as the original approach was to match what the keyboards on teleprinters used (with separate "newline" (\n) and "carriage return" (\r) symbols, where Mac OS classic ended each line of text with \r>, Unix (and modern Mac OS)with \n, and Windows with \r\n, and several other choices for systems you might have never heard of. And then we noticed that most of the world, even most of Europe, doesn't use just the symbols on a US/UK keyboard, and came up with a bunch of different encodings before finally settling on Unicode… which itself took a few attempts to find a representation everyone could agree on which (for the moment) is utf-8 — a variable length system which would be a bit much to describe here, but put simply: if you want to reverse a string, in ASCII you just reverse the sequence of bytes and you're done, but if you do that in UTF-8 you probably no longer have something that's even possible to display; even if you reverse the "code points" (the next layer of abstraction up) you're probably doing it wrong, as that switches the flags for Canada (CA) and the Ascension Islands (AC).

Algorithms and control flow

Structures were great, and did a lot for us. Indeed, we still find them all over the place as a fundamental concept. At this point I should divert from the data to the algorithms, as they too were given layers of abstraction: initially, algorithms were fixed as circuits; then we found ways to make circuits change other circuits, first mechanically (by rotating a drum with wires on them) and then electrically (with valves, relays, and transistors). The Turing machine was a formalisation of this abstraction.

Load instruction, execute (perform) instruction; load, execute; repeat. Nice and easy. Conditional instructions? A branch command: if ${some condition is true} then the next instruction you should load is at memory address ${xyz}, otherwise continue as normal. Nice and easy, good abstraction, the if statement is still with us. Loops?

#marker
load, execute
load, execute
load, execute
if not finished, go to #marker

Nice and easy, that's a do-while loop; there's several others I won't bore you with, they're still fairly common.

Next level of abstraction: functions. If you're doing one thing often, you probably only want to write it once and then re-use it in many places. We did this by borrowing an idea from maths, called a function. A function takes "arguments" as inputs, and then returns a value. Good stuff, nice and easy. Each instruction on a basic CPU is a little function, and more complex CPUs have "microcode" which is basically some built-in functions. Good stuff.

Objects

Why this diversion from data to processing? Because the next level of abstraction combines them both. "Object-oriented" code takes some data structure and combines it with the functions that are good for that data structure into a "class": for example, you might have a String class which has functions — in this context generally called "methods" — which perform operations on that string such as "make this uppercase" or "reverse this"; this is very useful, because it means a user of this doesn't need to think too hard about the underlying data structure — to go back to the previous point about the change from ASCII to UFT-8, I simply don't need to care if a modern String is using UTF-8 or ASCII, nor do I need to care about the rules for multiple bytes into code points, nor do I need to care about the rules for combining multiple code points into a single printable character, if I want to reverse a string, I can look for an existing reverse() method and I'm done.

They're really neat and useful. Indeed, objects were really popular for a long time, and although they have recently fallen out of fashion they are still widely available in modern languages. It will take a few more layers of abstraction to really say why they're no longer fashionable.

Next layer of abstraction: subclassing. Classes make it easier to divide code into logically relevant areas — string things in the string class, number things in a number class, network things in a network class, and so on. But things often also form hierarchies — say you're making an action video game in this paradigm (modern game development favours a different approach), and you want all the entities in your game to have a number of hitpoints, and that if depleted they will be destroyed. Great, so do you copy-paste a bunch of structures classes that each has a hitpoints property and a hitWith(damage) function? No! You have a parent class GameEntity with those things, and all your game's entities — chests/crates, the player, the enemies, exploding barrels — are made to "inherit" from this. Nice. Makes things logical and easy… except when it doesn't.

There are three big ways for this to cause complexity, the first two are easy to explain even though they can be a pain to resolve: multiple inheritance (if you've got two "parent" classes with a method of the same name, which one does the child class inherit from?); unclear hierarchies (is a square a special type of rectangle where the orthogonal sides are equal, or is a rectangle a special type of square where the sides can be different lengths?).

The third problem is typing — and no, I don't mean in the sense of what my fingers are currently doing on the keyboard. Each class (and structure, and also the "primitive" types the CPU uses such as "16 bit unsigned integer" and "64 bit floating point number") can be said to be "types", and each function and method will expect specific types on both input and output. But if you have a collection of GameEntity objects, you need something a bit more general, as any single GameEntity might be a Player or a Crate or a Enemy or an ExplodingBarrel, so you need a way to operate on both the higher-level concept of a GameEntity without erasing all the information about the specific detailed type. Unfortunately, this description just gave you the example where it's fairly easy, and everyone can see what's going on. This is very useful, and sometimes even mandatory — we want to have a generalised concept of a collection of objects without having to make special cases for both integers and GameEntity — the problem is that it's almost impossible to gauge the correct level of abstraction: often we either spend a huge effort to make something worth with any possible type when in practice there will be exactly one (it happens!) or we assume it will always be one of a small set of possible classes and make it too narrow.

These issues, combined with a few other more subtle issues, have led modern development to favour composition over inheritance: where related things are combined within some structure (or class) rather than inherited from a parent class — a Player (etc.) done this way would indeed have a property named "hitpoints", but that hitpoints property would be a class (rather than a primitive number) which would handle all the complexities and hide it from the Player.

Design patterns

Next level of abstraction: Design patterns. These come at two levels, code — for example, how are new objects created, how is the data presented to a different system expecting a specific format, etc. — and architectural. There's a lot of different ways to do this, each better or worse suited for a specific task, but you can imagine this as filling a similar role to classes, but one layer up: a good way to organise the code, all the things related to one task are found in one place. When I began iOS development, the standard architecture for apps was MVC: Model View Controller. The model is the data (perhaps a database on your device, perhaps a connection to an API on the internet); the View is your actual user interface; the Controller is the bit which glues the two together.

And this is where I get confused.

Apparently, lots of developers kept putting "too much" into the Controller.

(As an aside: I'm genuinely not sure what "too much" means — I've worked with code that clearly hit this category as there were 1,000 lines inside a single if-statement; but I've also seen some developers hold the opinion that anything more than 3 lines of code in a method is too many lines).

Too much into the Controller? The glue which combines the model with the view? OK, lets say this is so: what's the solution? There are several, because of course there are. Apple's replacement for MVC is MVVM: Model-View-ViewModel, where the ViewModel is a transformation of the Model to be specifically what the View needs to know about, and the glue (that was formerly the point of the Controller) is supposed to be automated. (This automation is what "Reactive" means in the context of UI development).

Don't get me wrong: if automation can be made to work, that's good. It's the point of computers, it's why they're everywhere. Even though the magic words to bind a value to a view are hard to remember, this is not the problem.

There are two problems here, one is fundamental, the other purely in how the magic binding works. The fundamental problem is now we have both an automated connection (View-ViewModel) and a manual connection (Model-ViewModel). No, just no. This gives us all the downsides of both. The implementation problem is that all this magic binding requires a lot of behind-the-scenes complexity of exactly the same kind which led to classes and inheritance becoming huge headaches that we all moved away from as much as possible (this also impacts web development, where Meta's React.js framework seems to be very popular).

Testing

Even the problems with magic bindings hasn't dissuaded people from going further. However, the next level of complexity needs another diversion, this time to testing.

Testing is hard. It's very easy to write code and think "that looks fine", when it isn't. Even getting humans to try to use your code only reveals some of the issues, though even this is much better than just taking a developer's word for it. One of the things we do to account for this difficulty is automated tests. You'll spot the first problem immediately: we write programs to test programs. How do we know the tests are correct? We don't.

There's one school of thought which says we should write the tests first, before any other code. Running these tests must produce at least one fail, because the code does nothing; then you add code until the tests pass. This mindset is hard to get into.

Another school of thought is that we should aim to cover 100% of our code with tests. I prefer this one over writing the tests first and the code later, partly because of how I approach development in general (I develop an understanding of the problem as I write the code), and partly because I think this is a tighter constraint than writing the tests first (there are probably more possible tests than you can immediately think of).

Now, why is this relevant? Well: how do you architect those tests? If you want to test a payment system, you don't want every single run of every single test of your system to end up triggering a real payment — these things should run multiple times per day per developer, more likely multiple times per hour per developer.

This leads to breaking down methods and functions into smaller and smaller units that do less and less stuff, and which "inject" the things they depend on as arguments. (This is called "dependency injection"). Dependency injection is a nice solution to an abstraction which isn't nice but is kinda necessary.

Baroque architectures

Now it's time to fold this diversion back into the previous topic of architectural design patterns: that some of these tests are better done with very narrow scope has led to people saying that all code needs to be architected with that kind of division in mind. As I wrote earlier, I've seen some developers hold that more than 3 lines of code in a function is too many lines — it seems absurd, but this is why they call for it. Unfortunately, it doesn't just seem absurd, it is absurd: breaking things down to that level means that there's more glue than function, and the glue is harder to test, is harder to follow and understand, and is harder to change.

For a specific example of this, consider the VIPER architectural design patterns:

This is too many parts, and the divisions don't represent the real logical splits: The Presenter and the Interactor both alter data, but are not themselves part of the data model; the Presenter is taking on two responsibilities but then forwarding both to other units; the Router is doing purely view-related tasks and could be part of the View (for example, when using Xcode to make UIKit/Interface Builder based apps, this would be a "segue"); and the Interactor is doing things to the Entity which should all be methods on the Entity itself in any language which supports this — it's the fundamental thing which defines the "Object" in "Object Oriented programming".

Analytics

And one final, extra, point: Analytics. You all know the annoying popups for consent to track you. Why do companies bother with them, when all users hate them? Because analytics reveal what users do, which parts of the app they engage with, what they have difficulty with — something you may not know as a normal user, is that they can easily track how far you scroll down the screen, which buttons you tap on, and exactly where your finger (or mouse) pressed (or clicked) on the screen. This is sold as "helping developers discover when buttons are too small to click on" and similar. That said, even though I am a developer, I am just as confused as everyone else about why on earth some websites have 1200 "partners" to "help" them analyse all this.

This website you're reading now (unless someone copy-pasted it), didn't show you a pop-up. This is due to the simple trick of… not tracking anything. I have no idea how many of you are reading this. I don't know your age demographics (which YouTube apparently provides, but I've not looked into that despite having a YouTube account). I don't know, or care, how far you scrolled. I don't know which countries you're all from, which my previous blogging platform, wordpress, told me.

But at the time of writing, this attitude seems to be rare: me, some other random bloggers, HackerNews, and possibly GitHub.

The Problem

It's all just too much: there's far too many things in software for any single developer to know it all; because it can't be known, we keep re-inventing the wheel, and each re-invented wheel has to re-learn the lessons of all the predecessors; this happens at multiple levels of abstraction all at the same time, so we get web frameworks that use JavaScript to handle loading of links and the display of images even though those things are built into the pre-JavaScript standard of HTML; and we have websites with so much excess stuff bundled with them that users would genuinely be better off if the site was a static image.

The thing is, I've seen complaints like this from day one:

So, am I just being an old man shaking their fist at a cloud? I think it all comes down to the outcomes: If the code is performant, if it's cheap to produce, if it's not prone to errors, does it really matter what's going on inside? The problem is only if one or more of those are absent or insufficient.

Well, looking at the apps and services around me, I don't think we have a fantastic state of affairs in any of those things:

Now, one might argue these are unavoidable costs created by "doing it right". When I was at university, there was "programming in the large vs. programming in the small" (when Java was popular despite having exactly the complex class names and hierarchies previously mentioned) was analogised with the difference between building a bridge with a few planks of wood vs. building a suspension bridge — A suspension bridge over a small ditch looks silly, a few planks of wood over the Golden Gate won't even support itself let alone traffic.

Likewise from the perspective of user expectations, there have been many changes in the transition from software of the 1990s, where you buy software once from a physical store, it runs on one machine which has one user, there are no security requirements, and you get to print system requirements on the side of the box it's sold in and it's the user's problem not the developers if the user doesn't have a 640x480 monitor that supports 256 colours, and it will be used in an office with constant lighting; to today, where you're expected to ship regular updates to justify the monthly subscription plan you're selling it on, and the users have a huge range of display sizes and orientations which are expected to be able to change dynamically during use, and the app has to switch between bright and dark modes depending on the time of day, and we have rules requiring support of people with various disabilities.

The converse point is another anecdote from university, when we were doing practice interview questions, and I was forced to justify what I meant by having written "committed to quality"; the me from 20 years ago just thought this meant "be as good as possible", but the pannelist (who had actual experience hiring people) had a better model: his example was to ask me:

"Which is better, a Porche or a go-kart?"

"The Porche, obviously"

"But what if you're a kid who just wants to get down a hill? You don't have a driving licence or even lessons, you're definitely not insured, you just want to have fun. You can't — don't know how and are not allowed — to use the Porche. Now? Now the best is the go-kart".

(All paraphrased, of course. Couldn't afford to record these sessions back in 2003 or so).

What to change?

Don't throw the baby out with the bathwater. Be mindful of Chesterton's fence.

The fences were made for a reason: software is easy to do wrong. The bathwater isn't the mere existence of abstractions, it's that there are too many at every level, and that even when they are very useful in specific cases, they are cargo-culted as silver bullets.

You aren't gonna need it

You, specifically, almost certainly don't need:

What to keep?

Based on my experience, the following are all genuinely useful, and must be kept:

Has anyone been here before?

Yes! When I was at university, we were taught about "extreme programming":

As with all trendy philosophies, even groups that say they follow it tend to only follow only about half of the things it calls for.

Let's bring back XP. (No, not that one).

Tags: Coding, Opinion, Programming, software engineers, Technology

Categories: Opinion, Professional, Technology

© Ben Wheatley — Licence: Attribution-NonCommercial-NoDerivs 4.0 International