Links - 8^th May 2022

Just to have it upfront, I will start with what The Tower Of Weakenings is, and then go back to the motivation and ideas behind it. The Tower Of Weakenings is simply the idea that having One True Memory Model is a frustrating and futile endeavour that leaves no one satisfied. Instead, we should have a tower of Memory Models, with the ones at the top being “what users should think about and try to write their code against”. As you descend the tower, the memory models become increasingly complex or vague but critically always more permissive than the ones above it. At the bottom of the tower is “whatever the compiler actually does” (and arguably “whatever the hardware actually does” in the basement, if you care about that).

Here is a sketch of what the tower looks like under my proposal:

1. The “Clean” Memory Model: a painfully strict and simple model that you can teach and check.

2. The “Real” Memory Model: similar to the “Clean” one, but messier to allow for Useful Crimes.

3. The Compiler’s Semantics: whatever random primitives compilers expose, and optimizations they do.

In some sense the bottom of the tower is the one that “matters” because that’s the thing that (mis)compiles your code, but it’s also a shifting target, and trying to expose its semantics leads to sadness. Unfortunately, this is also basically true of “real” memory models. We’re still trying to figure out what on earth C’s memory model is, and every language infamously defers to “the C11 memory model” on hard questions. So as an educator, I eventually have to bottom out on sending you a shrug-emoji if you keep asking for what the real rules are. No one really knows!

Ok yes get your jokes out about solving all problems with another layer of abstraction, but this is painfully needed. And really, what I’m doing here isn’t adding a layer but cleaving a bloated and messy layer into two much more manageable parts. No offense to all the people doing great work on memory models, but as far as I’m concerned they’ve been given an impossible task. There are simply too many competing concerns and stakeholders to produce a truly satisfying design for all of them. Here are some of the many stakeholders:

• Millions of lines of ancient code that are doing whatever because we can’t explain the rules

• Random programmers who barely read the docs and are just doing The Idiomatic Thing

• Hardcore performance junkies who insist the fucked up thing they made should be legal

• Hardcore safety junkies who insist they should be able to validate the correctness of their code

• Compiler developers who want to know what optimizations they can or can’t do

• Library developers who want to know what interfaces/idioms to design around

• Tooling developers who want to build sanitizers that reliably detect UB when it happens

• Teachers who want to help everyone else understand the PhD thesis on their desk.

• Memory model people who just want this all to be coherent and formalized and validated

And all of these people are working together so they all agree that whatever the other person claims they need to do their job should definitely be allowed, because they want to all mutually benefit. And definitely make sure not to break all that code or even make it run slower! kthxbai!

Critically, the fact this lie is part of The Tower Of Weakenings means that if you follow its rules, then whatever the “real” model is doesn’t matter anymore. We will absolutely guarantee that your code works, because you followed the really strict and simple rules that everyone can agree obviously have to work. Just as you’re not supposed to worry about the compiler backend you’re using, you shouldn’t have to worry about the fiddly details of what the current working memory model is.

I have been a big fan of the write libraries, not frameworks argument for a while now. Lately, I’ve come to ponder that there might be a fruitful expansion to this argument, that we should start to value principles over patterns, patterns over libraries, and libraries over frameworks.

Let’s clarify some terminology:

• Framework: This is (usually) someone else’s code that calls your code. In order for this to work your code will need to conform to constraints set down by the framework. These constraints are often firm boundaries that’s hard to code around. On the flipside by coding within the framework’s conventions and constraints you tend to get a lot of useful functionality out of the box making coding quicker.

• Library: This is (usually) someone else’s code that you call from your code. A library tends to be some code that imposes fewer constraints on your code as compared to frameworks. By using one or more libraries you’re able to reuse someone else’s code to solve your problems. Libraries are easier to combine and interchange, while putting frameworks on top of frameworks can lead to a bad time.

• Pattern: This is a descriptive, reusable approach to writing your code (see: software design pattern). Patterns range in their vaguenes, applicability and prescriptiveness. Examples of programming patterns include early-return pattern, builder-pattern, actor model, model-view-controller, onion architecture, microservices, majestic monolith, monorepos, and flux architecture. These are just some patterns that I can think off from the top of my head, but there are many books that cover programming patterns.

• Principle: This is some general guidance or philosophy expressed as rules (like a rule of thumb) that helps you write good code. Although programming is a very young field — for instance compare this to the field of carpentry who have been building houses for thousands of years — developers have managed to distill some useful principles to aid themselves in their work. Examples of principles include SOLID principles, Don’t repeat yourself (DRY), You Aren’t Gonna Need It (YAGNI), Agile manifesto, Law of Demeter, Hyrum’s law. There are many more to be found out there, and as I write this I just discovered this site that collects programming principles, nice!

Like previous developments in the computing industry, Agile combined counterculture and cyberculture; it was ostensibly rebellious, but committed rebels to sprinting toward corporate goals.

Given the shortage of qualified developers, technology professionals might have been expected to demand concessions of more immediate material benefit — say, a union, or ownership of their intellectual property. Instead, they demanded a workplace configuration that would allow them to do better, more efficient work. Indeed, as writer Michael Eby points out, this revolt against management is distinct from some preceding expressions of workplace discontent: rather than demand material improvements, tech workers created “a new «spirit», based on cultures, principles, assumptions, hierarchies, and ethics that absorbed the complaints of the artistic critique.” That is, the manifesto directly attacked the bureaucracy, infantilization, and sense of futility that developers deplored. Developers weren’t demanding better pay; they were demanding to be treated as different people.

Anti-management, maybe, but not anti-corporate, not really. It’s tempting to see the archetypal Agile developer as a revival of the long-haired countercultural weirdo who lurked around the punch card machines of the late 1960s. But the two personas differ in important respects. The eccentrics of computing’s early years wanted to program for the sheer thrill of putting this new technology to work. The coder of Agile’s imagination is committed, above all, to the project. He hates administrative intrusion because it gets in the way of his greatest aspiration, which is to do his job at the highest level of professional performance. Like the developers in Aaron Sorkin’s The Social Network, he wants most of all to be in “the zone”: headphones on, distractions eliminated, in a state of pure communion with his labor.

Hindsight bias is something somewhat similar to outcome bias, which essentially says that because we know there was a failure, every decision we look at that has taken place before the incident will seem to us like it should obviously have appeared as risky and wrong. That’s because we know the result, it affects our judgment. But when people were going down that path and deciding what to do, they were trying to do a good job; they were making the best calls they could to the extent of their abilities and the information available at the time.

We can’t really avoid hindsight bias, but we can be aware of it. One tip there is to look at what was available at the time, and consider the signals that were available to people. If they made a decision that looks weird, then look for what made it look better than the alternatives back then.

Another one also came from a previous job where an engineer kept accidentally deleting production databases and triggering a whole disaster recovery response. They were initially trying to delete a staging database that was dynamically generated for test cases, but kept fat-fingering the removal of production instances in the AWS console. Other engineers were getting mad and felt that person was being incompetent, and were planning to remove all of their AWS console permissions because there also existed an admin tool that did the same thing safely by segmenting environments.

I ended up asking the engineer if there was anything that made them choose the AWS console more than the admin tool given the difference in safety, and they said, quite simply, that the AWS console has an autocomplete and they never remembered the exact table name, so it was just much faster to delete that table often there than the admin. This was an interesting one because instead of blaming the engineer for being incompetent, it opened the door to questioning the gap in tooling rather than adding more blockers and procedures.

I reached out to the engineers in question and asked about what made them feel like they had enough tests. I said that we often write tests up until the point we feel they’re not adding much anymore, and that I was wondering what they were looking at, what made them feel like they had reached the points where they had enough tests. They just told me directly that they knew they didn’t have enough tests. In fact, they knew that the code was buggy. But they felt in general that it was safer to be on-time with a broken project than late with a working one. They were afraid that being late would put them in trouble and have someone yell at them for not doing a good job.

And so that revealed a much larger pattern within the organization and its culture. When I went up to upper management, they absolutely believed that engineers were empowered and should feel safe pressing a big red button that stopped feature work if they thought their code wasn’t ready. The engineers on that team felt that while this is what they were being told, in practice they’d still get in trouble.

There’s no amount of test training that would fix this sort of issue. The engineers knew they didn’t have enough tests and they were making that tradeoff willingly.

I’ve seen problems where folks were trying valiantly to move forward with the wrong abstraction, but having very little success. Adding new features was incredibly hard, and each success further complicated the code, which made adding the next feature even harder. When they altered their point of view from “I must preserve our investment in this code” to “This code made sense for a while, but perhaps we’ve learned all we can from it”, and gave themselves permission to re-think their abstractions in light of current requirements, everything got easier. Once they inlined the code, the path forward became obvious, and adding new features become faster and easier.

The moral of this story? Don’t get trapped by the sunk cost fallacy. If you find yourself passing parameters and adding conditional paths through shared code, the abstraction is incorrect. It may have been right to begin with, but that day has passed. Once an abstraction is proved wrong the best strategy is to re-introduce duplication and let it show you what’s right. Although it occasionally makes sense to accumulate a few conditionals to gain insight into what’s going on, you’ll suffer less pain if you abandon the wrong abstraction sooner rather than later.

When the abstraction is wrong, the fastest way forward is back. This is not retreat, it’s advance in a better direction. Do it. You’ll improve your own life, and the lives of all who follow.

As you can see a lot revolves around determining how many index tuples will be matched by the scan — as that’s the main expensive portion of querying a B-tree index.

The first step is determining the boundaries of the index scan, as it relates to the data stored in the index. In particular this is relevant for multi-column B-tree indexes, where only a subset of the columns might match the query.

You may have heard before about the best practice of ordering B-tree columns so the columns that are queried by an equality comparison (= operator) are put first, followed by one optional inequality comparison (<> operator), followed by any other columns. This recommendation is based on the physical structure of the B-tree index, and the cost model also reflects this constraint.

Put differently: the more specific you are with matching equality comparisons, the less parts of the index have to be scanned. This is represented here by the calculation of btreeSelectivity. If this number is small, the cost of the index scan will be less, as determined by genericcostestimate based on the estimated number of index tuples being scanned.

For creating the ideal B-tree index, you would:

• Focus on indexing columns used in equality comparisons

• Index the columns with the best selectivity (i.e. being most specific), so that only a small portion of the index has to be scanned

• Involve a small number of columns (possibly only one), to keep the index size small — and thus reduce the total number of pages in the index

If you follow these steps, you will create a B-tree index that has a low cost, and that Postgres should choose.

Your job title says “software engineer”, but you seem to spend most of your time in meetings. You’d like to have time to code, but nobody else is onboarding the junior engineers, updating the roadmap, talking to the users, noticing the things that got dropped, asking questions on design documents, and making sure that everyone’s going roughly in the same direction. If you stop doing those things, the team won’t be as successful. But now someone’s suggesting that you might be happier in a less technical role. If this describes you, congratulations: you’re the glue. If it’s not, have you thought about who is filling this role on your team?

Every senior person in an organisation should be aware of the less glamorous — and often less-promotable — work that needs to happen to make a team successful. Managed deliberately, glue work demonstrates and builds strong technical leadership skills. Left unconscious, it can be career limiting. It can push people into less technical roles and even out of the industry.

Let’s talk about how to allocate glue work deliberately, frame it usefully and make sure that everyone is choosing a career path they actually want to be on.

What does this have to do with management? Well, new managers often present exhausted and overwhelmed, and the question I often ask in this situation, is, “how are you house elfing your team?”

House elfing comes from a good place, often tied to some idea of “servant leadership”. People who internalize this idea that they exist to work for their team, and the way they know how to do that is to pick up all the small annoying things, run all the meetings, plan all the team activities, pick up the boring grunt work, tidy up the bug list etc.

The outcome of this is that they are:

• Wholly reactive: unable to focus on bigger / more impactful work.

• Buried in small details: unable to step back and see the bigger picture.

• Exhausted: running around all day picking up after people does that to you.

• Overwhelmed: see also: reactive. By being buried in the details, you don’t have time to make the meaningful improvements.

Worse, these managers often start thinking it’s their job to make their team happy. Wrong! It’s your job to make your team effective. Constant picking up of small things does not make your team more effective — noticing the patterns and improving the processes, or the projects themselves does that.

A second interesting age-related finding from the new study is that different fonts performed differently for young and old readers. The authors set their dividing line between young and old at 35 years, which is a lower number than I usually employ, but possibly quite realistic given the age-related performance deterioration they measured.

3 fonts were actually better for older users than for younger users: Garamond, Montserrat, and Poynter Gothic. The remaining 13 fonts were better for younger users than for older users, which is to be expected, given that younger users generally performed better in the study.

The takeaway is that, if your designers are younger than 35 years but many of your users are older than 35, then you can’t expect that the fonts that are the best for the designers will also be best for the users.

Also, the differences in reading speed between the different fonts weren’t very big for the young users. Sure, some fonts were better, but they weren’t much better. On the other hand, there were dramatic differences between the fastest font for older users (Garamond) and their slowest font (Open Sans). In other words, picking the wrong font penalizes older users more than young ones. The same takeaway applies: if the designers are young, they may not experience much reading-speed differences between different fonts, leading them to make design decisions based on mainly aesthetic criteria and assuming legibility to be less important. But those fonts that seem pretty much equally legible to young people can have vastly different legibility for older people. (And remember that “old” was defined as 35 years or above in this study.)

I’ve come to differentiate a small database from a large one using the following caveats. And while it is true that some of the caveats for a large database can be applied to a small one, and vice-versa, the fact of the matter is that most of the setups out there in the wild follow these observations:

1. Small databases are often administered by a single person.

2. Small databases can be managed manually.

3. Small databases are minimally tuned.

4. Small databases can typically tolerate production inefficiencies more than large ones.

5. Large databases are managed using automated tools.

6. Large databases must be constantly monitored and go through an active tuning life cycle.

7. Large databases require rapid response to imminent and ongoing production issues to maintain optimal performance.

8. Large databases are particularly sensitive to technical debt.

Large databases often bring up the following questions and issues:

• Is the system performance especially sensitive to changes in production load?

• Is the system performance especially sensitive to minor tuning effects?

• Are there large amounts of data churn?

• Does the database load system saturate your hardware’s capabilities?

• Do the maintenance activities, such as logical backups and repacking tables, take several days or even more than a week?

• Does your Disaster Recovery Protocol require having a very small Recovery Point Objective (RPO) or Recovery Time Objective (RTO)?

The key difference between a small vs large database is how they are administered:

1. Whereas it is common that small databases are manually administered, albeit it’s not best practice, using automation is the industry default mode of operation in many of these situations for large databases.

2. Because circumstances can change quickly, large databases are particularly sensitive to production issues.

3. Tuning is constantly evolving; while it is true that newly installed architectures are often well-tuned, circumstances change as they age and large databases are especially vulnerable.