Links - 14^th November 2021

Going from zero to agreement using a meeting fails way more often than succeeds. You can build ground for consensus in hallways, small groups, backchannel chats, and informal conversation, far more effectively than in a 10-person meeting. Use meetings to formalize and confirm.

The terminal is a ubiquitous platform that has been fairly stable for many years. There are plenty of resources out there for understanding its inner workings, but most of them are either fairly arcane or offer deep knowledge about a very specific area. This post aims to bridge this gap by offering a gentle and broad introduction to the terminal emulator as a platform for development.

We’ll talk about the different parts of the terminal and how they interact, build a small program to read input from the shell and understand how it’s interpreted, discuss how to create a user interface in the terminal and finally see how we can use all of this to cause some mischief.

That’s the opposite of what I’ve seen at two of the three big tech companies I’ve worked for, where the median person avoids touching problems outside of their team’s mandate like the plague, and someone who isn’t politically savvy who brings up a problem to another team will get a default answer of “sorry, this isn’t on our roadmap for the quarter, perhaps we can put this on the roadmap in [two quarters from now]”, with the same response repeated to anyone naive enough to bring up the same issue two quarters later. At every tech company I’ve worked for, huge, extremely costly, problems slip through the cracks all the time because no one wants to pick them up. I never observed that happening at Centaur.

A side effect of big company tech culture is that someone who wants to actually do the right thing can easily do very high (positive) impact work by just going around and fixing problems that any intern could solve, if they’re willing to ignore organizational processes and incentives.

Within each normal-sized team, limit the choice of alternatives for any class of technology to three. These are: the current sensible default, the one we’re experimenting with as a trial, and the one that we hate and want to retire.

Domain (strong force)	Vertical (weakened force)	Whole of org (weak force)
Systems that are owned within a single domain are relatively easy to change in a closely co-ordinated way. For example this means (slightly) less effort needs to be burned on maintaining backwards-compatibility of interfaces. Even “forbidden” approaches like database-level integration will have less impact within the systems in a single domain. Although we shouldn’t build a system that way deliberately, if it exists then we can contain the impact within a single domain.	Changes that must be co-ordinated across multiple domains within a vertical should be rare, but can be managed when absolutely necessary. Expand-contract changes to API contracts is quite effective where the impacts are contained within the vertical.	APIs and interfaces (e.g. events) that are published to the whole of MYOB must have the highest attention to published schemas, versioning, backwards compatibility, contracts, and deprecation strategy. The impact of highly-coupled integration (e.g. ETLs, database integration) is very high, and should be absolutely avoided.

Locking works extremely well for some software artifacts and extremely poorly for others. Developers who have worked on artifacts for which locking works well can be expected to have a much more positive opinion of locking than those who have worked on artifacts for which locking works poorly.

Unfortunately, as we have seen, it is difficult to predict whether or not a given validation effort will find important bugs. It is therefore all too easy to invest too little — or even to fail to invest at all, especially if development estimates proved overly optimistic or budgets unexpectedly tight, conditions which almost always come into play in real-world software projects.

The decision to nevertheless invest in validation is often forced by experienced people with forceful personalities. But this is no guarantee, given that other stakeholders might also have forceful personalities. Worse yet, these other stakeholders might bring stories of expensive validation efforts that nevertheless allowed embarrassing bugs to escape to the end users. So although a scarred, grey-haired, and grouchy veteran might carry the day, a more organized approach would perhaps be more useful.

In short, if you are working on a software project that is intended to help people you know nothing about, you should not be surprised when those people find fault with your project.

This section is adapted from portions of Rusty Russell’s 2003 Ottawa Linux Symposium keynote address, Slides 39–57. Rusty’s key point is that the goal should not be merely to make an API easy to use, but rather to make the API hard to misuse. To that end, Rusty proposed his “Rusty Scale” in decreasing order of this important hard-to-misuse property.

The following list attempts to generalize the Rusty Scale beyond the Linux kernel:

1. It is impossible to get wrong. Although this is the standard to which all API designers should strive, only the mythical dwim()^[1] command manages to come close.

2. The compiler or linker won’t let you get it wrong.

3. The compiler or linker will warn you if you get it wrong. BUILD_BUG_ON() is your users' friend.

4. The simplest use is the correct one.

5. The name tells you how to use it. But names can be two-edged swords. Although rcu_read_lock() is plain enough for someone converting code from reader-writer locking, it might cause some consternation for someone converting code from reference counting.

6. Do it right or it will always break at runtime. WARN_ON_ONCE() is your users' friend.

7. Follow common convention and you will get it right. The malloc() library function is a good example. Although it is easy to get memory allocation wrong, a great many projects do manage to get it right, at least most of the time. Using malloc() in conjunction with Valgrind moves malloc() almost up to the “do it right or it will always break at runtime” point on the scale.

8. Read the documentation and you will get it right.

9. Read the implementation and you will get it right.

10. Read the right mailing-list archive and you will get it right.

11. Read the right mailing-list archive and you will get it wrong.

12. Read the implementation and you will get it wrong. The original non-CONFIG_PREEMPT implementation of rcu_read_lock() is an infamous example of this point on the scale.

13. Read the documentation and you will get it wrong. For example, the DEC Alpha wmb instruction’s documentation fooled a number of developers into thinking that this instruction had much stronger memory-order semantics than it actually does. Later documentation clarified this point, moving the wmb instruction up to the “read the documentation and you will get it right” point on the scale.

14. Follow common convention and you will get it wrong. The printf() statement is an example of this point on the scale because developers almost always fail to check printf()'s error return.

15. Do it right and it will break at runtime.

16. The name tells you how not to use it.

17. The obvious use is wrong. The Linux kernel smp_mb() function is an example of this point on the scale. Many developers assume that this function has much stronger ordering semantics than it actually possesses. Chapter 15 contains the information needed to avoid this mistake, as does the Linux-kernel source tree’s Documentation and tools/memory-model directories.

18. The compiler or linker will warn you if you get it right.

19. The compiler or linker won’t let you get it right.

20. It is impossible to get right. The gets() function is a famous example of this point on the scale. In fact, gets() can perhaps best be described as an unconditional buffer-overflow security hole.