Links - 24^th March 2019

If you did so, that would look a lot like ZFS, which is a non-traditional filesystem that has squashed those layers into the filesystem itself. ZFS handles RAID, encryption, pools for volume management, and so on. There clearly is no separation of concerns for ZFS. Is that a good thing? It does lead to better error detection and recovery because ZFS is concerned with the block layer all the way up to the files themselves; it has checksums at each level. ZFS can also do deduplication more easily and take faster snapshots.

[…]

To explain what is going on with ZFS, Anderson introduced the idea of a “sympathetic abstraction”. It can be thought of as an “intentional leakiness” where layers are sympathetic to those above and below. For example, ZFS is sympathetic to whether the block device it runs on is spinning rust or an SSD. ZFS is also sympathetic to errors that can occur on the raw block device, while ext4 and other traditional filesystems are not: they leave that kind of error handling to some other layer below.

These kinds of sympathetic abstractions happen all the time, he said. For example, SSH sets the TCP_NODELAY option (to disable Nagle’s algorithm) for interactive sessions. But SSH should not care what transport it is running on; it should be able to run on top of UDP tunneled inside DNS packets. The Apache Cassandra database wants to know if it is running on spinning rust or SSD; shouldn’t that be the concern of layers below it?

He introduced “Kyle’s Abstraction Law”, which he has come up with, though it is “not canon”, he cautioned. It says that over time, layers tend toward becoming more sympathetic (or merged) in the pursuit of performance. We start off with “beautiful layers”, each with a single purpose; over time, we realize the picture is not quite so clear cut, so we squash the layers some.

However, being scrappy is not without its disadvantages. Hardening alert configurations and expanding them to detect a wide array of potential outages is labor intensive work, and it’s an ongoing challenge to prioritize better monitoring over feature development. The consequences of a small team expand beyond prioritization, and providing 24/7 coverage with only two or three engineers can be a punishing experience.

Large companies (1,000-10,000 employees)

At this scale a company has become truly massive, moving from a handful of on-call rotations to dozens of them. Outages are immensely expensive, and when one does occur, prioritize recovery over understanding, as it’s too expensive to debug in production.

Facebook and twitter are trying to be third places but they are the shopping malls of social media you are not allowed to take action to customize or improve them at a structural level they’re owned by mega corporations that decide what you need FOR YOU, without ASKING YOU and games have the potential to be an alternative to that!

so games — social games like minecraft and fortnite — are becoming our NEW DIGITAL THIRD PLACES they are the community gathering spaces for a whole generation of kids who are growing up online and critically, these games work as third places in part because they enable players to change their environment!

At SoundCloud, we use what is canonically known as Kappa Architecture for our search indexing pipeline. The main idea here is: have an immutable data store and stream data to auxiliary stores.

So, how does this work? Once a user, for example, uploads a track or a playlist on SoundCloud, we receive a live update^[1] in the eventfeeder. Then the indexer fetches the uploaded data^[2] from our primary data source — MySQL — and creates a JSON document, which is published to Kafka. The Kafka topics have log compaction enabled, which means we will only retain the latest version of the documents. Finally, the shipper reads the JSON documents from Kafka and sends them to Elasticsearch via the bulk API.

Since we use Kafka to back the document storage, we can reindex a new cluster from scratch by simply resetting the Kafka consumer offset of the shipper component, i.e. replaying all stored messages in Kafka.