It's taken me a while to wrap my head around how security might work in a completely
decentralized, peer-to-peer application. I'll dig into the big questions around local-first
authentication and authorization and explain how I've approached solving these problems in a new
library, <a href='https://github.com/local-first-web/auth'>@localfirst/auth</a>.

Establishing trust in the strange world of peer-to-peer collaboration

Alice and Bob in wonderland

There are a number of reasons why we don’t live in a local-first world yet, which can mostly can be
summarized as “there are a bunch of things we’re still figuring out how to do without a server.”
One of the things that makes a server a server is that it’s authoritative. By that we mean that
we’ve agreed that it is our single source of truth. An application built around a single central
server gets consistency for free, meaning that we can edit shared data without having to worry
about conflicts.
This is maybe the first problem that comes to mind for most people when thinking about peer-to-peer
collaboration, and it’s the one that has generated the most attention and research: How do you
resolve conflicts when two peers try to sync up? Without an authoritative server, we don’t get
consistency for free. If two peers are equally authoritative and they diverge — there are
differences between the data on their respective devices — then how do we resolve those
discrepancies and get them back in a consistent state?
After spending much of the last couple of years thinking about this and working on this, I feel like
I have a pretty solid handle on how this problem is solved, and I’d like to explain it as simply as
possible — because, despite appearances, the solution is fairly straightforward.
The solution involves something with the fearsome name of conflict-free replicated data types
(CRDTs). These were first defined in 2011 and have been the subject of a lot of research since then.
The academic papers on the subject might be interesting and they serve to put the field on a solid
theoretical footing; but you don’t need to understand what a semilattice is, or be able to follow
the formal correctness proofs, in order to understand how this all works.
But do keep in mind that decentralized systems work very differently from centralized systems, and
so there are some very basic things that we take for granted that we have to think about in a very
different way.
<h3 id=""></h3>
Let’s start out by talking about how traditional cloud-based applications work and why we don’t have
to worry about consistency in that world.
Let’s suppose Alice and Bob are using a cloud-based project management tool – think of something
like Basecamp, or Asana, or Monday. They both have the same permissions. Alice assigns a task to
Bob; at about the same time, Bob assigns the same task to Alice. Which assignment is kept?
Clearly, it depends on which change is most recent – in other words, the last change “wins”. If
Alice makes her change at 10:01 and Bob makes his change at 10:02, then if I go to look at the task
at 10:03, I’m going to see Bob’s edit.
Even if they make their changes at precisely the same time and both hit <code>Enter</code> simultaneously, one
change will randomly get to the server first, and the later change will be the one that’s kept. Of
course, Alice and Bob could get into an “edit war”, assigning the task back and forth like two
Wikipedia editors who disagree on who killed JFK. But once the dust settles, whatever assignment was
entered last is the one that you and I will see the next day; there’s no possibility that you and
I will see different assignments. Our views of the data are guaranteed to be consistent, simply
because there’s only one authoritative version of the data, and it can only have one value for that
field.
<h3 id="what-is-time-anyway">What is “time”, anyway?</h3>
But in a local-first world, we only have peers: The very term implies equality. One peer is
presumed to be just as authoritative as another.
Now, if Bob and Alice are connected, then last-write-wins still applies. The difficulty comes when
Alice and Bob make their changes while disconnected: Say Alice is working offline while on a plane,
and she assigns the task to Bob. While she’s still in the air, Bob assigns the task to Alice. When
Alice lands and reconnects to Bob, which assignment is kept?

All about CRDTs

Consistency without a central authority

As an American living in Spain, every once and a while I’m mildly surprised to be reminded that Spain still has a king (and a queen, and two little teenage princesses) whose lineage — and thus his authority — can be traced back to the dark ages.
<img src="https://firebasestorage.googleapis.com/v0/b/firescript-577a2.appspot.com/o/imgs%2Fapp%2Fherbcaudill%2Fs0Fj3TOGjF.png?alt=media&token=5af24a27-a3da-4cbc-bdae-8663d0e9f23e" alt="">
<img src="https://firebasestorage.googleapis.com/v0/b/firescript-577a2.appspot.com/o/imgs%2Fapp%2Fherbcaudill%2FK-kSwVAllu.png?alt=media&token=111ce383-7e03-4b2b-9884-d49a2c17d586" alt=""> - <img src="https://firebasestorage.googleapis.com/v0/b/firescript-577a2.appspot.com/o/imgs%2Fapp%2Fherbcaudill%2F_t0auUMmV-.png?alt=media&token=f22103fd-ec93-42a6-8c3c-93bd32a284b5" alt="">
Back when some humans first got the notion of replacing monarchs with more democratic forms of governments, most other humans thought this was a preposterous suggestion.
For starters, it took a while for the idea to sink in that kings and queens were actually just people like the rest of us.
But even once that was conceded, there were still legitimate questions to be answered. Monarchs played a number of roles, both practical and symbolic, that we would need to fulfill some other way. Who would lead us into battle? Who would be the decider of last resort in contentious court cases? Who would unify us as a nation? Who would represent us to the world?
<h2 id="this-is-a-metaphor">This is a metaphor</h2>
If we’re going to get rid of servers, we’ll need to understand the utility that they currently provide, and figure out different ways of fulfilling those needs.
<h4 id="servers-are-just-computers">Servers are just computers</h4>
The first thing to understand, though, is that servers are just computers. Just like any human could, in principle, serve as a leader, any computer could be a server.
I think maybe in the popular imagination, we need servers because they’re bigger and more powerful than consumer-grade computers. That’s certainly the case for some applications.
But as CPUs and GPUs get faster and faster and SSDs get bigger and bigger and everything gets cheaper and cheaper, there are fewer and fewer applications that are too big or too complex to run on users’ own devices.
<h2 id="what-servers-are-good-for">What servers are good for</h2>
Servers do a lot, but the value they add may not be what you think it is.
I think maybe in the popular imagination, we need servers because they’re bigger and more powerful than consumer-grade computers. That’s certainly the case for some applications. But as CPUs and GPUs get faster and faster and SSDs get bigger and bigger and everything gets cheaper and cheaper, there are fewer and fewer applications that are too big or too complex to run on users’ own devices.
So for the moment let’s disregard use cases that involve vast amounts of data or server farms’ worth of computation. Instead, think of something like a community group’s project management system, or a medium-sized company’s financial records, or a sales team’s database of leads and deals and customers.
Even with these modest applications, we depend on servers to make collaboration happen. I’ll divide the essential services they provide into five big categories: consistency, discovery, persistence, authentication, and authorization.
Consistency A server provides a single central point of contact for collaborative applications to read and write data. The data in the cloud acts as our single source of truth. There’s no possibility of two people making conflicting change
Discovery There are a couple of things that turn an ordinary computer in to a server. One is that it has a stable IP address. This makes it easy for two users to find each other in the vastness of the internet. I connect to a URL from wherever I am in the world, you connect to the same URL from wherever you are, and just like that we’re connecte
Persistence Another thing that makes a computer a server is that it’s always on and always online. Typically there are teams of skilled people who have that as their only job. This means that we can collaborate asynchronously: I can make some changes, log off and go to sleep, and when you come online you’ll see my change
Authentication A server can securely store sensitive information — passwords, for example — and then use that to challenge me to prove who I am. If I can’t prove that I know my password, it can refuse to connect with me, and thereby prevent me from connecting to others. So the server acts as trusted third party, essentially vouching for my identity to my collaborator
Authorization If there’s something I’m not allowed to see, a server just won’t let me have it; and if there’s something I’m not allowed to do, a server simply won’t allow me to do i
<h2 id="how-do-we-do-these-things-without-servers">How do we do these things without servers?</h2>
So if we want to build local-first software, we first need to figure out how to provide these services without a server.
<ol>
<li>Consistency Without a server, if we each have our own local copy of the data, how do we stay in sync? What if our edits are in conflict with each othe
<div class='spoiler'>
**A:** An emerging class of technologies called **conflict-free replicated data types (CRDTs)**
allows software on users' own devices to sync up directly with each other, automatically merging
changes on both sides, and resolving any conflicts in a predictable way.

 </div></li>
<li>Discovery Without a server, how can my laptop connect directly to your phone? Neither my laptop nor your phone has a stable IP address, so how do we find each other in order to communicat
<div class='spoiler'>

A: While we’re waiting for true peer-to-peer networking technologies to become mature and widespread, we can use relays — tiny, generic servers that can be deployed to the cloud at little or no cost — that allow two devices anywhere on the internet to instantly connect to each other.
 </div></li>
<li>Persistence Without a server, how do we communicate asynchronously? Our laptops and tablets and phones are often offline or powered down. We can’t really collaborate effectively if we always have to be online at the same tim
</li>
<li>Authentication Without a server acting as a trusted third party, who vouches for me so I can prove who I am to my collaborators? They don’t know my passwords (I hope!), so how can they be sure that I’m not an impostor
</li>
<li>Authorization Without a server to keep track of the rules, how dfo we know who’s allowed to do what? And if everybody has a full copy of the data, how can we keep some people from seeing sensitive information
</li>
</ol>
| | 💿 traditional apps | 🌧 cloud apps | 👍 local-first apps |
| ——————————————————- | :————————-: | :——————: | :————————-: | ——– |
| ⌚ fast: my data is on my device | ✅ | ⛔ | ✅ |
| 🚙 robust: network optional | ✅ | ⛔ | ✅ | **** |
| 🕵️‍♀️ private: only I decide who sees it | ✅ | ⛔ | ✅ |
| 📦 mine: I own it and can take it with me | ✅ | ⛔ | ✅ |
| 👪 multiplayer: I can work with others in real time | ⛔ | ✅ | ✅ |
| 📱 multidevice: move from laptop to phone to tablet | ⛔ | ✅ | ✅ |
| 🌩 backed up: no worries about data loss | ⛔ | ✅ | ✅ |
<table>
<thead>
<tr>
<th>A server is…</th>
<th align="left">because…</th>
<th align="left">as a result it can provide…</th>
<th align="left">which means we can…</th>
<th>In a peer-to-peer world, we can instead use…</th>
</tr>
</thead>
<tbody><tr>
<td>available</td>
<td align="left">it’s always plugged in, powered on, and online</td>
<td align="left">persistence</td>
<td align="left">collaborate asynchronously</td>
<td>cloud clients</td>
</tr>
<tr>
<td>findable</td>
<td align="left">it has a public IP address</td>
<td align="left">discovery</td>
<td align="left">find each other</td>
<td>relays</td>
</tr>
<tr>
<td>trusted</td>
<td align="left">it is under the control of a known entity</td>
<td align="left">authorization</td>
<td align="left">enforce read and write permissions</td>
<td>signature chains, lockboxes, and attention</td>
</tr>
<tr>
<td>a third party</td>
<td align="left">it is not a peer</td>
<td align="left">authentication</td>
<td align="left">be sure of each other’s identity</td>
<td>signature chains and signature challenges</td>
</tr>
<tr>
<td>authoritative</td>
<td align="left">we agree that it is the only source of truth</td>
<td align="left">consistency</td>
<td align="left">edit shared data without conflicts</td>
<td>CRDTs</td>
</tr>
</tbody></table>

Cloud-based software has made collaboration easier, but at a cost to both developers and users.
Could a new model for software give us the best of both worlds?


Lorem ipsum dolor sit amet, consectetuer adipiscing elit.

Is this a server?

Imagine you were trying to reach a mountain peak, in heavy fog, with no maps or trails. How would
you proceed?
One smart heuristic might be simply “head uphill, until you can’t get any higher”.
If you think about it for a moment, though, you might realize that there’s a problem with this
approach. Maybe if the mountain you’re climbing looks like Mt. Fuji, you’re fine - but what if, like
most mountains, it includes a number of smaller peaks at different altitudes?
<blockquote>
TK pictures of Mt. Fuji &amp; maybe Montserrat or Montarto
</blockquote>
If you only go up and never go down, you’ll very likely find yourself in what mathematicians call a
<a href="TK">local optimum</a>.
And if others are also using your same mountain-climbing heuristics, you might quickly find yourself
in the ridiculous situation of competing with a bunch of other people for this same false summit.
Here’s the thing, though - even if you can see higher peaks from where you are, it’s not altogether
irrational to keep heading for the nearer, lower peak. After all, you’re almost there. If everyone
else is there too, then the trail is well-worn so you won’t get lost. Going downhill just feels
wrong - especially while others are making progress uphill.
<blockquote>
THIS IS A METAPHOR
</blockquote>
I think that we in software industry have ended up in a local optimum. When I say the “software
industry”, I’m thinking specifically of those of us who make collaboration software: multiplayer
apps that help people work together over the internet.
In this metaphor the false summit, the local optimum, is cloud software, also referred to as
“web-based software” and “software as a service”. My hope with this article to persuade you that
there’s a higher peak out there - a better approach to creating collaboration software, which goes
by the name of local-first software. And I’d like to not only persuade you that this other peak
exists, but that we should be excited about it, that it’s worth the additional effort to get there.
<hr>
<h3 id="from-client-server-and-local-only-software-to-cloud-software">From client-server and local-only software, to cloud software</h3>
Twenty or thirty years ago, if I had an important document to write — say, a term paper or a
proposal - the only real choice was to fire up WordPerfect or Microsoft Word or MacWrite on my
computer and started typing.
But what if this wasn’t a solo project? What if I was working on a team? Maybe different people are
writing different sections. Maybe other people are helping with proofreading or editing.
Well, if you’re old enough to remember, you’ll know how this went: Before you know it everyone is
emailing each other files and you have 27 different conflicting versions, with filenames that only
grow longer and longer. One person might ask everyone to hold off on making any further changes
while they’re working on the document; but inevitably at some point you would still end up
laboriously re-applying edits from one document to another.
<blockquote>
TK alice/bob/charlie illustration?s
</blockquote>
Around this time, if you wanted multiplayer software, you generally had to host it yourself; so this
was mostly the province of bigger companies that had the kind of infrastructure to do this.
Accounting software, CRM tools to help sales teams work together, that kind of thing. Sprawling
“enterprise resource planning” systems intended to do everything for everyone, provided by giants
like SAP and Oracle. For the most part these were delivered as “client-server” software - the
company would host a server, and each user would install the client application, which you could
only use within the company’s local network.
<hr>
Today, instead I can use web software like Google Docs. This makes it super easy for me and my
teammates to work together on a document, without emailing files or worrying who has the latest
version.
<blockquote>
TK graph of SaaS replacing installed software?
</blockquote>
The story of software over the last couple of decades is largely the story of installed software
giving way to software in the cloud. You have Google Workspace and Microsoft 365. You have project
management tools like Basecamp, Asana, Monday.com, Trello, and Jira. You have QuickBooks and
FreshBooks. For shared notes, there’s Notion, Coda, and Roam. Designers have tools like Canva and
Figma. Sales teams have Salesforce, FreshSales, and Pipedrive.
All of these products have roughly the same architecture: Your data lives on the provider’s server,
not in files on your computer. You have an account and you log in with a username and password. The
technical architecture involves a database layer and at least a couple of codebases — one for the
server and one for the client, which is typically delivered as a web application, and perhaps also
as native apps for platforms like iOS, Android, MacOS, or Windows.
<h2 id="why-we-like-cloud-software">Why we like cloud software</h2>
I’m going to be talking about an alternative way of building software, but to be clear, the
cloud-based model has a lot going for it. I depend heavily on the software products I just
mentioned myself. Not only that, I’m a cloud software provider myself. My small company, DevResults,
makes web-based monitoring and evaluation software for international aid projects.
What’s good about this approach?
<h4 id="users-get-easy-collaboration">Users get easy collaboration</h4>
For the user, the big advantage of cloud software is having a single source of truth: Your data
lives in only one place, and that makes collaboration easier.
You never have to worry about upgrading your software — it’s continuously upgraded so you’re
always using the latest version.
You can access it from anywhere, and it’s backed up for you — you don’t have to worry about
losing data if something happens to your device.
<h4 id="providers-get-control-over-customers-and-their-data">Providers get control over customers and their data</h4>
The advantages for the software developer are even greater. It’s very convenient having everything
on your servers; it gives you a lot of control. It’s much easier to debug, because you have
access to all the data and there are fewer unknowns regarding the environment.
Once a customer starts to depend on your software, they’re kind of locked in. It’s very
difficult for them to leave, because you’ve got their data. If they don’t pay their bills, you can
shut off their access.
<h2 id="what-weve-traded-off">What we’ve traded off</h2>
So there’s a lot to like here. But everything is about trade-offs, and I think it’s worth thinking
about what we traded off when we moved everything to the cloud.
<h4 id="users-have-given-up-the-ability-to-work-offline">Users have given up the ability to work offline</h4>
For the user, the tradeoffs all have to do with the fact that our data, which used to live on
our own computer, is now inconveniently located hundreds or possibly thousands of miles away, on
someone else’s computer. For starters, this means that we have to have a good, working networking
connection in order to do anything at all. If we’re offline, or having network problems, we have
no access to our stuff.
<h4 id="and-responsiveness">…and responsiveness</h4>
And our own network connection is just the beginning: Every leg of the journey between you and your
data represents a possible point of failure. And even if everything is working perfectly and every
step takes place at the maximum possible speed, which it won’t, you’ll still experience some
latency with every interaction. You’re ultimately limited by the speed of light, which starts to
matter at planetary scale.
<h4 id="and-ownership-of-our-data">…and ownership of our data</h4>
So the “hundreds or possibly thousands of miles away” part is inconvenient, but the “someone else’s
computer” part is what’s really troublesome. “Possession is nine tenths of the law”, and in a very
real sense, my document on Google Docs is not mine, it’s Google’s. My project on Asana is not
mine. It lives on someone else’s hardware, not mine, and if they want to, they can keep it from me.
There’s always the threat of being locked out of my stuff because my credit card expired, or
because my product was discontinued, or the company went out of business, or just because someone
goofed up.
<h4 id="-and-our-privacy">… and our privacy</h4>
Putting my stuff on someone else’s computers also means that I have less control over who sees my
stuff.
Once a provider starts to accumulate lots and lots of people’s personal data on its servers, those
servers become an attractive target for attackers.
You have no guarantee that your cloud software company’s employees aren’t <a href="http://vice.com/en/article/xwnva7/snapchat-employees-abused-data-access-spy-on-users-snaplion">rifling through your
things</a>.
In the U.S., the fourth amendment to the constitution specifically protects my “papers and effects”
from “unreasonable searches and seizures”; and so if law enforcement wanted access to my phone or my
laptop, they would need a warrant, and I would have all the legal protections of due process. But my
stuff stored in the cloud is <a href="https://epic.org/ecpa/">not protected</a>: Not only can the government
require providers to hand it over with just a subpoena, but then they can issue a gag order that
prevents my providers from even letting me know about the search and seizure that’s taking
place.
Now maybe you feel like you have nothing to hide, and you’re blasé about the prospect of law
enforcement going through your stuff. If you’re like me, your “papers and effects” are pretty boring
and the idea that they would be of interest to Interpol or the FBI is kind of hilarious.
But if that’s the way you feel, I think it’s worth thinking about the very big picture, and
reflecting on what a rare luxury it is to live in a time in history, and a place in the world, where
you generally are not worried that people in power will abuse that power. The culture and the
institutions that have made that possible aren’t universal facts of life — they’re human constructs,
and they’re fragile, and they’re not available to everyone. If you’re a government whistleblower,
you can’t be blasé. If you’re an activist in Hong Kong, you don’t have that luxury.
<h4 id="as-a-society">As a society</h4>
And I’ll say one last thing about what we as consumers and as a society have given up.
When we’re talking about power, and abuses of power, at least when it comes to the government, in a
democracy, we have the rule of law and the protections of due process. If I’m accused of something,
I can get a lawyer, I can ask to see the evidence against me, and so on.
Something that’s starting to worry me more and more is that we depend so much on just a few tech
companies, that they end up with the kind of power over our lives that normally we reserve for
democratically elected governments. So much is at stake if you’re booted off Twitter or Google or
Amazon that these companies have what amounts to internal judicial systems to deal with various
kinds of misbehavior. But unlike the real thing, in these shadow judiciaries you have no rights.
Big tech doesn’t have to prove anything to you, or even tell you what you’re accused of.
So I think as a society, “we the people” should be worried about the quasi-governmental power that
we’re conceding, little by little, to unelected, unaccountable private companies.
<h4 id="developers-have-taken-on-lots-of-extra-complexity">Developers have taken on lots of extra complexity</h4>
So the shift to cloud software has resulted in us users giving up a lot of control to software
providers. But for software developers, cloud software has been a mixed bag as well — especially for
small startups, and for individual developers with a good idea.
Matthew Weidner makes this point vividly in <a href="http://youtube.com/watch?v=Exr0iY_D-vw">this talk</a>. He
asks us to imagine that we have an idea for a better recipe-sharing app.
The client application itself — the software that the user interacts with — isn’t complicated: You
could code it up in a weekend.
But to turn that into a multi-user cloud-based app?
For starters, you need servers, and those cost money.
And you need a database, and a whole new codebase on the server. Before you know it, most of your
code is just about getting data from the client to the server to the database and back, and adding
the smallest new feature requires tons of soul-crushing boilerplate code.
And you need to think about user accounts, and passwords, and resetting passwords, and securing
passwords.
Suddenly this is not a weekend project, it’s a startup in search of funding — all because of the
added cost and complexity that comes with creating a cloud-based app.
<h3 id="where-we-are-today">Where we are today</h3>
In sum: Cloud software has made it much easier for us to collaborate, and that’s a wonderful thing.
But that has come at a cost.
We’ve decided that we’re OK with paying $10 per month here and $25 per month there, every month of
our lives without fail until we’re dead and maybe afterwards as well.
We’ve given up our ability work offline.
We’ve given up much of our expectation of privacy in our digital lives.
We’ve given unaccountable companies the power to decide whether or not we deserve to have access to
our own stuff.
And we’ve raised the barrier of entry for new software products, further entrenching the power of
the incumbent software companies.
<h2 id="is-there-a-better-way">Is there a better way?</h2>
Maybe the value we’re getting from this software is worth the cost we’ve paid — I’m not saying it’s
not!
But I think it’s worth asking: Is this state of affairs inevitable? Or is it possible to imagine
another way, that would give us the benefits of collaboration without giving up ownership of our
digital lives?
The local-first model starts with the question: To collaborate, couldn’t our devices just talk to
each other directly?
You and I want to work together, and we both have computers. So what do we need a third party’s
computers for?
After all, today’s laptops are far more powerful than the beefiest servers from just a few years
ago. Even the phones in our pockets would vastly outperform the servers that the SaaS industry
started out with.
Going back to the word processing example we started out with: With good ol’ Microsoft Word, I
didn’t have to give up my privacy, or the ability to work offline, or the confidence that I’d still
have access to my document next year. The only reason I switched to Google Docs was because it made
collaboration easier.
But what if my Microsoft Word could talk to your Microsoft Word directly, allowing you and I
to work together on a document, Google Docs-style? We would both have the full document stored on
our devices, and the software would take care of keeping our copies synced up.
That’s the vision behind local-first software: Applications that are primarily local to your
device — giving back the control and agency you should have over your data — but that still enable
collaboration, via peer-to-peer communication with the local applications on other people’s devices.
I think that’s a really exciting vision of the future — one that would get us back everything we
traded away when we switch to cloud applications, without making us give up the ability to
collaborate smoothly. The best of both worlds.
But if you look around, you’ll see that very little software these days works that way. Why is that?
If this is such an attractive vision, why are we still building web apps? Why did Microsoft go to
the trouble of <a href="https://www.microsoft.com/en-ww/microsoft-365/free-office-online-for-the-web">rebuilding Word from the ground up as a web
application</a>, rather
than just adding collaboration capabilities to the existing program?
The short answer is that as an industry, we know how to build collaborative apps using a central
server, and we really haven’t figured out any other way to enable collaboration.
So if we want to build local-first applications without central servers, we need to understand
the jobs those servers are currently doing for us — and figure out a way to fulfill those needs
without them.

Collaboration without giving up control.

Seven reasons to be excited about local-first software

Cryptography is a fundamental piece of the digital infrastructure we all rely on, and yet most of
us are pretty fuzzy on how it actually works. I've written this for programmers — as well as
curious civilians — who would like a clearer idea of what the pieces are and how they fit together.


Everything you need to know about digital encryption and signatures but were embarrassed to ask.

Cryptography 101

I started writing this as a way of organizing my own thoughts. I originally intended 
just to share it internally with my team at DevResults, but when it was nearly done I thought 
I might as well put it online. 
Less than an hour after I hit publish on Medium, DHH himself <a href="https://twitter.com/dhh/status/1097937474003685376">tweeted it out
approvingly</a>. By the next day it was coming
back to me in my own Twitter feed, as software developers who I admired responded to it: Nicholas
Zakas, Anil Dash, Dan Abramov, Simon Willison. 
At a conference a few weeks later, I met Amir Sahlifendic, the founder of
<a href="www.todoist.com">Todoist</a>. The conversation turned to the question of mature codebases. Amir
said, “You should really read this article that came out a while back, about rewriting
software…” 
At any rate, this is by far the most widely-read thing I’ve ever written. According to Medium,
it’s been read from beginning to end by nearly 40,000 people – not bad for a half-hour read!

A new take on the age-old question: Should you rewrite your application from scratch, 
or is that “the single worst strategic mistake that any software company can make”? 
Turns out there are more than two options for dealing with a mature codebase.


Lessons from 6 software rewrite stories

Rewrite, refactor, or reinvent?

In a traditional client/server/database web app, so much effort goes into transferring and translating data between layers. Can we eliminate unnecessary complexity with a peer-to-peer design?

Rethinking the stack

A web app with no web server?

A key difference between a table in a spreadsheet and a table in a database is that database columns are typed: If you’re storing dates in a column, all the values in that column have to be dates.
Consistent data types are a big part of what makes a database a database — and what makes a database more powerful than a spreadsheet.
But, <a href="https://medium.com/all-the-things/a-single-infinitely-customizable-app-for-everything-else-9abed7c5b5e7">as we’ve seen</a>, data types are also a key point of friction for non-technical users, because they have to be nailed down too early in the process.
To get the best of both worlds, we want to avoid being heavy-handed: We don’t want to force Celeste to lay out her table schema in advance, but we do want to be smart and try to figure out her intentions. So if she types in a date, we’ll assume this is a date column. We’ll pick up the date format from the way she’s typed it in. And if she changes her mind and needs to change a column from one type to another, we want to make that as effortless as possible.
<figure>

<img src="https://miro.medium.com/max/3844/1*OZBN_m-HIuntmyIsHJfPyQ.png" alt="">
We want a column to pick up the date format from the way it’s typed in — just as it would in Excel.
</figure>

Before we go any further with this, though, let’s talk about data types.
<h2 id="towards-humane-data-types">Towards humane data types</h2>
I propose that the friction here doesn’t necessarily all come from having to choose a type in advance. The problem is also that most database systems force you to think about types from the computer’s perspective.
You only have a few fundamental types to choose from (text, number, date, boolean) and within those categories you have to make some commitments about how much space the computer should set aside. So for example, if you’re going the store numbers in a column, the system really wants you to make a lot of decisions in advance about size, precision, and so on.
<figure class='figure-lg'>

<img src="https://miro.medium.com/max/5002/1*8GaD1XKAYTfiyZ8MLwaaBg.png" alt="">
SQL Server’s datatypes. Do you want int or bigint? Any programmer worth her salt can tell you a story about making the wrong choice and coming to regret it.
</figure>

Human beings have different concerns. Does this number represent an amount in Euros or in Dollars? Or is it a measure of weight, or length, or area? What if it’s a distance in kilometers, but I want to show it in miles to some users? Does this text represent phone numbers or RGB colors?
With today’s databases, if you care about any of those things, you’re going to have to do the heavy lifting.
So basically you, the human, have to give a lot of thought to implementation details that should be the computer’s concern; but then when you want the computer to help you out with silly things like currencies or countries or weights or measures or oh, I don’t know, BASICALLY ANYTHING HUMANS CARE ABOUT, you’re on your own.
I think that’s the opposite of how it should be.
So what data types would actually be helpful to Celeste?
I think these categories ought to be front and center:
<ul>
<li>Dates and times</li>
<li>Physical measures: length, distance, weight, etc.</li>
<li>Names, addresses, and other multi-part human artifacts</li>
<li>Countries, states, currencies, languages, and other canonical lists</li>
<li>References to other things in the same system</li>
</ul>
Let’s look at each of these in turn.
<h2 id="dates-and-times">Dates and times</h2>
The passage of time is of fundamental concern to us humans, perhaps owing to the fact that we all die.
Most computer systems have something like a <code>datetime</code> datatype that allows you to identify an instant in time, down to the millisecond: for example, <code>Wednesday, June 7, 2018, 9:45:23.1234 (Greenwich Mean Time)</code>.
And maybe that is, in fact, what we are interested in.
But it’s often either a date with no time or a time with no date:
<ul>
<li><code>June 7, 2018</code></li>
<li><code>9:45 AM</code></li>
</ul>
Or, it could be a related, but distinct, concept such as:
A duration:
<ul>
<li><code>30 milliseconds</code></li>
<li><code>5 days</code></li>
<li><code>42 years</code></li>
</ul>
A period of time:
<ul>
<li>the year <code>2007</code></li>
<li>the month of <code>May 1997</code></li>
<li><code>Quarter 1 of 2019</code></li>
</ul>
A standalone day of the year, month or day of the week:
<ul>
<li>every <code>March 16</code></li>
<li>every <code>February</code></li>
<li>on <code>Tuesdays</code></li>
</ul>
Today’s databases are of very little help with managing these constructs.
Now, there are libraries in most programming languages to help programmers represent these concepts cleanly. For example, the <a href="https://js-joda.github.io/js-joda/">js-joda</a> library for JavaScript (based on the <a href="https://github.com/JodaOrg/joda-time">joda-time</a> package for Java) does a great job. But database tools supposedly designed for non-programmers — like Access or Airtable — mostly leave those users to struggle unassisted with the database’s raw data types.
This is going to be a recurring theme: There are so many great programming libraries and frameworks and concepts that have made things very clean and simple and straightforward for programmers. But no one has taken these ideas one small step further and wrapped them up in a form that makes them usable to civilians!
<h2 id="physical-measures-length-distance-weight-etc">Physical measures: length, distance, weight, etc.</h2>
Another amusing human idiosyncrasy is that we have multiple incompatible systems to measure the physical world.
The good news is that the rules for converting, say, <code>5&#39;8&quot;</code> to <code>172.72 cm</code> are well-known and involve simple math. Computers are better at simple math than people are. A thoughtfully designed system would take advantage of that fact to transparently take care of those conversions — so that one user could enter data in feet and inches, while another users would see the same data expressed in centimeters.
Again, there’s no shortage of free libraries to help with these concerns: For example, the <a href="https://github.com/ben-ng/convert-units">convert-units</a> library for JavaScript can handle not only everyday things like length, distance, weight, and temperature — but also more specialized physical measures like acceleration, frequency, or volume flow rate. But again, unless you have the wherewithal to write your own software, these don’t do you much good.
<h2 id="names-addresses-and-other-multi-part-human-artifacts">Names, addresses, and other multi-part human artifacts</h2>
If you’ve ever created a table of contacts in Excel, the first thing you’ve had to deal with is human names.
Perhaps you started with a single “name” column. Then, you realized you wanted to sort by last names, so you figured out how to automatically split a full name into a first name and a last name:
<img src="https://miro.medium.com/max/1260/0*Uny5s1vCSrUEtHCE.png" alt="">
But then start seeing names like these…
<ul>
<li><code>J. Shane Kunkle</code></li>
<li><code>Herbert “Herb” Caudill III</code></li>
<li><code>Josep Puig i Cadafalch</code></li>
<li><code>Dr. John P. Doe-Ray CLU, CFP, LUTC</code></li>
<li><code>Brigadier General Zebulon Pike</code></li>
</ul>
…and you decide to call in sick and spend the day under the covers.
And then there are addresses. Do you need one or two address lines? Three? Do you support international addresses? Do you need to break out street numbers? Apartment numbers?
What would the “pretend it’s magic” solution for handling names and addresses look like?
What I’d like would be to just paste them in, as-is — but have the computer figure out their component parts. That way I’d still be able to sort people by last name, or pull out their initials or their preferred given names. And I’d be able to display addresses in a standard format, or group them by zip code or by country.
<figure class='figure-xl'>

<img src="https://miro.medium.com/max/5496/1*G9H2sL8gsDaKF-TjgKIrgg.png" alt="">
</figure>

The good news is that we don’t need magic in this case, neither do we need to write a lot of code. Parsing and normalizing human names is hard, but it’s mostly a <a href="https://www.google.es/search?q=parse+human+names">solved problem</a>. Same for <a href="https://github.com/openvenues/libpostal">addresses</a> .
The same approach applies to other fascinating human artifacts like <a href="https://github.com/catamphetamine/libphonenumber-js">phone numbers</a>, <a href="https://github.com/bgrins/TinyColor">RGB colors</a>, and <a href="https://github.com/beaugunderson/ip-address">IP addresses</a>. There are tons of existing services and open-source libraries to choose from for each of these. There’s no reason to force people to come up with their own home-grown solutions to these basic needs.
<h2 id="countries-states-currencies-languages-and-other-canonical-lists">Countries, states, currencies, languages, and other canonical lists</h2>
Another hassle you might have dealt with in your contact spreadsheet is the problem of inconsistently entered names — for U.S. states, countries, and so on.
If you’re given this data and you want to do anything at all with these columns, you have to go through and make them consistent by hand.
<figure class='figure-lg'>

<img src="https://miro.medium.com/max/2784/1*O_msxV9aPmK-hM47iyux7Q.gif" alt="">
</figure>

And if you want to make a spreadsheet that forces people to choose from as set of options so that it stays clean, you have some hoops to jump through.
<figure class='figure-lg'>

<img src="https://miro.medium.com/max/2768/1*yd9gk-MNC8I7UQRBORrCyQ.png" alt="">
If you know how to make this happen, you’re in the elite 1% of spreadsheet power users.
</figure>

A humane system would take into account that (a) this is a really common scenario! and (b) we have standard lists for these things! — and make it easy to create a column with a datatype of <code>Country</code> or <code>US State</code> (or <code>Canadian Province</code>, or <code>Kyrgyz Oblast</code>, or what have you). Same goes for <code>Currency</code> and <code>Language</code>.
<h2 id="references-to-other-things-in-the-same-system">References to other things in the same system</h2>
One of the big limitations of spreadsheets is that it’s hard to link data from one table to another. Relational databases have this capability as a core strength — as the name implies — but as we’ve seen, most databases force us to think in an indirect and abstract way that doesn’t come naturally to most people.
This point — where Celeste realizes that what goes in a cell is actually a reference to a complex object that is defined elsewhere — is one of two key mind-melting junctures.
This shouldn’t be complicated.
Suppose I have a table of <code>Classrooms</code> and a table of <code>Teachers</code>. I want to assign a <code>Teacher</code> to each <code>Classroom</code>, so I make a column called <code>Teacher</code>. What’s the field type of that column?
It’s a <code>Teacher</code>.
What it’s NOT: A <code>Link</code>, a <code>Reference</code>, a <code>Foreign key</code>, or any other nerdy term that focuses attention the underlying implementation and makes it scary to civilians. Celeste shouldn’t have to decide whether this will be stored in a normalized or denormalized way. Celeste shouldn’t have to know an embedded document from a pointer. These are implementation details and they shouldn’t leak through to the UI.
Suppose I’m building a system that includes <code>Tasks</code>, <code>Projects</code>, <code>Companies</code>, <code>Customers</code>, and <code>Employees</code>. I should be able to treat each of these as a data type, and see them listed right there with all the other data types we’ve discussed. What goes in this field? A <code>Company</code>. Or a list of <code>Employees</code>.
Again, we’ve made this kind of thing straightforward and intuitive for ourselves as programmers. We’ve been living in an object-oriented world for over three decades now. We the have built-in primitives, yes, but we can define our own custom objects. Objects have properties that are in turn primitives or objects. This elegantly recursive paradigm makes it possible for us to model pretty much any real-world concept we can dream up, in all its detail and complexity.
So why don’t we put these tools in the hands of users?
Instead we restrict them to stupid grids and give them a handful of stupid datatypes to choose from. It ain’t right.
<h2 id="collections-of-things">Collections of things</h2>
The elegance of the object model depends, crucially, on our ability to represent arrays of objects in addition to individual objects. So there might be multiple <code>Employees</code> working on one <code>Project</code>, or multiple <code>Customers</code> in the same <code>Company</code>.
This is the second mind-melting juncture for Celeste: How do we fit a list of things — or a whole other table — into this little cell?
Again, as programmers we have no problem modeling this. You can have a thing, or you can have an array of N things.
In the latter case, it’s easy to end up with a clunky UI. But that’s a solvable problem.
What’s required is a willingness to try, and a willingness to stop infantilizing the user.
<h2 id="stop-treating-civilian-users-like-babies">Stop treating civilian users like babies</h2>
On that last point, a brief rant.
Since the early days of personal computing, we’ve made great progress towards making software easier to use. We’ve learned to say <a href="https://signalvnoise.com/posts/1626-the-most-powerful-word-is-no">no to new features</a>, we remind ourselves to <a href="https://www.amazon.com/Dont-Make-Think-Revisited-Usability/dp/0321965515">Don’t Make Me Think</a>, and we aspire to create software that <a href="https://en.wikipedia.org/wiki/Unix_philosophy">does just one thing well</a>.
That’s all healthy and positive — for the most part.
But I feel like one result of that has been that we software makers have collectively and individually ratcheted down our level of ambition. We’ve internalized a condescension towards end users that has led us produce more focused applications that do less and less, on the theory that anything more would overwhelm our poor customers.
As a result, there’s a huge chasm these days between what you can accomplish with user-facing tools and what you can accomplish by writing code.
Turns out, there are lots of intelligent non-programmers out there. There was a time — I know because I remember it — when there was tons of competition for that market. Throughout the late 1980s and early 1990s, there were dozens of innovative database systems designed for non-programmers to use: dBase. Hypercard. FileMaker.
Part of what happened, I think, was an accident of history: After Microsoft Office steamrolled the “office productivity” segment of the software industry, we were left with just two viable products in the user-facing database space: Excel and Access. Access has never sparked much enthusiasm or creativity, on account of it sucks and has always sucked. Excel has proved to be approachable and versatile enough that, in spite of not being a database, it has become the general-purpose database of choice. In doing so, it’s almost completely suffocated innovation in this space.
But those users are still out there, and I think they can be trusted with the concepts of arrays and objects and recursion.
It’s our job, as programmers, to offer up these facilities in a way that maps naturally to people’s intuitions; and there are a million ways to get that UX wrong. It’s our job to hide everything but what’s essential, to provide affordances, to avoid exposing distracting or intimidating implementation details. But it can be done, and it should be done, because it’s not right that all that creative energy is being wasted on twisting Excel into unspeakable contortions.
I’m going to stop there for now. Hopefully we’re a bit closer to having a tangible idea of what this tool might look like.
<figure class='figure-xl'>

<img src="https://miro.medium.com/max/3844/1*9d8mCVO-6fPfawXnPRXzTA.png" alt="">
</figure>

One possible way of presenting data types.
We want to make a tool that serves as a platform for humans to organize information for themselves and for teams they work on. To get there, I think we need to give a lot of thought to the types of data that humans are likely to organize. It believe it will be crucial not to dumb down this part down, but rather to use the familiar concept of the data type as a way of sneaking in the full flexibility and power of the object model.

Today’s databases force us to think about data in terms of what’s important to the computer. What if we had systems that started with things that are important in human society - things like timespans, birthdays, weights and lengths, names, addresses, and countries?

Software should know what kinds of information matter to people

Data types for humans

I have a long-standing love-hate relationship with the way computer interfaces are portrayed in films.
On the one hand, they’re so unrealistic! Lynne has patiently sat through any number of post-movie rants about this kind of thing:
<img src="/images/posts/secretagent/2.gif" alt="">
<img src="/images/posts/secretagent/3.gif" alt="">
On the other hand, they’re kind of awesome! At their best, film UIs capture the way we wish computers worked:
<img src="/images/posts/secretagent/4.gif" alt="">
<img src="/images/posts/secretagent/5.gif" alt="">
In the movies, no one ever spends five minutes looking for a file. In the movies, no one struggles to print something from their laptop.
In the movies, when you type someone’s name into the computer, you’re immediately presented with everything you need to know about them:
<img src="/images/posts/secretagent/6.png" alt="">
<img src="/images/posts/secretagent/7.png" alt="">
<img src="/images/posts/secretagent/8.jpeg" alt="">
<img src="/images/posts/secretagent/9.jpeg" alt="">
<img src="/images/posts/secretagent/10.jpeg" alt="">
<img src="/images/posts/secretagent/11.jpeg" alt="">
<img src="/images/posts/secretagent/12.png" alt="">
<img src="/images/posts/secretagent/13.png" alt="">
<img src="/images/posts/secretagent/14.jpeg" alt="">
<img src="/images/posts/secretagent/15.jpeg" alt="">
<img src="/images/posts/secretagent/16.jpeg" alt="">
Mugshots, surveillance photos, maps, scans of paper records, ID cards, maps, fingerprints, 3D body scans: It’s all there, in one place, when you need it.
When I see the computer systems used by spy organizations in the movies, I always have the same three thoughts:
Thought #1: Why do secret agents type so loudly?
Thought #2: I’d love to just get a glimpse at the systems real-world spies use. Going by the Agency databases I have seen, I’m guessing they’re not all that great:
<figure class='figure-2up'>

<img src="/images/posts/secretagent/17.png" alt="">
<img src="/images/posts/secretagent/18.png" alt="">
Maybe the CIA saves all the good software contractors for their internally-facing systems — who knows!
</figure>

Thought #3: It sure would be nice if, at my job, I could type in, say, a customer’s name — and see everything I need to know about them, in once place.
So let’s take the secret agent use-case as a way of anchoring our thinking. Let’s focus on an imaginary user — let’s call her Celeste. Celeste has a desk job at the headquarters office of an intelligence organization. She’s smart. She’s not a programmer, and she doesn’t have a team of programmers at her disposal. It’s her job to put all the organization’s information about secret agents at her team’s fingertips.
Specifically, Celeste wants to be able to type in an agent’s name (klackety-klackety-klack) and see something like this:
<figure class='-xl'>

<img src="/images/posts/secretagent/19.png" alt="">
</figure>

This shouldn’t be rocket science, should it?
Now, this screen has no slowly rotating 3-D models, no live satellite tracking in grainy grayscale, no ANSI 381-encoded fingerprint templates. Just a bunch of facts. It doesn’t even need to make little futuristic beeps as it loads. This shouldn’t be hard, right?
Unfortunately, none of the tools we have today are of any use to Celeste.
So how is she making do?
Perhaps she already has a spreadsheet with a row for each agent, with their names and vital stats. But it’s limited: There’s no place for photos or documents. And when she was trying to come up with a good way of capturing agents’ language capabilities — a table inside a cell? a couple of linked tables? numbered columns? — <a href="https://medium.com/all-the-things/a-single-infinitely-customizable-app-for-everything-else-9abed7c5b5e7">her brain melted</a>.
<ul>
<li>She tried some database-spreadsheet hybrids like Airtable and Fieldbook, but they were not much better than spreadsheets: Neither one lets you customize a form like this; and neither one can handle things like the log of notes or the language skills table.</li>
<li>She looked at easy-to-use off-the-shelf systems for storing information about people— contact managers, CRMs, ERPs, etc. — but they weren’t made for the espionage business. With a little imagination, maybe you can stretch things a little bit — think of <code>Agents</code> as <code>Customers</code>, and <code>Missions</code> as <code>Deals</code>, and so on—but at some point you run into the limits of whatever customizability these systems offer.</li>
<li>And she probably considered highly customizable application builders — Mendix, Salesforce, Microsoft PowerApps, Google App Maker —but found them intimidating and unapproachable.</li>
</ul>
So it’s up to us to design a system that could help Celeste. Her happiness is in our hands.

Why shouldn’t our tools for organizing information be as cool as the ones in the movies?

A secret agent database for&nbsp;the rest of us

Computer, enhance

Before we create a “single infinitely-customizable app for everything else”, we need to understand
why we’re currently using spreadsheets for everything.


What would it take to replace the spreadsheet?

Excel hell: A cautionary tale

No matter what applications we choose, there's lots of duplication and overlap between them. We're paying for lots of stuff we don't need or use. And there are still gaps that we have to cover with home-made systems. Surely there's a better way?