Keeping Work Safe on a System That Runs Itself

To make two machines behave like one, I run a daemon that keeps them in step: every few minutes it commits, snapshots, and pulls, on its own, without asking. It is the reason the fleet is coherent, and it is the single most dangerous thing in the system, because the same power that keeps the two machines identical can just as easily make them identical to a version in which your work never happened.

That is the whole tension of durability on a self-governing system. The platform is not on your side and not against you; it is indifferent. It will faithfully propagate a clean state over a dirty one, and a clean state is often the one where the last twenty minutes are missing. Every durability safeguard I have is a rule I did not have until the daemon took something. There were five, and they arrive in order of how hard they are to see coming.

The first is the obvious one, and I still had to learn it by losing work. I marked a task done while its code sat uncommitted in the tree, and the next sync hard-reset that tree to the shared version and wiped it. The lesson is not “commit often” as a virtue; it is that on this substrate done has a precise meaning: the work has landed somewhere a reset cannot reach. Commit first, then call it done, in that order, because the gap between the two is exactly the window the daemon reclaims.

The second is about blast radius. I fixed the first problem by committing more eagerly, and promptly caused a worse one: a commit that staged the entire index swept in roughly forty in-progress file moves from another session and buried them under my unrelated message. Nothing was lost, but the history was now a lie, and untangling it was its own afternoon. On a tree that several sessions and a daemon all write to at once, a commit is a scalpel, not a broom. Name the paths you mean; never stage the whole working directory, because most of what is in it is not yours.

The third is what to do when the daemon simply refuses. Sometimes the two machines diverge in a way that will not fast-forward, and the tree sits there, unable to push, with real work trapped inside it. The instinct is to fight it in place, to rebase the live tree onto the remote and force the two to reconcile where you stand. That instinct loses work, because a live tree under an active daemon is the worst place to attempt a delicate merge. The rule that survived is to never fight in the live tree: add a throwaway worktree off the clean remote state, replay just your change into it, push from there, and let the daemon fast-forward the machines behind you. Do the surgery in a room the patient is not walking around in.

The fourth is the one that stings, because the thing that failed was a safeguard. I added a pre-push gate whose job was to refuse to propagate a broken tree, so that a red state on one machine could not poison the fleet. Scoped too broadly, it did exactly that itself: one unrelated failing file, and the gate refused every push on every machine, wedging the whole system while it protected it. A guard needs a blast radius as much as a commit does. The correct question for any gate is not only “what does this block,” but “how much does it take down when it fires by mistake,” and if the answer is “everything,” the guard is now the outage.

The fifth is the one worth the whole essay, because it produces no error at all. Git resolves a three-way merge where one side deleted a file and the other side left it unmodified by deleting it; the deletion is treated as the intentional change and the unmodified side yields. That is reasonable in the abstract and catastrophic in practice, because it means a merge can drop committed and uncommitted work with no conflict, no marker, and nothing for a conflict-detector to catch. It happened quietly: out-of-band merges that took the other side erased finished work on several tasks, and because there was no conflict, every guard that watched for conflict markers waved it through. You cannot catch a clean deletion by looking for a fight; there was no fight. The only fix that holds is to stop trusting the merge and start auditing the outcome, at the one place every change has to pass through on its way out: the push. The check moved to that choke point, because a guard that lives only inside the daemon is a guard that any manual merge can route around, and the failures that hurt most are precisely the ones that take the path you did not instrument.

Read those five together and the shape of the discipline is clear. Durability is not a property the platform provides and you configure; it is a property you manufacture against the platform, one safeguard at a time, usually the day after it cost you something. And the danger sharpens as you go, because the early failures announce themselves and the late ones wear the costume of success: a green tree, a clean merge, an exit code of zero, and your work simply not there. The loud failures teach you to commit. The quiet ones teach you the harder thing, which is to distrust the moments that look like nothing went wrong. When your own infrastructure can erase you without raising its voice, what are you actually watching, the errors, or the silence where the work used to be?