Photograph of a chained reference book on a wooden reading desk in an old library, five wooden stools standing around it and light falling through a tall window.
|

The export redacted its own documentation

The second agent moved today, and it went the way the first move should have gone: profile copied, service started, chat platform reconnected, scheduled jobs ticking. Then it tried to use its publishing credentials and died on a syntax error.

The file it died in is a small helper that builds an HTTP Authorization header. On the machine it came from, that line reads return { Authorization: `Basic ${basicToken}` };. On the machine it arrived on, the same line read return { Authorization: *** ${basicToken}` };. Three asterisks where a backtick and the word Basic used to be. A template literal without its opening quote is not a template literal, so the helper refused to load at all. The same substitution had happened on two more lines.

Nobody typed those asterisks. Between the original file and the moved copy there was exactly one step, the profile export, so we compared the two copies byte by byte. The moved copy is ten bytes lighter. Three of its 338 lines differ, no lines were added or removed, and each difference is one substitution: the six bytes of `Basic became three asterisks, twice, and the seven bytes of `Bearer became three asterisks once. Everything else in the file is identical.

Then we widened the comparison: every text file in that export that also exists on the exporting machine, 153 of them. Five had changed. Twelve lines in total, and every one of those lines contains something the exporter decided was secret-shaped. Four of the five files are documentation.

Card comparing the live and exported copies of lines from a migration: a JavaScript Authorization header whose Basic prefix became three asterisks, a documentation line listing mask patterns that gained three asterisks, an example whose value changed to three asterisks, an environment example whose token was masked, and a webhook placeholder shortened to your-w...here.
Five of the twelve lines the profile export rewrote: the publishing helper’s auth header and four documentation lines about masking. Byte counts are the measured lengths of each line before and after the export.

The page that lists which patterns get masked came through with this line: - Generic `token=`, `key=`, `API_KEY=*** `password=`, `secret=` patterns. The masker read API_KEY= as a name-value assignment, masked the empty value, and swallowed the closing backtick with it. The page documenting the switch that turns masking off now reads export HERMES_REDACT_SECRETS=***, where it used to say =false. A webhook guide’s placeholder arrived as your-w...here. Example tokens in a reference page for tool integrations came across as ghp_xx...xxxx, using a second mask shape: three asterisks for assignments and auth prefixes, three dots for anything that looks like a key.

So the exporter edited the documentation about the exporter, including the page that explains how to switch it off. That is the detail that makes this more than a bad afternoon. A masking rule that runs over a log file is a reasonable precaution. The same rule running over the files you intend to keep is a content transformation with no review step, and this one used a name-based guess to decide what to rewrite.

Why only one of the five failures was loud

The helper script failed immediately and unambiguously, and that was luck. A syntax error cannot be ignored. The documentation changes were silent, and they are the kind that ambush you months later: an example whose value has been shortened, an instruction with a hole in it, a reference page whose list no longer matches what the code does. Nobody re-reads a reference file after a migration to check whether it changed. You read it when you need it, follow it, it doesn’t work, and you assume you misread it.

There is a worse version of this that we got lucky on. When the masked thing is a value in a config file that is supposed to hold a real secret, the service starts anyway with three asterisks where a key should be, and that looks exactly like a wrong key. Ours landed in a script, which is the version that fails loudly and also the version that costs an hour for anyone who trusts their editor.

The check that would have caught it

Masking is not hygiene when it runs over data you plan to keep. It is a transformation of your content, and it deserves the suspicion we give every other transformation: scope it, make it reviewable, and diff the result. Exit codes cannot catch this one. The export exited zero, the archive was valid, and the target booted and served traffic. The only detector we had was a runtime error hours later, in a code path that runs when the agent publishes something.

What we do before any profile moves now: compare the archive with the source file by file, treat every byte difference as something to explain, and classify what we find. A mask substitution is an expected difference, and it still gets listed, with the line number and the number of bytes it removed, so whoever is doing the move can see that a rewrite happened and decide whether the thing that was masked was a secret or just a word the masker disliked. The archive from this morning still carries the asterisks, which is how we know the rewrite happened during the export and not during the hours of work on either side of it.

The move itself changed shape

Making that check worth running forced a bigger question about how much we were copying. The plan we started with gave every agent its own complete copy of the runtime: its own interpreter, its own dependency environment, its own update path. Five agents meant five copies of the same tree, five things to verify after every move, and five more chances for a file to be rewritten on the way.

The second move replaced the copies with one install. It is a sealed, relocatable tree, owned by the system and readable by every account that runs an agent from it, carrying its own interpreter, tools and dependency environment. Each agent keeps only its own profile directory, plus a templated service unit whose launcher points at the shared tree. An instance that is not explicitly pinned refuses to start rather than running something unpinned.

The test for whether it is genuinely shared is that an account owning nothing inside the tree can boot it. Ours reports the packaged interpreter, no source update, and no missing dependencies, and the per-agent checkout is gone from that host.

Diagram of a shared read-only install tree that ships its own interpreter and dependency environment, next to five agent profile directories, two marked as running a service and three not moved yet.
The install after the second move: one sealed read-only tree, five agent profiles, two services running from it. Measured on the host, 2026-09-26.

What the second move bought

Two agents now run from the one install and three are still on the old machine. Each of the moves added something the previous ones could not have shown us. From this one, beyond the rewrite: an inbound-message allowlist that lives in a file we deliberately do not copy, so the agent comes up healthy and silently ignores everyone who talks to it. And a process guard that stops an agent from restarting its own service, which means the last step of a cutover has to run from somewhere else entirely.

The runbook that came out of the first move has grown to nine pitfalls. The export rewrite is one of the newest, and it is the one I would have bet against, because it wasn’t a mistake in the moving. It was a defensible safety rule applied one layer too broadly. A copy stops being a copy the moment something downstream is allowed to decide what counts as a secret.

Similar Posts