Documenting the undocumented quirks and unwritten rules

Most runbooks look good on paper. They describe the happy path: how to deploy, how to restart a service, how to run the daily job. Then something goes wrong, and the person who knows the fix is the one who built it three years ago. The real documentation that keeps a system running is often not in the runbook at all — it lives in Slack threads, in one engineer's memory, or in the quiet instinct of "oh yeah, that warning is actually fine."

These are the undocumented things: config quirks, environment variables nobody can explain, deploy orders that are not enforced but absolutely matter, and error messages that do not mean what they look like they mean. Documenting them is not about writing more — it is about capturing a small, specific class of knowledge that makes the difference between a calm on-call shift and a scramble at two in the morning.

Where the undocumented things hide

They are not hidden maliciously. They build up over years, each one small enough to feel not worth writing down. Together they form a quiet layer of know-how that a new hire or a new on-call engineer cannot possibly possess. The most common hiding places:

• Environment variables that were set once, during a migration, and never questioned again
• Deploy or restart sequences where the order matters but nothing enforces it
• "Known" errors that appear in logs but are actually safe — alarms that have been going off for months with a note in someone's brain that says "ignore this one"
• API keys or service accounts whose purpose nobody remembers but whose rotation would break a pipeline
• Configuration values that look wrong on purpose, because changing them to the obvious value broke something in 2022 and nobody wrote down why

How to capture them: the undocumented-runbook page

The practical fix is not to rewrite every runbook. It is to add one page: an "undocumented" section in each system's documentation — or better yet, in your handover runbook — where the quirks live openly.

Give each quirk one line, a short explanation, and a date:

``` # Undocumented (until now)

• SERVICE_TIMEOUT=300s — set high because the legacy API takes

unpredictable time. Do not reduce below 120s. (2024-03)

• /var/log/app/warn — the "auth failure" line at 3:14 AM is a

health check with expired test creds. It is safe and known. (2023-11)

• Deploy order: run the migration BEFORE restarting worker-2,

or the column mismatch will log silently for 30 min. (2024-07)

• API key xyz-789 — used by the deleted reporting service that

nobody has run since 2022. Safe to revoke. (last confirmed 2025-01) ```

This format is intentionally light. It does not replace proper architecture docs or incident postmortems — it supplements them by capturing the kind of knowledge that does not fit neatly into a diagram or a procedure but still matters when something goes wrong.

Building the list: the quiet interview

If you are joining a team or preparing to hand over, spend twenty minutes with each senior engineer and ask one question: "What is the one thing you know about this system that nobody else knows and that could break something if you left tomorrow?"

You will get answers that do not appear in any runbook. Write them down exactly as spoken, in the one-line format above. No need to polish or formalise — the goal is capture, not presentation. A raw, readable note written today is worth more than a perfectly formatted document that was never written.

The Handover Folder Starter includes the undocumented-runbook template ready to fill, and the Complete version adds the system-by-system tracker for team-scale rollouts. Prefer to start free? The free Handover Quick-Start gives you the runbook skeleton today.

"Documenting the Undocumented: Config Quirks, Workarounds, and the Unwritten Rules That Keep Things Running": FAQ

How is this different from a normal runbook or postmortem?

A normal runbook covers procedures — how to deploy, how to restart, how to roll back. An undocumented-quirks page covers the things those procedures assume you already know: the environment variable whose purpose is lost to time, the error log that is actually fine, the deploy-order dependency nobody enforces. It is the layer between standard documentation and tribal knowledge.

When should I update the undocumented-quirks page?

Add to it whenever you discover something surprising — a config value that looked wrong but was intentional, an incident whose root cause was an unwritten assumption, or a Slack thread where someone asked "why does this alert keep firing?" and got an answer that was not written down anywhere else. It grows slowly, one quirk per discovery.

What if I am not sure whether something is worth documenting?

If you had to ask someone else to understand it, it is worth writing down. The bar for an undocumented-quirks entry is not "this will cause an outage" — it is "this will confuse the next person who looks at it." When in doubt, write the one line.

Keep reading

• How to Reduce Your Team's Bus Factor (Without Heavy Process)
• The Engineering Handover Checklist (A Calm, Complete System)