Our incidents got 42% faster to resolve. We did not buy a tool. We moved people.
This is the boring, slightly embarrassing story of how the org chart was the bug.
The setup
Every customer problem crosses four teams before anyone can fix it. A frontend team. A backend team. A platform team. A data team. Each one owns a layer. It looks tidy on a slide.
The checkout page breaks. The change is a config flag. Frontend gets paged. Frontend says the API is slow. Backend says the database is slow. Platform says the database is fine, it is the queries. Data says those are not their queries. Forty minutes later, someone finds it. It was always a config flag.
Everybody was right about their own layer, and that was the problem. Nobody was lazy. Layers do not have customers. Products do.
The change
We reshaped teams into product cells. A cell owns one product slice, top to bottom: the page, the API, the data, the alerts, the pager. If checkout breaks, the checkout cell gets paged, and the checkout cell can fix every line of it.
Here is the ownership map. It is boring on purpose. Boring is readable at 3am.
# who gets paged
cells:
checkout:
owns: [cart, payments, receipts]
pager: checkout-oncall
search:
owns: [query, ranking, suggest]
pager: search-oncall
accounts:
owns: [login, profile, billing]
pager: accounts-oncall
The map only helps if the pager reads it. So alert routing stopped asking “which layer is this?” and started asking “which product is this?”.
// Route an alert to the cell that owns the service.
// No owner is a bug, and it pages a human about the bug.
type Alert = { service: string; severity: "page" | "ticket" };
export function route(alert: Alert, cells: Record<string, { owns: string[]; pager: string }>) {
for (const [name, cell] of Object.entries(cells)) {
if (cell.owns.includes(alert.service)) return { cell: name, pager: cell.pager };
}
return { cell: "unowned", pager: "eng-leads" };
}
That last line matters more than the loop. An unowned service pages the leads. Leads hate being paged. Give it a week and every service has an owner. Incentives are the best linter.
The number
Measure time to resolve per incident, not per team. Here is a query in the shape of the one you want. The table and column names are made up. The 42% is the real one.
-- median minutes from first alert to resolved, by quarter
select
date_trunc('quarter', opened_at) as quarter,
percentile_cont(0.5) within group (
order by extract(epoch from resolved_at - opened_at) / 60
) as median_minutes
from incidents
where severity = 'page'
group by 1
order by 1;
Incident resolution dropped by 42%. Not because people got faster at typing. Because the first person paged had the fix. No handoffs. No “can you join the bridge”. No forty minutes of polite blame.
What it cost
It was not free.
- Specialists get lonely. A database expert inside a product cell has nobody to argue indexes with. Give the specialists across cells a regular hour to argue in peace. People call it a guild. It is a support group with a calendar invite.
- Some code gets duplicated. Two cells will write two retry helpers. That is fine for a while. Then you pull the good one into a shared library, on purpose, with an owner.
- The pager gets quieter, and people get nervous. A quiet pager feels like a broken pager. It is not broken. It is bored.
How to try it on Monday
You do not need a reorg memo to start. You need a spreadsheet and a little nerve.
- List every service. All of them. The one in the corner that only runs on the first of the month counts too. Especially that one.
- Write one owner next to each. One. Not “shared”. Shared means nobody, with extra steps.
- Group the services by the product a customer would name. Customers do not say “the payments microservice is degraded”. They say “I cannot pay”. Name cells the way customers complain.
- Point the pager at the cells. Use the map. Delete every rule that pages a layer.
- Make unowned loud. Anything without an owner pages a lead. Watch how fast ownership becomes popular.
Then wait a quarter and look at the number. If it does not move, your cells are still layers wearing a product hat. Look for the handoffs. Every handoff is a place where an incident sits and waits for a human to notice.
Ownership is a verb
The map is not the point. The point is that one group of people can say “that is ours” about a problem, and then fix all of it without asking permission from four other queues. A map only writes that down so nobody can argue at 3am. People will still argue at 3am. They will just argue with the person who can actually fix it.
What I would tell you
If your incidents bounce between teams, your tools are not the problem. Your boundaries are. Draw the org chart around products, give every service exactly one owner, and make “unowned” page somebody senior.
Then go to bed. The pager will be fine.
Related: the essay version of this, with horses, is Every Team Is a Chess Set.