Case note: one on-call rota for the whole stack

When one team owns order capture through delivery, an incident is a page, not a conference bridge. Nobody spends the first hour deciding whose fault it is: a single correlation ID follows the failing order across every service on one event bus, the fix rolls back through the same pipeline that shipped it, and the postmortem action lands the same week. This is a field note on what changed when we collapsed four vendor rotas into one.

What does an incident look like with four vendors on a bridge call?

The classic route-to-market incident starts as a customer noticing an order that never turned into a delivery. In the four-vendor shape, nobody can see the whole path: the CRM vendor owns order capture, a commerce systems integrator owns the storefront, a middleware team owns the pipes, and internal IT owns whatever sits closest to the financial core. Each can prove its own component is green. None can prove where the order died. So you open a conference bridge, and the first hour goes to establishing whose problem it is rather than fixing it.

The drag is not the fix; it is the correlation. Four vendors means four log stores, four consoles, four clocks that are not quite synchronised, and four contracts that each define 'my bit works'. A human has to stitch the order's journey together by hand across all of them before anyone can even name the failing hop. That is why the numbers on recovery are so lopsided. In DORA's 2019 State of DevOps data, elite teams restore service 2,604 times faster than low performers, with elite recovery under an hour where low performers measure it in days to weeks. A cross-vendor bridge call is a machine for landing in the low band, whatever the individual engineers on it are worth.

Why does one event bus collapse the bridge call?

Because the correlation stops being manual. When order capture, stock, loyalty, returnables, delivery and analytics all publish to one event bus, a correlation ID minted when the order is captured rides every event that order produces. The failing hop is not something you deduce from four log stores at 3 a.m.; it is the last span in a single trace. The on-call engineer opens one view, follows one ID, and sees exactly which service stopped forwarding the order.

The second half is ownership. The team that wrote the service is the team that gets paged and the team that can roll the change back — through the same deploy pipeline that shipped it, not a ticket handed to a separate ops group overnight. That is the structural reason the recovery numbers move: DORA's elite teams not only restore faster but ship change failures seven times less often, because the feedback from a bad deploy lands on the people who can fix it immediately. The 2024 report keeps recovery under an hour as part of its definition of elite, a band only about 19% of respondents reach — and almost none of them get there through a cross-vendor bridge.

This is the same ownership argument we made when we retired the 2 a.m. stock sync: a live projection that a human now owns at 3 a.m. belongs behind the same rota as the code that feeds it, not with a batch team who clocked off long before the errors surfaced.

How do you put the whole stack behind one rota?

You do not do it by writing 'one team owns everything' on a slide and keeping four consoles open underneath it. The moves that actually collapse the bridge call, in the order we make them:

1. Mint one correlation ID at the edge and propagate it everywhere. Order capture stamps every downstream event with the same ID. If a service cannot carry it through, that service is your next piece of work — the blind spot will cost you an incident later.
2. Give one team the whole value stream and its pager. The people who write order capture through delivery are the people who get paged for it. Ownership that stops at 'hand it to ops' is a hand-off wearing a rota.
3. Put the trace in one place. One dashboard that follows the correlation ID across every service, so the on-call engineer never logs into a second vendor's console to find the failing hop.
4. Roll back through the pipeline that shipped it. The rollback path is the deploy path in reverse, owned by the same team. No change ticket, no separate approval queue convened at 3 a.m.
5. Make the boundary to the financial core explicit. The SAP core stays where it is; the rota's job is to page correctly across that boundary and know which incidents are the platform's and which are genuinely the ledger's. Draw the line once, in the open, so nobody re-litigates it mid-incident.
6. Run blameless postmortems and ship the action the same week. Because the team that ran the incident also owns the pipeline, the fix does not wait for a release window. If a postmortem action needs a quarter to land, the ownership is still broken somewhere.

Several of those events do double duty. The movement and returnable events that flow past the on-call engineer are the same ones that let us end the monthly empties-balance argument, and the order-to-delivery trace is the same spine that closes the loop between a rep visit and the next order. One bus, one ID, many uses. The shape that makes this possible is the platform architecture itself.

What did we see, and what went wrong?

Framed as our own monitoring rather than any client's claim, the direction was clear once the rota consolidated. Across the incidents we handled in the first quarter after cutover:

Now the honest part: the single trace had a hole in it for the first few weeks, and it was exactly where we least wanted one. A legacy adapter sitting at the SAP boundary did not propagate the correlation ID — it accepted an event, did its work, and emitted a fresh event with a brand-new ID. So any incident that crossed that adapter still needed hand-correlation, and because that boundary is where finance-sensitive failures live, it was the worst possible place to lose the thread. We found it during a real incident, not a drill, which cost us an evening of exactly the manual log-stitching the whole exercise was meant to kill. Patching the adapter to carry the ID through took a day; trusting the trace again took a fortnight. The lesson was blunt: 'one correlation ID' is a property you verify at every hop, not one you get by declaring it at the edge.

The next thing we want to fold into the rota is the alerting itself. A single trace tells you where an order died; it does not yet tell you which deploy made it die. We are wiring change events onto the same bus so a spike in failures can point at the commit that caused it before a human forms the hypothesis. When that lands, the postmortem starts with the suspect already named — and the bridge call, already gone, will not be missed.