Handling high-stakes escalations when every option is bad

Engineering blogs love to pretend that executive escalations are just a communication problem: summarize the bug in a clean one-pager, present three balanced options, and everyone in the room will find an agreement and pick a sensible option.

Anyone who has actually sat in an escalation with VPs and a frustrated customer CTO knows that it is a fantasy.

When critical deadlines are blowing up, the room doesn't want a sensible 4 day solid plan: they want a magical solution that makes everything go away before tomorrow morning.

One person climbing, and another securing them with a rope

Most people think moving into a Staff+ Individual Contributor (IC) role is about designing larger, and more complex, distributed systems, or better architecture docs. In peacetime, it could be. But in wartime, you earn your paycheck in the 15 minutes when there is a huge issue, every option forward is bad, and an exec asks if you can take a shortcut or two to hit tomorrow's deadline.

Why the room wants a miraculous option C

When an important milestone is blocked by a critical bug, nobody in leadership wants to hear that a customer deadline is slipping by three weeks, and nobody wants to sign off on a risky technical solution that might blow up during a maintenance window.

So the room naturally starts bargaining for a third choice that doesn't exist: Option C.

Option C is the comforting tale where everyone does their best, pushing through the night, skipping testing and validation, and is able to hit the deadline with zero issues.

Junior and mid-level engineers offer option C because they want to be helpful under pressure. And executives love option C (and they will actually push for it) because it makes their business problem disappear without having to have a hard conversation with the customer.

And to be fair to them, they push because half of the time an engineering team says something takes 3 weeks, just for 2 of those to be bureaucratic padding. They are pushing to see where the fat is. If you are the senior engineer in the room, caving on the actual physics to look like a 'team player' isn't helpful: your job is cutting the bureaucratic fat yourself, and then holding the line on the real technical limits, grounding the choices in reality.

Bound the risks in a 1-pager

We started saying that the 1-pager everyone agrees on is a fantasy: true, but it is still necessary to lay the foundation of the meeting, and force the conversation to be a choice between the only two real levers there are: time or bounded risk.

Start with the context, in no more than 3 sentences: what failed, what is the customer impact today, by when we need a decision. Then, the two possible choices:

  • Option A, pay in time: wait the required time for the full standard release cycle. We accept a 3 weeks schedule slip, and the hit to customer trust, in exchange for zero risk.

  • Option B, pay in bounded risk: put together a small and efficient working team, skip the slow non-critical test suites, and ship a targeted hotpatch in 4 days. We accept a localized risk to unblock the customer this week.

The keyword for option B is bounded: VPs are paid to take business risks. What creates panic is unbounded risk. "Give us a few days to try something and hope it works". As the SREs say (and the subheading of this blog as well): "Hope is not a strategy".

If you want leadership to give you the 4 days you need, instead of forcing a rushed solution tomorrow, you have to show what these 4 days buy, and what are the safety stops.

In a recent escalation I managed, two VPs signed off on our hotpatch plan the day we presented it, because we presented Option B with clear rules: which specific test suites we were skipping, and which validation gates we refused to cut (such as testing the upgrade path on mirror hardware first). We had a clear rollback plan if things were to go south, and a hard deadline that would have us steer in another agreed direction if our solution wasn't working.

Disagree and commit

4 times out of 5, the above is enough. However, the fun begins that 5th time, when the business pressure is so high that your leadership still pushes for Option C. "Your bounded risk approach is still too slow. Push a fix tomorrow".

This is where many senior engineers either cave, and pretend Option C is safe, or turn the meeting into a screaming match. Neither of them is the right way to go.

As senior ICs, we own the technical systems, but not the company P&L (profit and loss) or the customer contracts. This is what a VP owns. And they are allowed (and paid) to make a high-risk business bet (as long as nothing is illegal or safety-critical, of course). What they can't do is ask you to pretend that the risk is low.

So, instead of arguing or silently caving, hand the decision back to them as a business call. Clarify in plain numbers what shipping tomorrow means skipping, and what the consequence of failure looks like (e.g., "Skipping the 4-day mirror test means flying blind on state recovery; if we hit a mismatch, we face a 5-day outage"). Make it clear that if leadership decides the business deadline outweighs a potential 5-day outage, it is their call, and add an explicit line in the doc for the VP to sign off on accepting that specific risk.

Putting an explicit 'Risk Exception' sign-off in the doc isn't about covering yourself or trapping an executive. Often, seeing the exact blast radius written down in cold numbers often makes leadership realize the shortcut isn't worth the bet. More importantly, if they decide to go with the option anyway, an explicit sign-off gives your team the guidance they need to break standard release policies without taking the blame if it blows up.

But if the VP does sign the exception, you have to actually commit, not sulk and wait to say "I told you so."

Being right in retrospect can give you some satisfaction, but it ain't a team player move, doesn't help the company, and ultimately doesn't help your career or your team, that still has to clean up the consequences.

Properly committing to a high-risk business bet doesn't mean running your normal pipeline and crossing your fingers. Quite the opposite: because you skipped the safety gates in test, you have to build temporary safety gates in production:

  • Take manual state snapshots right before the maintenance window so you can rollback to the last working state;

  • Study a rollout strategy that allows you to have the smallest possible blast radius for each step;

  • Have a SWAT team of your best engineers ready to jump in, making sure you know what are the metrics to be careful about;

To be clear: you shouldn't be asked to do this every week: if every release is an executive exception, your release cycle is broken, and you should be working on fixing that before the escalations.

Leadership takes decisions, and if they knowingly pulled the trigger on a business bet, you own the execution as if the decision were yours. Your job as a technical leader isn't to win every argument: it is to make sure the organization never takes a blind bet by lying to itself about the odds, and to hold the safety line when it decides to go over the edge anyway.

Comments