Contents
- Week 1: don’t fix anything
- What to instrument first when there’s no observability
- The three changes that buy the most stability, fastest
- Taking over another vendor’s codebase
- What 8 to 12 weeks realistically fixes
- FAQ
- If your system is on fire right now
Every firm in this category sells project rescue. Almost none of them publish how they do it.
That is a marketing decision, and it’s the wrong one, because the playbook isn’t the moat. Execution under pressure with people who have done it before is the moat. So here’s the sequence we run, in enough detail to follow yourself.
It works in this order and it degrades badly out of order. The single most common failure in a rescue is starting to fix things in week one.
Week 1: don’t fix anything
The instinct on arriving at a burning system is to start putting out fires. Resist it for five days.
A team that starts changing a system it doesn’t understand generates new incidents while investigating old ones, and now nobody can tell which problems are original and which are yours. You’ve destroyed your own signal in the first week.
The week 1 triage protocol
Days 1–2: catalogue, don’t diagnose.
Pull the last 90 days of incidents. For each: what broke, how it was detected, how long to detect, how long to recover, what fixed it, whether anyone understood why. That last column is the important one and it will be mostly empty. Write “unknown” honestly — a catalogue with honest gaps is more useful than one with confident guesses.
Days 2–3: interview, including the quiet people.
Every engineer who touches the system, individually, not in a group. Ask three things: what breaks most often, what are you afraid of, and what have you been saying for a year that nobody acted on.
That third question is where the answers are. In a failing system, somebody usually knows exactly what’s wrong and has stopped saying it because saying it cost them something. Ask the most junior person and the longest-tenured person on the same day.
Days 3–4: map the blast radius.
Not the whole architecture — just: what does each failure mode take down, and what does the business lose while it’s down? You’re building a severity model, and you need it before you can prioritize anything.
Day 5: the stop-doing list.
Usually the highest-leverage output of week one. Some portion of current activity is actively making things worse: a migration half-executed, a feature being built on a subsystem that’s about to be replaced, a manual process someone invented to work around a bug that could just be fixed. Stop those before starting anything.
Why the “do nothing” week gets skipped
Because it is politically very difficult.
You have been brought in at expense during a crisis, and week one produces no visible improvement. Somebody senior will ask what they’re paying for. The pressure to ship a visible fix by Friday is immense and it is the single most common reason rescues fail.
The counter is to make week one’s output visible: the incident catalogue, the severity model and the stop-doing list are all deliverables. Hand them over on Friday. They are worth more than a hotfix, and presenting them that way is part of the job.
What to instrument first when there’s no observability
Most failing systems can’t be diagnosed because nobody can see them. You cannot fix what you cannot measure, and you cannot afford a full observability build before you start.
Instrument in this order. It’s ordered by diagnostic value per hour spent, not by completeness.
1. Error rate at the boundary. Errors returned to users or calling systems, by endpoint. Cheapest to add and answers “is it broken right now” — the question you’ll be asked most.
2. Latency at the same boundary, at p50/p95/p99. Averages hide everything that matters. The p99 is where your angriest customers live.
3. Whether the last deploy correlates with anything. A deploy marker on your error and latency graphs. Trivially cheap and it resolves a shocking share of incidents in seconds.
4. The three or four business-critical transactions. Not technical metrics — “orders completed per hour,” “invoices generated,” “logins succeeded.” These catch silent failures where nothing errors and nothing is slow and the system has simply stopped doing its job. This is the most valuable instrumentation in the list and it’s almost always missing.
5. Resource saturation on the constrained resource. Usually database connections, a queue depth, or a thread pool. One number, not a dashboard of forty.
6. Structured logging on the top three failure paths. Only after 1–5. Logging everything produces a haystack.
Resist building a comprehensive observability platform. You need six signals, fast, not a project.
The three changes that buy the most stability, fastest
After week 1, three changes reliably deliver the most stability per unit of effort.
1. Make deploys reversible before making them better.
Not “improve the pipeline.” Ensure any deploy can be undone in minutes without a discussion. A one-command rollback, tested under realistic conditions, with a documented decision rule for who pulls it and when.
This changes the psychology of the entire engagement. Every subsequent change becomes cheaper because it becomes reversible. Do this first, always.
2. Put a safety net under the highest-blast-radius path.
Not comprehensive test coverage — that’s a year. Integration tests around the single path that costs the most when it breaks, identified from your week 1 severity model.
You’re not trying to prove the system is correct. You’re trying to make one specific catastrophe detectable before a customer finds it.
3. Remove the top recurring incident cause.
Your week 1 catalogue will show a single cause accounting for a disproportionate share of incidents. It’s usually unglamorous — a missing index, an unbounded query, a retry storm, a cron job overlapping itself.
Fix that one thing. The team’s time comes back immediately, and that recovered capacity is what funds everything after it. This is also the change that buys you political room for the rest of the engagement.
Notice none of the three is a feature or an architectural improvement. Stability first, capability later. why architectural decisions made during instability are bad decisions
Taking over another vendor’s codebase
A large share of rescues involve inheriting work from a firm that is being replaced, and the technical problems are rarely the hard part.
Get access sorted before anything else. Repositories, cloud accounts, DNS, CI, third-party services, the monitoring tool nobody remembers paying for. Make a complete list and get every credential transferred in writing. This takes longer than anyone expects and it blocks everything.
Assume the documentation is aspirational. Not maliciously — documentation written during a project describes intent, and intent drifts. Verify every claim against the running system before relying on it.
Be scrupulously fair about the previous vendor. You will find bad decisions. Some were made under constraints you can’t see: a deadline, a budget cut, a client instruction. Blaming your predecessor is easy, feels good, and costs you credibility with the client’s own engineers — who were there, know the context, and will decide within two weeks whether you’re honest.
Interview the departing team if you possibly can. Even one hour. They know things that exist nowhere else and are often willing to hand them over if approached without accusation.
Find out why they were fired. Sometimes the answer is that the vendor failed. Sometimes the answer is that the client is very difficult to work with, changed requirements constantly, or wouldn’t fund what was needed. That’s information about your own engagement and you want it early.
What 8 to 12 weeks realistically fixes
Honest scoping, because a rescue that oversells produces a second failure.
What it fixes: incident frequency falls substantially; time-to-detect and time-to-recover improve sharply; the team stops firefighting continuously and regains real capacity; you gain a documented understanding of the system; deploys become routine rather than events; the worst recurring failure is gone.
What it does not fix: the architecture. Twelve weeks does not rebuild a system, and any firm promising otherwise is selling you a second rescue. Nor does it clear accumulated technical debt broadly — it addresses the specific debt causing the current bleeding. It won’t fix an organizational problem either, though it will make one much more visible. And it won’t deliver new features; that capacity is going somewhere else, deliberately.
The output of a good rescue is a system that is stable enough to make decisions about. That’s the actual goal. A burning system can’t be assessed, can’t be rebuilt safely, and can’t be sold — and its team can’t gather requirements, because the people who hold them are exhausted. why a rebuild decision made during instability is a guess
Our Stabilization Sprint runs $85K–$180K over 8–12 weeks and is scoped to exactly this. Afterwards, most clients do a Systems Audit — the decision about what comes next is worth making with evidence, and evidence is only gatherable once the fire is out.
FAQ
How do you rescue a failing software project?
In sequence: one week of triage without fixing anything, then minimal observability on six specific signals, then three changes — reversible deploys, a safety net on the highest-blast-radius path, and removing the single largest recurring incident cause. Starting to fix things in week one is the most common reason rescues fail.
How long does it take to stabilize a failing production system?
Eight to twelve weeks for meaningful stabilization with dedicated engineering. That gets incident frequency down, detection and recovery times sharply improved, and the team out of continuous firefighting. It does not fix the architecture.
Why shouldn’t we start fixing problems immediately?
Because changing a system you don’t understand generates new incidents while you’re investigating old ones, and you lose the ability to distinguish original problems from ones you introduced. You destroy your own diagnostic signal in the first week.
What should we instrument first if we have no observability?
Six things, in order: error rate at the boundary; latency at p50/p95/p99; deploy markers on those graphs; three or four business-critical transaction counts; saturation on the constrained resource; structured logging on the top failure paths. The business-transaction counts are the most valuable and the most commonly missing — they catch silent failures where nothing errors and nothing is slow.
Can you take over a codebase from another vendor?
Yes, and it’s a large share of this work. The hard part is rarely technical: access and credential transfer takes longer than anyone expects and blocks everything else. Find out why the previous vendor was replaced, too — sometimes that’s information about the client rather than the vendor.
What happens after stabilization?
An assessment, usually. A stable system can be evaluated properly; a burning one can’t. Most clients follow a stabilization sprint with a systems audit, because the rebuild-or-repair decision is worth making with evidence rather than under pressure.
If your system is on fire right now
The first thing to do is the incident catalogue. Last 90 days, what broke, how it was detected, how long to recover, whether anyone understood why. You can start it today and it costs nothing.
If you want engineers who have done this before to look at it with you, the 90-minute Code & Risk Review is free — two of them, no salespeople, and you keep the findings. If you already know you need help executing, that’s what the Stabilization Sprint is for.
