Ask an operator what keeps a data center up and you will get one of two answers. The first is the equipment: the generators, the UPS, the chillers. The second is the procedures. The second answer is the right one, and it is the subject of this article, because the equipment only does what the procedures tell people to do with it, and the procedures are where data centers most often fail.
Uptime Institute's outage analysis keeps returning to the same finding. In its 2026 analysis, 92% of respondents said human error was at least a minor contributor to their most recent impactful outage (VERIFIED, Uptime Institute, Annual Outage Analysis 2026, May 2026, Fig. 13, Resiliency Survey 2026). Failure to follow established procedures was the leading driver of human-error outages at 59% (VERIFIED, same source, Fig. 14, Resiliency Survey 2026, n=199). Human error is a contributing factor here, not a root cause, and the 92% must never be read as 92% of outages caused by human error.
Those are survey results with stated uncertainty, not exact counts, but the direction is unmistakable: the dominant preventable cause of outages is procedures, missing, flawed, or ignored.
There are three kinds of procedure in a data center, and confusing them is the first failure. They have different purposes, different authors, and different moments of use. Learn the distinction once and the rest of the library makes sense.
A SOP, standard operating procedure, is how routine work is done. It describes the normal operation of a system: starting and stopping a chiller, taking rounds, doing a shift handover, checking battery voltages. A SOP is written for the steady state. It assumes the plant is healthy and the task is one the team does regularly. Its virtue is consistency: the same task, done the same way, by any qualified operator, on any shift. Its failure mode is staleness, a SOP written when the plant was commissioned, never updated after a control change, describing a screen that no longer exists.
A MOP, method of procedure, is a detailed, step-by-step plan for a specific piece of work that can affect critical load. A MOP is written for one job, on one system, on one day: replacing a UPS battery string, racking out a breaker, transferring a load to maintenance bypass. It has numbered steps, with the expected indication after every state change.
What the operator should see after doing each step, so a wrong step is caught immediately rather than three steps later. It has prerequisites, notifications (who must be told before the work starts), a back-out plan for reversing the work if something goes wrong, and abort authority: who can stop the work, and under what condition. It has hold points, timestamps, initial blocks, and signature fields. A good MOP is very detailed, and a procedure library for a real facility runs to hundreds of documents.
The MOP's failure mode is authorship by someone who never did the task: steps in the wrong order, wrong expected indications, and no back-out plan that works, discovered while the load is at risk.
An EOP, emergency operating procedure, is the pre-authorized response to a predicted failure mode. Loss of utility power. Loss of cooling. A UPS on bypass with a second fault. A fire alarm in a hall. The EOP is written before the failure happens, reviewed by the owner, the operator, the engineer, and the constructor, field-validated after energization, and reviewed at least annually. It tells the operator what the first stabilizing action is, who does it, and what "stable" looks like, before the alarm, not during it. Its failure mode is the shelf: an EOP that was written, filed, and never drilled.
An EOP means little if it is not rehearsed, and the best readiness guidance says the test is to regularly simulate every EOP scenario, including disconnecting pumps, chillers, and air handlers to trigger authentic alarms. The target in one practitioner template is that the first stabilizing action is inside sixty seconds of opening the document. That is a demanding standard, and it is the right one: in a liquid-cooled hall, chip thermal limits can be reached in seconds when circulation is lost, not minutes.
The three types work together, and a worked example shows how. Take a UPS battery replacement, one of the most common high-risk jobs in a data center.
The SOP says how the UPS is normally operated: how to read its status, what normal looks like, what the alarm thresholds mean. The operator knows the SOP before the job starts.
The MOP is written for this specific replacement, on this UPS, on this day. It lists the prerequisites (the load, the bypass path, the notifications), then the steps in order: notify, verify the bypass path is available, transfer the UPS to maintenance bypass, with the expected indication after the transfer, and the warning that the load is now on unconditioned utility power with no stored-energy reserve, isolate the battery string, replace it, verify the new string, return the UPS to service, and verify the load is back on the UPS with the batteries charging.
At each step, the expected indication. At the end, the back-out plan and the abort authority. If the transfer to bypass does not produce the expected indication, the MOP says stop, and the operator stops, because the MOP said so in advance, not because they improvised in the moment.
The EOP is what the operator reaches for if the job goes wrong anyway: the UPS fails while on bypass, or the transfer does not complete. The EOP does not repeat the MOP. It says what the failure looks like, what the first action is, and who takes it. The operator does not have to decide, under alarm, whether to restore utility power or to shed load. That decision was made months ago, in a calm room, by people who had time to think.
Notice what the example shows: a procedure is a decision made in advance, by the people who will live with it, in the order they will use it. The library fails in predictable ways, written by someone who never did the task, copied from another site without checking this site's equipment, never field-validated, never drilled, or never reviewed after a control change.
The fix is the work of Weeks 33 through 40 of this series: one procedure per document, under document control; every MOP with numbered steps, expected indications, a back-out plan, and abort authority; every EOP field-validated and drilled until the first action is fast. The equipment is the plant. The procedures are what make the plant operable.
Further reading
- SOP / MOP / EOP guide, procedures that hold under pressure: the ten writing rules, the pass/fail quality checks, and drill vs tabletop.
- Case Study 4: SOP / MOP / EOP Authorship and Drill Validation, how procedures get good enough to trust at 3 a.m.
- Contact, start a conversation about a procedure review; a conversation, not a pitch.
Sources
- Human error as at least a minor contributor for 92% of respondents' most recent impactful outage (Fig. 13), and failure to follow established procedures as the leading driver of human-error outages at 59% (Fig. 14, n=199): Uptime Institute, Annual Outage Analysis 2026, May 2026, Resiliency Survey 2026.
- Chip thermal limits are reached in seconds when circulation is lost in a liquid-cooled hall.
- The sixty-second first-stabilizing-action target is a practitioner target from the AI Data Center Guide template, not a standard.
Published Saturday, September 19, 2026. By Scott Stillings. Scott is the author of Where AI Lives: Inside the Hyperscale Data Centers Powering the AI Boom (Inside Hyperscale). His background is in critical infrastructure operations. U.S. Navy submarine reactor operator, commercial nuclear operator and senior-reactor-operator-certified instructor, and hyperscale data center critical operations manager.