Building Operator Capability

Drawn from the author's operating experience. Client details are anonymized.

Data Center Strategies: Inside Hyperscale series Author: Scott Stillings

The point of this case, in one paragraph

A new data center can be built faster than the team that has to run it can learn it. This case is about closing that gap on purpose: hiring the operators before turnover, writing procedures for the plant that actually exists, and teaching the team the nuclear control-room habits of curiosity, psychological safety, trend reading, and seeing the whole picture, so that by the end of the first year the team, not the building, is the reliable part of the operation.

No uptime figure, cost saving, or efficiency improvement is claimed for this case, because none was measured in a form that can be published. What is recorded here is qualitative: what was done, and what changed in how the team worked.

1. Situation

A newly constructed, new-design, liquid-cooled AI facility was approaching turnover. The author was brought in to build the operation from scratch: hire the team in advance of turnover, prepare the standard operating procedures (SOPs) and emergency operating procedures (EOPs), set shift-operations expectations, and take over at turnover, the signing moment after which responsibility shifted instantly to a team that had never owned or operated a data center.

The team that was hired had real operating backgrounds, prior experience in the merchant marines and as power-plant operators, but none of them had operated a data center before, and none had operated this plant before, because the plant had never been operated by anyone. That is the normal condition of a new facility, and it is the condition most handover plans understate: the building is new, the cooling technology is new, and the people are new to both at the same time.

Three features of the situation made the capability gap wider than a headcount plan would show:

The technology was moving under the team's feet. Cooling technology was changing, direct evaporative cooling, chiller improvements, and direct-to-chip and rack-level liquid cooling, so experience carried over from air-cooled facilities, or from other industries, did not transfer directly. Knowledge of this plant had to be built internally, on this plant, rather than imported.
The knowledge base was thin by construction. In the author's nuclear background, depth of knowledge, procedures, and decades of accumulated operating experience were the norm. In a fast-grown data center, that depth had not had time to accumulate. There was no generation of senior operators to ask, because the facility, and in many cases the local industry, was younger than the questions being asked of it.
The first year was going to teach the team whether or not anyone planned for it. The following twelve months brought, as recorded in the author's account, a cluster of early equipment failures, seasonal cooling transitions, chiller tuning and calibration problems, and a cascading loss-of-cooling incident. The question was never whether the team would learn; it was whether the learning would be captured in writing and shared across shifts, or paid for one incident at a time.

2. What was found

The findings below are the author's account of the facility, stated qualitatively. No site, employer, date, or measured result is given, and no figure in this section is a measured outcome.

Finding 1: The team could follow a procedure, but could not yet predict the plant. The operators were capable and experienced in other high-consequence settings. What they did not yet have was a mental model of this specific plant: which indication moves first when a cooling loop degrades, what normal looks like on this system at this load, and which reading, if it changed, would mean something different from what it appeared to mean. That is not a hiring failure. It is what a new plant looks like on day one.

Finding 2: Procedures existed, but the knowledge behind them was still being built. SOPs and EOPs were prepared before turnover, as they should be. But a procedure written before the plant has been operated is a hypothesis about the plant.

Several procedures could only be validated, and in some cases corrected, once the plant was running at real load, at steady-state temperature, under its actual operating profile. The chiller tuning and calibration problems of the first year are the clearest example: control sequences that had never been exercised at real load did not behave as the commissioning record suggested they would.

Finding 3: The fastest-growing risk was the gap between the team's experience and the plant's novelty. Liquid cooling sharpened this. A direct-to-chip loop can drift invisibly, coolant composition, entrained air, stainless-steel passivation, pressure, and surface as damage months later.

That is documented industry guidance, not a finding about this site: ASHRAE Technical Committee 9.9 warned in its May 2026 bulletin on technology cooling system (TCS) coolant integrity that these failure modes can appear weeks after startup, once a loop reaches steady-state temperature under heavy load (see §6, Research and context). A team that has never seen a coolant-integrity failure cannot recognize its early indications unless someone teaches them what to trend and what to look for.

Finding 4: The signing moment was real, and irreversible in practical terms. Once the team signed for the facility, the responsibility was theirs. There was no period in which the construction team would keep operating the plant while the operators learned it. Whatever capability had not been built before turnover had to be built while operating, under load, on shift, at 3 a.m. as well as at 3 p.m.

3. What was done: the method

Method numbers in this section, the 27 Gate checks, the score bands, the 7-day review window, the Top-10 list, are the consultant's own method, not verified facts.

The method was translated, deliberately and explicitly, from the author's nuclear control-room background. It was not a copy of nuclear procedures, the plant, the hazards, and the regulatory setting are different, but a transfer of the habits that make a control-room team reliable when no single person knows everything. Four habits carried the work.

3.1 Curiosity as a job requirement, not a personality trait

In the nuclear control room, reliability is day-to-day attention to detail, trend, anticipation, and condition-based response. It is not heroics during an incident; it is the quiet work of noticing a trend before it becomes an alarm. The author taught that as a method:

Predict indications aloud before acting. Before an operator opened a valve, started a pump, or changed a mode, they said what they expected to see, which indications would move, in which direction, and by roughly how much. The prediction, not the action, was the point: a wrong prediction is information, and it is cheapest when it is caught before the action is taken.
Trend, not only alarm-watch. Operators were taught to read trends against the plant's own baseline, the normal established by this plant's data in its actual modes, rather than waiting for an alarm threshold. A slow drift in a controlled variable against its setpoint, or against its sister unit, is a finding. An alarm is a late finding.
Ask the question the procedure does not answer. Curiosity was treated as part of the job, and questions were treated as evidence that the operator was paying attention, not as evidence that they were unprepared.

3.2 Psychological safety: it is safe not to know

The single most load-bearing rule in the method was that it is safe not to know, and safe to say so, out loud, on shift, in front of the team. This was enforced by the author's own behavior, not by a poster:

Stop openly when indications differ. If the indications did not match the prediction, the operator stopped and said so. Stopping was the correct action, not a delay to be explained away. No operator was penalized for stopping a job because the plant was telling them something different from the procedure.
No blame for the gap between the procedure and the plant. When a procedure proved wrong or incomplete in the field, the finding was recorded, the procedure was corrected, and the correction was shared. The review asked what in the procedure, the training, or the plant made the outcome likely, not who was to blame.
The least experienced person can stop the job. Abort authority was real and named in the procedures: who may stop a task, and the words to use. A team that cannot stop is a team that will keep going past the point where stopping was cheap.

This matters most at 3 a.m., which is when the method is actually tested. An operator alone, tired, under alarm, with a procedure that does not quite match the indications, needs to know, from experience, not from a briefing, that stopping and calling for help is the expected behavior. That knowledge is only built if stopping is rewarded in daylight first.

3.3 Seeing the whole picture: the control-room team, not a single operator

The author's confirmed framing of the nuclear control room, which this case follows exactly, is that reliability comes from the control-room team scanning all indications together for the one that does not make sense, not from a single operator catching a trend on their own. This is not a single-operator story, and it is not retold as one here.

In practice, that meant:

Shift handover transferred risk, not status alone. The outgoing shift briefed the incoming shift on what had changed, what was degraded, what was being watched and why, and what the next shift should expect, a continuous narrative, not a checklist of completed tasks.
Redundancy was tracked as a live state, not a design property. The team recorded the actual redundancy state of each system each shift. A plant that is 2N on paper can be N on a Tuesday afternoon, a unit in maintenance bypass, a breaker racked out and never racked back, a loop isolated for repair and never returned to service. The board showed the truth; the design drawing showed the intention.
Every alarm had a disposition. Every alarm that could fire had a named procedure or a named "no action, monitor" disposition, so that an alarm flood did not bury the one alarm that mattered.

3.4 Building system knowledge internally: procedures, drills, and shift expectations

The capability was built through the ordinary instruments of operations, used in sequence:

Hire early, and be present during commissioning where the handover allows it. [Method prescription, not a recorded fact about this team: the account records that the core team was hired before turnover, as in Case Study 1; it does not record commissioning attendance, and Gate check 1 requires attendance to be evidenced by a log, not asserted.] The team should see the plant proved, and see where the proof was thin, before they own it.
Write the procedure library for this plant. SOPs for routine operation, MOPs (methods of procedure) for planned work that could affect critical load, each with numbered steps, expected indication after every state change, hold points, a back-out plan, and abort authority, and EOPs for the predicted failure modes, written so the least experienced authorized person could use them at 3 a.m. under alarm.
Field-validate every procedure. The author walked each procedure in the field with an operator who had not written it, on the real plant, in the real state the procedure assumed. Steps that could not be performed as written were rewritten, not annotated. EOPs were field-validated after energization.
Drill the EOPs instead of only reading them. Drills were rehearsals of a predicted failure, run until the first stabilizing action was prompt and practiced, not tabletop discussions. A tabletop is a discussion; a drill is a drill, and the record says which one happened.
Set shift expectations in writing. What it means to respond to an alarm at 3 a.m.: who is called, what is checked first, when to escalate, when to stop, and what is written down. Expectations that live only in the manager's head do not survive the first night shift.

4. Outcome, as recorded in the author's account

This section reports only what the author's account records, in qualitative terms. No measured outcome is claimed. There is no PUE figure, no downtime figure, no cost-saving figure, and no percentage improvement in this case study, because none is available in publishable form, and inventing one would mislead the reader about what a first-year capability program can prove.

What the account records is:

The team took over a new liquid-cooled AI facility and operated it through its first twelve months, including a cluster of early equipment failures, seasonal cooling transitions, chiller tuning and calibration problems, and a cascading loss-of-cooling incident. The first year was, in the author's account, a year of triage and learning, not a year of steady state.
The loss-of-cooling incident is the sharpest illustration of why the method mattered. The site ran a closed-loop cooling system of approximately 150,000–250,000 gallons, as recorded in the author's account [Author's account: roughly 200,000 gallons; the range is shown because the account gives an approximate figure, not a measured volume].
The thermal mass of that volume bought the team a minute or two of grace while chillers were restored, not the hours the volume might suggest. In a cold-plate hall, chip thermal limits can be reached in seconds once flow or cooling is lost, so a large water volume is a buffer measured in minutes at full load, not a ride-through.
The lesson recorded is that the mitigation is UPS-backed TCS pumps and CDU controls, and a drilled loss-of-cooling EOP with a prompt first stabilizing action, not the size of the loop. The 60-second first-stabilizing-action target sometimes quoted for such an EOP is [Practitioner target, AI Data Center Guide template, not a standard]; it is not a measured result for this site.
The figure sometimes cited for how fast throttling can follow a CDU pump failure, as little as 23 seconds, is a single cited case, not a general figure [VERIFIED for the seconds-scale claim; the 23-second figure is a single cited case, not a general rule].
Chiller tuning problems that did not appear during commissioning appeared at real load, because the control sequences had never been exercised at real load, at steady-state temperature, under the actual operating profile. The recorded lesson is that tuning is a first-year operations task, not only a commissioning defect to argue about: trend the controlled variables against setpoints and sister units, check sensor calibration before changing any setpoint, and route every setpoint change through change control.
Early equipment failures were treated as a triage problem, not a surprise. Each failure was logged with age and hours, batch and serial number, and operating conditions, and classified by cause, manufacturing defect, installation error, or unproven control sequence, because each has a different owner and a different fix, and because the failure log is warranty evidence before the clock runs out.

What did not happen, as far as the account records, is equally important: the account does not record a measured improvement in any performance metric, and this case study does not claim one. A team that is more capable after twelve months is not the same claim as a plant that is measurably more efficient, and the two should not be conflated.

5. What changed afterwards

The changes recorded in the account are changes in how the team worked, not changes in the plant's measured performance:

Procedures stopped being hypotheses and became records of the plant. Field validation and drill findings were fed back into the SOP, MOP, and EOP library, so the library described the plant as operated, not the plant as designed.
Trends replaced alarms as the first line of attention. Operators learned the plant's own baselines, per system and per mode, and deviations from baseline became findings before they became alarms.
The team could say what it did not know, and stop. Psychological safety, practiced in daylight, became the behavior under alarm. That is the change that is hardest to measure and easiest to lose, and it is the one the author regards as the foundation for the others.
Knowledge that had been held by individuals became held by the team. Shift handover, the redundancy board, the failure log, and the corrected procedure library are all mechanisms for making knowledge collective. A team that depends on one person's memory of a quirk is one resignation away from losing it.
The first year ended with the team knowing what it still did not know. Seasonal modes that had never been exercised, control sequences that had only been tested at partial load, and warranty items still open were listed as residual risks, not assumed to work. That list, not a claim of completion, is the honest output of a first year.

6. Research and context: why this case matters beyond one facility

This section is industry context from published sources, labeled per figure. It is not a description of the facility in this case, and its figures must not be read as results for that facility.

The human and procedure share of outages is rising. In Uptime’s 2026 analysis, 92% of respondents said human error was at least a minor contributor to their most recent impactful outage [VERIFIED, Uptime Institute, Annual Outage Analysis 2026, May 2026, Fig. 13, Resiliency Survey 2026], and failure to follow established procedures was the leading driver of human-error outages at 59% (n=199) [VERIFIED, same source, Fig. 14].

In Uptime's 2026 analysis, failure to follow established procedures was the leading driver of human-error outages at 59% (n = 199) [VERIFIED, Uptime Institute Annual Outage Analysis 2026, May 2026]. For Category 3–5 outages, 87% of operators believe better management and processes would have prevented their most recent incident, on the Uptime survey's own base [VERIFIED, Uptime 2025 survey PDF for the 87% figure, different base].

Hiring is the constraint. 53% of operators report difficulty finding qualified candidates for open roles, up from 46% in 2025 (REPORTED: Uptime Institute, Global Data Center Survey 2026, July 2026, Fig. 17; n=607 in 2026 versus n=650 in 2025, via a reputable secondary summary, primary not inspected).

The biggest skills gaps are electrical at 38%, junior and mid-level operations at 38%, operations management at 35%, and mechanical at 34%. A new facility cannot hire its way out of a thin knowledge base in the local market; it has to build the knowledge internally, which is the premise of this case.

Fatigue is a safety and reliability input, not a staffing inconvenience. Sustained overtime at or above 10% correlates very strongly with increased rates of incidents, including outages and serious injury [VERIFIED, Uptime Institute, "Top 14 Considerations for Addressing Data Center Facilities Management Risks", February 2021, practitioner guidance]. A rota that depends on sustained overtime is therefore a readiness finding, not a resourcing preference.

There is no single public operational-readiness standard. Commissioning is covered generically by ASHRAE Guideline 0 and Standard 202; the data-center-specific ASHRAE Guideline 1.6P was only at first public review draft in October 2026, not final [VERIFIED, draft status, October 2026].

Uptime's Operational Sustainability and M&O Stamp of Approval assess operations, but they are assessments, not a step-by-step readiness checklist an operator can download and follow [VERIFIED, Uptime Institute]. That absence is why a consultant's readiness framework must be labeled as the consultant's own method, not as a standard, and why this case study presents a method, not a certification.

Liquid cooling widens the knowledge gap. ASHRAE Technical Committee 9.9's May 2026 bulletin on technology cooling system coolant integrity warns that direct-to-chip loops can drift invisibly at commissioning and surface as damage months later, coolant composition, entrained air, stainless-steel passivation, and pressure ratings [VERIFIED, ASHRAE TC 9.9 bulletin, May 2026, documented guidance].

Schneider Electric and Data Center Knowledge document eight direct-liquid-cooling deployment challenges, including no widely adopted CDU efficiency and capacity standards and warranty ambiguity between cooling and IT vendors [VERIFIED, Schneider Electric / Data Center Knowledge, secondary]. The warranty-boundary problem is a governance problem as much as a technical one: when a failure sits between the cooling vendor, the IT vendor, and the installer, someone has to hold the common agreement.

The first year is a distinct phase, not an extension of handover. Commissioning continues for up to one year post-occupancy, deferred and seasonal testing, optimum-operation adjustment, a warranty review before expiry, and a lessons-learned workshop [VERIFIED, Data Center Knowledge / Vertiv on commissioning phases; Ethyco on the 10-month review].

Integrated Systems Testing cannot fully simulate seasonal variation, real IT load profiles, or the control drift that appears once a loop reaches steady-state temperature under heavy load. That is the structural reason the first year produces the failures it does, and the reason a bounded post-turnover watch exists as a service.

One figure deliberately not quoted here: the claim that nearly 70% of early equipment failures trace to design, installation, or startup deficiencies (Donato, 2013) is dated and unverified today [UNVERIFIED, Donato, Emerson/Vertiv ERS, 2013; do not present as a current statistic]. It is omitted from the argument above for that reason. Similarly, no market-size figure for transition-to-operations consulting is quoted, because none was found in public sources that measures this niche specifically [UNVERIFIED, no public source found; not quoted].

7. What a reader can take away

Hire before turnover, and be present during commissioning. The cheapest capability is the capability built before the signing moment. A team that first meets its plant on the day it owns it is learning under load.
Predict indications aloud, and stop when they differ. The habit costs seconds and saves incidents. It is the single most transferable practice in this case.
Make it safe not to know, and prove it by rewarding stopping. Psychological safety is not a value statement; it is a behavior the manager demonstrates when an operator stops a job in daylight. If stopping is punished, it will not happen at 3 a.m.
See the whole picture as a team. Scan all indications together for the one that does not make sense. Track redundancy as a live state. Hand over risk, not status alone.
Write procedures for the plant that exists, then correct them in the field. Field-validate every procedure with someone who did not write it. A superseded procedure in a drawer is a live hazard.
Treat the first year as a phase with its own work: baselining, trend review, seasonal-transition planning, chiller and control tuning, infant-mortality triage, and a warranty review at month 10, before the clock runs out.
Do not confuse a more capable team with a measurably better plant. Capability is a qualitative judgment until it is evidenced by trend data, drill records, and a failure log. Claim only what the evidence supports.

8. How the consulting offers map to this case

The mapping below describes what the offers cover for a client in a similar position. The three offers are used by their exact names, and nothing in this section is a promise of a result.

Offer mapping: what each offer covers in this case’s terms
Offer	What it covers, in this case's terms
Transition-to-Operations Readiness	The readiness assessment before the signing moment: people, procedures, asset information, and support arrangements checked before operations takes responsibility. Includes the scored Handover Gate: Sign, Sign with conditions, or Do not sign, applied per area or building. The case's hiring-in-advance, procedure-library, and field-validation work is the substance this offer checks.
Post-Turnover Operational Watch (90-day / 12-month)	The bounded first-year watch: baselining in the first 30 days, weekly trend and alarm review, seasonal-transition support, infant-mortality triage against a failure-mode catalogue, and the month-10 warranty-expiry punch-list. The case's first twelve months are the reason this offer exists.
Operational Efficiency & Procedure Review	Review of the SOP, MOP, and EOP library and operating efficiency: procedure quality checks, drill validation, and setpoint and cooling-efficiency opportunities. Advisory only, the consultant recommends operational changes and identifies where a design change would be needed, for the engineer of record to assess. No setpoint is changed by the consultant, and no PUE or saving figure is promised.

Scope and limits: read before inquiring

In scope: data-center operations; transition from construction to operations; operational efficiency. Operability review, handover criteria, and operations-and-maintenance input at the operator's side of handover are in scope.
Not offered: Tier or M&O certification (only Uptime Institute awards those); engineering design or a PE stamp; commissioning re-performance or witnessing and signing of commissioning test records; 24×7 cover, acting as duty manager, or operating the site; licensed or energized work, or directing energized work; a fixed price for undefined scope, or a fixed timeline before scoping; a zero-outage guarantee, a guaranteed uptime figure, or a guaranteed PUE or saving figure. Construction and build consulting is out of scope except for operability, handover, and operations-and-maintenance input as above.
"Power In, Heat Out, Nothing Down" is the book's core model, not a consulting promise. It is not presented here as an uptime guarantee, and no statement in this case study should be read as one.

Sources and labeling

Sources consulted for this case study:

The author's operating account (hiring in advance at a new liquid-cooled AI facility; SOP/EOP preparation; shift expectations; first-year infant mortality, seasonal transitions, chiller tuning, and cascading loss of cooling; nuclear control-room method).
Research report, 1–2 November 2026, industry context, outage data, commissioning levels, provider coverage.
Uptime Institute, Annual Outage Analysis 2025 (via CoreSite summary) and Annual Outage Analysis 2026, May 2026; Uptime Institute, "Top 14 Considerations for Addressing Data Center Facilities Management Risks", February 2021.
ASHRAE Technical Committee 9.9, TCS coolant-integrity bulletin, May 2026; ASHRAE Guideline 1.6P, first public review draft, October 2026.
Future-tech, "Data Center Best Practices Article 7: Data Center Project Acceptance and Operational Reporting" (2020, updated 2024), handover doctrine.
Schneider Electric / Data Center Knowledge on direct-liquid-cooling deployment challenges; AI Data Center Guide procedure skeleton (60-second first-stabilizing-action target [Practitioner target, AI Data Center Guide template, not a standard]).

Case Study 6: Building Operator Capability in a Fast-Growing Facility