Post-Turnover Operational Watch

Drawn from the author's operating experience. Client details are anonymized.

The first 12 months: infant mortality, seasonal transition, and the watch method that would have caught the drift earlier

Data Center Strategies · Case Study 2 · November 7, 2026

1. Situation

A newly constructed, new-design liquid-cooled AI facility was handed over to its operations team at turnover. The team had been hired in advance of turnover, the SOPs and EOPs had been prepared, and shift-operations expectations had been set.

At the signing moment, responsibility shifted instantly to a team that had never owned or operated a data center, a pattern the consulting method calls the Handover Gate moment, and one the industry's own handover doctrine describes plainly: no responsibility for "completed" construction areas should be taken on by site operational management without formal acceptance against pre-defined criteria, and ideally the core operations staff should be present during commissioning (Future-tech project-acceptance practice, 2020, updated 2024, VERIFIED, secondary practitioner source).

The facility was new in two senses that matter. It was newly built, so its equipment had no operating history. And it was a new design, direct-to-chip liquid cooling for AI load, so neither the plant nor, in most cases, the team had a season of operating history with that cooling architecture at that load. The 12 months after turnover [Author's account: the period covered by this case] therefore had to do double duty: operate the facility, and learn how it actually behaved.

Three things arrived in that first year, in the author's account:

Equipment infant mortality, a cluster of early-life equipment failures in the first months.
Seasonal cooling transitions, the move between free-cooling (dry cooler) and chiller postures, and back, at real load and real ambient conditions.
Chiller tuning and calibration struggles, control sequences that had never been exercised at real load, at steady-state temperature, under the actual operating profile.

Those three then combined into the event this case turns on: a cascading loss of cooling during the spring-to-summer transition, at the hottest period of the day.

Why the first year is the blind spot

Commissioning continues up to one year post-occupancy, deferred and seasonal testing that could not be completed during construction, adjustment of systems for optimum operation, a warranty review before the construction warranty expires, and a lessons-learned workshop (Data Center Knowledge / Vertiv commissioning guidance, VERIFIED, secondary).

Integrated Systems Testing (IST, Level 5) cannot fully simulate seasonal variation, real IT load profiles, or the control drift that appears once a loop reaches steady-state temperature under heavy load. That is not a criticism of commissioning; it is a statement about what commissioning can and cannot prove. The first year is where the unproven part gets proven, in operation, with live load behind it.

The industry data say the same thing from the other direction. Outages are getting less frequent overall, but the human and procedure share is rising: In Uptime’s 2026 analysis, 92% of respondents said human error was at least a minor contributor to their most recent impactful outage [VERIFIED, Uptime Institute, Annual Outage Analysis 2026, May 2026, Fig. 13, Resiliency Survey 2026], and failure to follow established procedures was the leading driver of human-error outages at 59% (n=199) [VERIFIED, same source, Fig. 14].

In Uptime's 2026 analysis, failure to follow established procedures was the leading driver of human-error outages at 59% (n=199) [VERIFIED, Uptime Annual Outage Analysis 2026, May 2026].

None of those figures is a claim about this facility. They are the reason a first-year watch is a service at all: the failures that appear after turnover are, in the main, procedure, tuning, and drift failures, the kind a structured watch is designed to catch.

2. What was found

2.1 Infant mortality arrived as a cluster, not a single failure

In the first months after turnover, equipment failed early and in a cluster [Author's account: cluster of early failures in the first 12 months; no count is given because no count was recorded in the account]. The consulting method's first-year failure-mode catalogue (F1) treats this as expected, not exceptional, and the triage question is not "why did something fail?" but "which kind of early failure is this?", manufacturing defect, installation error, or unproven control sequence, because each has a different owner and a different fix.

One nuance matters, and the case states it plainly because the industry literature does. The bathtub curve is a real model, but the USENIX study of ~100,000 disks found "rather than a significant infant mortality effect, we see a significant early onset of wear-out" [VERIFIED, USENIX FAST '07 disk-failure study, ~100,000 disks]. For facility equipment, the documented early-failure mechanism is more often design, installation, or startup deficiency and control or calibration error than component bathtub mortality.

The credible statement is therefore: new plants see a cluster of early failures from a mix of manufacturing defects, installation errors, and unproven control sequences, which is why burn-in, IST, and a first-year watch matter, not that a single bathtub percentage applies. A dated claim that nearly 70% of early failures trace to design, installation, or startup deficiencies (Donato, 2013) is UNVERIFIED today and is not quoted here as a current statistic.

2.2 The seasonal transition exposed what commissioning could not test

The cooling plant moved between free-cooling (dry cooler) posture and chiller posture as the seasons changed [Author's account]. The first hot season exposes a chiller, tower, dry cooler, or economizer mode never tested at that ambient; capacity shortfall or hunting at the changeover is the characteristic symptom (consulting method, failure mode F2). The watch method's rule is to run the transition as a planned, staffed event, not a background mode change, with a pre-season checklist two weeks before the expected changeover and a trend review two weeks after.

2.3 Chiller tuning and calibration drifted invisibly

Chiller tuning and calibration problems in the first year did not appear during commissioning, because the control sequences had never been exercised at real load, at steady-state temperature, under the actual operating profile [Author's account]. This is failure mode F3 in the method: a chiller hunts, short-cycles, or fails to stage; staging logic fights the load; sensors drift. The diagnostic discipline is trend over alarm, compare the controlled variable against its setpoint and against the sister unit, and check sensor calibration before changing any setpoint. Every setpoint change goes through change control.

The same drift mechanism is documented in the liquid-cooling literature: ASHRAE TC 9.9 warns that air and coolant problems can appear weeks after startup once the loop reaches steady-state temperature under heavy load, and that commissioning shortcuts "show up as failures long after the install crew has left" [VERIFIED, ASHRAE TC 9.9 May 2026 TCS coolant-integrity bulletin, as reported by StorageReview; the bulletin is documented guidance, not a standard].

ASHRAE TC 9.9's May 2026 bulletin also warns that direct-to-chip loops can drift invisibly at commissioning and surface as damage months later, coolant composition, entrained air, stainless passivation, and pressure ratings [VERIFIED, same source].

Silent compressor failures belong in the same family and are worth naming here, because they are the quiet version of the same drift: a compressor that has failed silently does not alarm as a failure; it shows up as a slow loss of capacity, a chiller that cannot hold its setpoint, or a sister unit carrying more than its share.

The watch catches it the same way, by trending the controlled variable against setpoint and sister unit, not by waiting for an alarm. Silent failures were happening at this facility: five to six silent compressor failures a day, as recorded in the author's account [Author's account]. Case Study 3 records the same plant's silent-compressor behavior in detail. The drift-detection method covers them, whether or not they alarm.

2.4 The cascade: loss of cooling at the hottest period of the day

During the spring-to-summer transition, at the hottest period of the day, chiller failures cascaded to a complete loss of cooling [Author's account]. Thousands of servers were at risk [Author's account: "thousands of servers at risk"; no exact server count is given because none was recorded in the account].

The team kept a single facility water pump running, and recirculation plus the residual heat capacity of the closed loop, approximately 150,000–250,000 gallons [Author's account: roughly 200,000 gallons; the range is shown because the account gives an approximate figure, not a measured volume], bought a minute or two of grace [Author's account: "a minute or two"], while one chiller was restored and then a second, over several hours [Author's account: "over several hours"].

Two facts in that paragraph deserve emphasis, because both correct a natural misreading:

Thermal mass is a buffer measured in minutes at full load, not a ride-through. A 150,000–250,000-gallon loop sounds like hours of protection. It bought a minute or two. In a cold-plate hall, chip thermal limits can be reached in seconds once flow or cooling is lost, the method cites a single case of a CDU pump failure reaching throttling in as little as 23 seconds [VERIFIED for the seconds-scale claim; the 23-second figure is a single cited case, not a general rule].
The mitigation is therefore not the volume of water; it is UPS-backed TCS pumps and CDU controls and a drilled loss-of-cooling EOP whose first stabilizing action is inside 60 seconds of opening the document [Practitioner target, AI Data Center Guide template, not a standard].
The restoration was measured in hours, the grace in minutes. That gap, minutes of buffer against hours of restoration, is the whole argument for the watch. Everything that shortens restoration, or that catches the drift before it cascades, has to be built before the day it is needed.

3. What was done: the method, and the watch that would have caught the drift earlier

Method numbers in this section, the 27 Gate checks, the score bands, the 7-day review window, the Top-10 list, are the consultant's own method, not verified facts.

The team in the case did the hard part well: it kept a pump running, and it restored chillers one at a time over several hours [Author's account]. The account records no server-loss or downtime figure for the event, the pump kept running, the chillers were restored one at a time over several hours, and no outcome beyond that is recorded, so no protected-load or zero-impact outcome is claimed here. What follows is not a criticism of that response.

It is the watch method, the structured first-year discipline that would have caught the drift earlier, so that the response would have been a planned intervention rather than an incident. The method is the consultant's own, labeled as such; there is no public standard "first 90 days" program for data-center operations, and any 30/60/90-day framework must be labeled as the consultant's own method, not a standard [VERIFIED, transition-to-operations research, November 2026: no standard first-90-days program was found].

3.1 The weekly check-in

The watch runs on a weekly check-in with a fixed agenda, each producing a one-page note, decisions, owned actions with dates, and an updated redundancy board, filed with the shift log:

Redundancy board, the actual redundancy state of each system (N / N+1 / 2N / degraded), any erosion, and return-to-service dates. A plant can be 2N on paper and N on a Tuesday afternoon.
Failure log, new failures since last week, by system, with age and hours and warranty status; any repeat failure flagged. Every failure is triaged against the first-year catalogue (F1–F12): infant mortality, seasonal transition, chiller tuning and calibration drift, cascading loss of cooling, alarm floods, warranty-boundary disputes, redundancy erosion, coolant-integrity drift, controls or supervisor loss, maintenance-induced failure, generator or transfer failure on demand, and fire, leak, or water ingress interacting with cooling.
Trend review, the week's trends against baselines: supply and return and approach temperatures, CDU flow, pressure, and conductivity, UPS load and battery, generator test results, alarm rate. Trend over alarm: rate-of-change and deviation alarms on trended data. This is the check that catches chiller tuning drift (F3) and silent capacity loss before either becomes a cascade (F4).
Alarm review, alarm count per operator, any flood, bad actors, shelved or suppressed alarms and their expiry.
Work performed and planned. MOPs completed with return-to-service verification, MOPs planned, and any setpoint or sequence change with its change-control record.
Gate conditions, progress on any "sign with conditions" items from the Handover Gate; any new condition.
Risks for next week, seasonal change, load change, vendor visit, maintenance window, staffing gap.

3.2 Baselining in the first 30 days

The first month establishes the baseline for every trended variable. Normal is defined by the plant's own data, not by a vendor brochure. Baselines are set per system and per mode, normal, economizer, transfer, maintenance, and re-baselined after any material change (load step, seasonal change, plant modification).

A baseline without a stated mode and load is not a baseline. In this case, baselining is what would have made the chiller drift visible: a controlled variable moving away from its setpoint, or away from its sister unit, is a finding on a trend chart weeks before it is an incident.

3.3 Seasonal-transition checklists

Run by site staff as a planned, staffed event, two weeks before the expected changeover and again two weeks after; the consultant observes and records. Into the hot season: the site's own chiller, tower, and dry-cooler capacity evidence checked against forecast load and ambient; economizer changeover points checked against the site's sequence-of-operations record; condenser and tower water treatment and blowdown checked; CDU and cooling-loop chemistry sampled; alarm setpoints for high temperature reviewed; load-shed and transfer provisions confirmed; spares for the hot season (fans, pumps, filters) checked.

After each transition: trend review against the baseline, with any hunting, short-cycling, or staging fight logged as a tuning action, and lessons recorded for the next year's transition.

3.4 The month-10 warranty-expiry punch-list

Issued at month 10, not month 11: a failure-log review classifying every failure in months 0–10 (warranty claim, vendor defect, installation defect, operations, wear), with every warranty-classified failure carrying a claim reference or a written reason it was not claimed; an issue-log sweep so nothing warranty-covered is left to expire unclaimed; seasonal evidence, with any mode never exercised listed as a residual risk rather than assumed to work;

coolant evidence across the 10 months; controls evidence (alarm rationalization results, setpoint-change log, supervisor-loss and failure-domain re-tests after any change); and a punch-list per item with description, evidence, owner, due date before warranty expiry, and consequence if it slips past expiry. The watch ends with a close-out report and a lessons-learned workshop.

3.5 Incident learning, not incident blame

After any incident or near-miss: stabilize and restore using the EOP; preserve the record (BMS/EPMS trends, sequence of events, alarm log, shift log, MOP in use); within 7 days, a structured review, what happened, what the procedure said, what the operator did, where they diverged and why, what the plant did, what the indications showed; owned actions with dates; and the EOP or MOP re-issued and re-drilled if it was changed. The review asks what in the procedure, the training, or the plant made this the likely outcome, not who is to blame.

4. Outcome as recorded in the author's account

The outcome is qualitative, because that is all the account records, and this case does not invent numbers to fill the gap:

The team kept a single facility water pump running through a complete loss of cooling [Author's account].
Recirculation plus the residual heat capacity of the approximately 150,000–250,000-gallon closed loop bought a minute or two [Author's account].
Thousands of servers were at risk during the event [Author's account]; no server-loss, downtime, PUE, or cost figure is given, because none was recorded in the account, and none is estimated here.
One chiller was restored, then a second, over several hours [Author's account].
The facility continued into the remainder of its first 12 months with the infant-mortality, seasonal-transition, and tuning problems as its standing first-year work [Author's account].

No measured PUE improvement, no downtime figure, no cost saving, and no percentage improvement is claimed for this case. Any such figure would be fabrication.

5. What changed afterwards

What changed was the operating posture, not the plant. The lessons the author draws from the year, and the ones the watch method is built to install from day one, are:

Early failures are triaged, not greeted as surprises. Log every failure with age and hours, batch and serial, and operating conditions; separate manufacturing defects from installation errors from unproven sequences, because each has a different owner and a different fix; use the failure log as warranty evidence before the clock runs out.
Tuning is a first-year operations task, not a commissioning defect to argue about. Trend the controlled variables against setpoints and sister units, check sensor calibration before changing any setpoint, and route every setpoint change through change control.
Thermal mass is not a ride-through. Minutes of buffer against hours of restoration means the protection has to be the drilled EOP, the UPS-backed pumps and controls, and the trend review that catches drift early, not the volume of water in the loop.
The seasonal transition is a staffed event. The changeover is planned, checked two weeks before and reviewed two weeks after, and its lessons are written down for the next year.
The warranty clock is a deadline, not a background fact. The month-10 review exists so that failures the contractor owns are claimed while the contractor still owns them.

6. What a reader can take away

If you are taking over a new facility, or are already in its first year, the transferable lessons are:

Budget the first year as a learning year, not a steady-state year. Commissioning proves the plant at a point in time; the first year proves it across seasons, load profiles, and drift. Staff and schedule for that.
Trend, not only alarm. An alarm tells you something has already failed. A trend against a baseline tells you something is drifting. The chiller that hunts today is the cascade next month.
Name the redundancy you actually have, every shift. The redundancy board, actual state, not design state, is the cheapest control in the building.
Treat the seasonal changeover as an operation. Staff it, check it before and after, and write down what it taught you.
Start the warranty clock on day one. Schedule the month-10 review at turnover; by month 11 it is too late to claim what the log never recorded.
Write the loss-of-cooling EOP for the least-experienced authorized person at 3 a.m., and drill it as an injected fault, not a tabletop. A tabletop is not a drill. The first stabilizing action needs to be fast, the guide target is inside 60 seconds of opening the document [Practitioner target, AI Data Center Guide template, not a standard], and speed like that only comes from rehearsal.

7. How the consulting offer maps to this case

Post-Turnover Operational Watch (90-day / 12-month) is the second of Data Center Strategies' three offers (the others are Transition-to-Operations Readiness and Operational Efficiency & Procedure Review). It is a bounded first-year operator watch after turnover: baselining, trend review, control-sequence tuning advice, seasonal-transition support, and a warranty-expiry punch-list. The consultant is the operator's second pair of eyes and the keeper of the baseline; the site's own team operates the plant.

Watch elements: what each does and where it would have caught the drift in this case
Watch element	What it does	Where it would have bitten in this case
90-day watch: weekly check-in, trend and alarm review, failure-log triage, procedure and drill catch-up, Gate conditions close-out	Catches drift and early failures in the window where they cluster	Infant-mortality triage (F1); trend review catching chiller drift (F3)
12-month watch: as above, monthly after day 90, plus seasonal-transition checklists and the month-10 warranty review	Covers the full seasonal cycle and the warranty deadline	Spring-to-summer transition (F2); month-10 punch-list
Baselining (first 30 days)	Defines normal from the plant's own data, per system and per mode	Makes tuning drift visible before it cascades
Seasonal-transition support	Planned, staffed changeover with pre- and post-checks	The transition at which the cascade occurred
Close-out report and lessons-learned workshop	Records what the first year taught and what the next site should do differently	Turns one facility's year into repeatable practice

Scope and boundaries, stated on the page, not in the small print. This offer is advisory and operator-side. It is not 24×7 cover, not acting as duty manager, and not operating the site. It does not design or re-design plant, does not carry a PE stamp, does not perform or re-perform commissioning (L1–L5 / IST) or sign Cx test records, and does not direct licensed or energized work, all live work follows the site's permits, lockout/tagout, and NFPA 70E practice, performed by the site's own authorized persons.

It does not award or guarantee Tier or M&O certification, only Uptime Institute awards those, and it promises no uptime, PUE, or saving figure and no zero-outage outcome. Pricing is set only after scoping; no fixed price or fixed timeline is quoted before the scope is written. The watch starts after turnover, not before it; if turnover slips, the watch start slips with it. Each watch is a dedicated engagement, staffed for that site; concurrent watches are scoped and staffed as separate engagements.

Who it is for. The owner, developer, or operator who will live with a new plant through its first seasons, most usefully the operator who takes responsibility at turnover and wants a second pair of eyes through the window when infant mortality, seasonal change, and control drift actually appear.

8. Research and context

This section is the sourced industry context behind the case. It is context, not a claim about the facility in the case.

Human error and procedures. In Uptime’s 2026 analysis, 92% of respondents said human error was at least a minor contributor to their most recent impactful outage [VERIFIED, Uptime Institute, Annual Outage Analysis 2026, May 2026, Fig. 13, Resiliency Survey 2026], and failure to follow established procedures was the leading driver of human-error outages at 59% (n=199) [VERIFIED, same source, Fig. 14].
Failure to follow established procedures was the leading driver of human-error outages at 59% (n=199) in 2026 [VERIFIED, Uptime Annual Outage Analysis 2026, May 2026]. A different Uptime cut reports 87% for Category 3–5 outages, a different denominator.
Cost of getting it wrong. 57% of most recent significant, serious, or severe outages cost more than $100,000; 1 in 5 cost more than $1 million [VERIFIED, Uptime Annual Outage Analysis 2026, May 2026; base: respondents’ most recent major outage, 2025 survey]. Cooling was ~13% of most-recent impactful outages [VERIFIED: ComputerWeekly reporting Uptime 2025; denominator is respondents' most recent outage. Do not present the 45% power figure (2025, base: respondents who had an outage in the prior 3 years; 56% in 2026) and the ~13% cooling share-of-most-recent-outage figure as the same measure; they have different bases].
Staffing. 53% of operators report difficulty finding qualified candidates for open roles, up from 46% in 2025 (REPORTED: Uptime Institute, Global Data Center Survey 2026, July 2026, Fig. 17; n=607 in 2026 versus n=650 in 2025, via a reputable secondary summary, primary not inspected). The biggest skills gaps are electrical at 38%, junior and mid-level operations at 38%, operations management at 35%, and mechanical at 34%. Sustained overtime at or above 10% correlates very strongly with increased rates of incidents, including outages and even serious injury [VERIFIED: Uptime Top 14 guidance, February 2021, secondary practitioner guidance].
No single readiness standard. There is no single public operational-readiness standard: ASHRAE Guideline 1.6P was only at first public review draft in October 2026 [VERIFIED, ASHRAE public-review draft, October 2026; draft, not final], and Uptime's Operational Sustainability and M&O are assessments, not a step-by-step readiness checklist. That absence is the market opening, and it is why the watch method in this case is labeled as the consultant's own method, not a standard.
Liquid cooling is newer and less standardized. ASHRAE TC 9.9's May 2026 bulletin on technology-cooling-system coolant integrity warns that direct-to-chip loops can drift invisibly at commissioning and surface as damage months later [VERIFIED, ASHRAE TC 9.9 bulletin, as reported by StorageReview]. There is no widely adopted CDU efficiency or capacity standard, and warranty ambiguity between cooling and IT vendors is a documented deployment challenge [VERIFIED, Schneider / Data Center Knowledge liquid-cooling deployment coverage].
Commissioning continues into the first year. Deferred and seasonal testing, optimum-operation adjustment, warranty review, and lessons-learned all fall in the first year after occupancy [VERIFIED, Data Center Knowledge / Vertiv commissioning guidance; Ethyco first-year and 10-month review]. L6 is explicitly post-occupancy and seasonal testing [VERIFIED, GATE Energy L0–L6 table].
What could not be verified. No public search volumes, consulting prices, or client names for this niche were found in the research; none are given here. The "nearly 70% of early failures" figure (Donato, 2013) is dated and unverified today and is not used as a statistic. Recent 2025–26 cooling incidents reported in trade and blog sources were not primary-verified and are not cited here as established fact.

Sources. Uptime Institute Annual Outage Analysis 2025 and 2026; CoreSite summary; ComputerWeekly; Uptime Top 14 (Burgess, February 2021); Future-tech project acceptance and operational reporting; Ethyco commissioning services; ASHRAE TC 9.9 TCS bulletin via StorageReview; ASHRAE Guideline 1.6P first public review draft, October 2026; Schneider liquid-cooling challenges; Data Center Knowledge liquid-cooling deployment; AI Data Center Guide MOP/SOP/EOP skeleton; USENIX disk-failure study. Data Center Strategies consulting method (§2.2, Post-Turnover Operational Watch; failure-mode catalogue F1–F12).

Case Study 2: Post-Turnover Operational Watch