PIR-2025-03: generator fire and Hall 2 temperatures at SEC1

Drawing: PIR-2025-03
Title: Post-incident report
Rev.: A
Date: 2025-07-18
Drawn: TW
Checked: PF
Scale: Public
Sheet: —

01 Summary

Summary

Both utility feeders to SEC1 were lost at 13:41 on Tuesday, June 24, 2025, the second day of a regional heat wave. The site transferred to generator power as designed, without interrupting any customer load.

At 15:12 oil from a failed turbocharger seal on generator 3 reached the set’s exhaust manifold and set fire to the insulation around it. The engineer who saw the smoke stopped the set, and the fire was out by 15:16.

The fire was the visible failure. The more serious one followed it. With nine sets instead of ten, the generator controls shed load group LS-2, two of the five chillers, as they had been programmed to do. Those chillers served the air handlers in Hall 2. The programming assumed that a hall could ride through thirty minutes without cooling. At an outside temperature of 99 °F, Hall 2 rode through eighteen. Its supply air was above our committed range from 15:31 to 16:07, peaking at 91.4 °F, and fourteen customers reported equipment that shut itself down on temperature. No customer lost power. Hall 4, with its own 2N plant, and NWK1 were not affected.

02 Timeline

Timeline (Eastern Daylight Time)

Time	Event
13:41:06	Utility feeder 1 trips on the utility’s protection during a regional fault.
13:41:09	The automatic throw-over to feeder 2 begins; feeder 2 trips at 13:41:11.
13:41:20	All ten generators start. UPS carries the critical load on battery.
13:41:31	All ten sets on the bus; load transferred from battery by 13:41:44. No customer load interrupted.
14:05	Ambient temperature 99 °F in the generator yard; intake air on sets 1 to 5 at 112 °F.
15:12	An engineer on yard rounds sees smoke from the exhaust section of generator 3 and stops it with its emergency stop at 15:12:40.
15:13:05	With nine sets, the generator controls shed load group LS-2 as programmed: chillers 4 and 5, which serve the Hall 2 air handlers.
15:16	Fire out, using two portable extinguishers. No one hurt.
15:19	First status page update about the fire.
15:24	Fire department arrives; confirms the fire is out at 15:41 and leaves at 15:52.
15:31	Hall 2 supply air passes 80.6 °F, the top of our committed range.
15:38	Chiller 4 restored by manual override after the duty engineer confirmed spare capacity on the remaining sets.
15:42	Hall 2 supply air peaks at 91.4 °F.
16:07	Hall 2 back within range. Out of range for 36 minutes.
17:46	Utility feeder 2 restored.
18:20	SEC1 transfers back to utility power.
June 25, 06:30	Feeder 1 restored. Remaining nine sets inspected and load-tested by 09:00.

03 The fire

Why the fire started

Generator 3 had run 2,914 hours. Its turbocharger’s oil seal had begun to seep some time before June; the oil had soaked into the insulation blanket around the exhaust manifold, where it could not be seen without removing the blanket. Our inspection in April 2025 did not remove it. On June 24, after ninety minutes at high load in extreme heat, the seal failed outright, oil reached the manifold, and the soaked insulation caught fire.

The enclosure had no heat detection. The fire was found by an engineer on rounds, which is a procedure working as intended and a design we should not have relied on.

04 Hall 2

Why Hall 2 overheated

Our generator controls group loads by priority so that, if the site has less generation than it needs, the least important loads go first. In 2015 we put the chillers for Halls 1 to 3 in a lower priority group than the UPS systems, on the reasoning that IT equipment fails at once without power and only slowly without cooling. That reasoning was right in a cool hall and wrong in a hot one. We had tested the shed sequence, but never on a 99-degree day with a hall at full load.

Hall 2 was hit hardest because it carried the highest load of the three and because chillers 4 and 5 were the only ones on its loop that afternoon; chiller 3 was in planned maintenance.

05 Actions

What we have changed

#	Action	Status
1	Rewrite the load-shed sequence so that cooling for occupied halls is never shed before non-critical loads, and limit shedding to one chiller per lost set	Done, July 3, 2025
2	Replace the turbocharger on generator 3; inspect turbochargers and exhaust insulation on all ten sets	Done, July 11, 2025
3	Halve the turbocharger inspection interval, from 500 to 250 run hours, and replace oil-soaked insulation on sight	Done, July 2025
4	Fit heat detection and water mist in every generator enclosure	Done, October 24, 2025
5	Fit selective catalytic reduction and particulate filters to every set	Done, December 2025
6	Withdraw SEC1 from heat-wave demand response	Done, September 2025
7	Add a standing instruction: when ambient exceeds 95 °F with the site on generator power, a second engineer stays in the generator yard	Done, July 2025

06 Customers

What we owe customers

Supply air in Hall 2 was outside the range in our service level agreement for 36 minutes. Under the agreement that is a credit of 5% of each affected cabinet’s monthly recurring charges for each hour or part of an hour: 5% for every cabinet in Hall 2. We have applied it to every Hall 2 customer’s July invoice whether or not they asked, and waived all remote hands charges for June 24 at SEC1.

Some customers have asked whether we will pay for equipment that failed after the shutdown. The agreement limits our liability for service failures to its credits, and we have told those customers so. We have also asked to see the failed equipment, because if the temperatures in Hall 2 were enough to destroy hardware, we want to know.

07 Neighbors

Our neighbors

The generators ran for four hours and thirty-nine minutes that afternoon, and the smoke from generator 3 was seen from the homes west of the site. We have written separately to our neighbors and changed how we test; see the community page.