Jennifer Wells
Toronto Star
September 27, 2003
Years after U.S. experts made recommendations to fix the province’s nuclear power stations, most remain idle.
Please see Tom Adams’ response: Safety deficiencies caused reactor shutdown
On the morning of Aug. 12, 1997, Dr. Allan Kupcis walked softly, as is his manner, down the hall from his office and into the executive boardroom at Ontario Hydro. The 19th-floor environs were familiar territory. Two years and eight months earlier, Kupcis had been named chief executive officer of the massive utility, the apogee of a Hydro career that had spanned almost a quarter century. As CEO, Kupcis presided over monthly board meetings, attended by the Hydro board itself and often a smattering of general counsel, office directors and vice-presidents, who were ringed around the periphery of the room. Aside from Jim Bullock, the chief executive officer of Laidlaw Inc. who had just been appointed to the board and was attending his first meeting, Kupcis had more than a passing familiarity with the board members who gathered that morning of the 12th. There was Bill Farlinger, the Hydro chair named to that post by premier Mike Harris in November, 1995. Eleanor Clitheroe, the utility’s chief financial officer. David Kerr, CEO of Noranda, and a host of others.
Of all the people who had been surprised by the ascendance of Allan Kupcis, the most surprised may have been Kupcis himself. The son of a stonemason born in Riga, Latvia, Kupcis was just 8 when his family immigrated to Canada. In time, he pursued a stellar academic career. He earned a Ph.D at the University of Toronto – his thesis was on strengthening mechanisms in copper single crystals – before being awarded a post-doctoral scholarship to study at Oxford. In 1973, he took a job at Hydro, in the research division, at a time when Hydro could see only go-go days ahead.
A 24-year career trajectory took Kupcis through research and into management and thence the job of chief operating officer. Ultimately, he landed in the top job, where he found himself buck naked in organizational terms. On the morning of the 12th, he knew he was living his final Hydro minutes. He had, actually, ordered up his own execution. Seven months earlier he had parachuted in a group of U.S. engineers to assess the state of the province’s woebegone nuclear reactors. Their report, which had been distributed to the Hydro board just days before the meeting, was incendiary, itemizing widespread and deeply rooted management deficiencies throughout the nuclear group. In the practical assessment of senior Hydro staff, it was widely assumed that someone, somewhere, would have to fall upon his sword.
Al Kupcis was the natural choice. "I was there," Kupcis says today of his own culpability. "I was in charge." If he didn’t resign, he says, he was sure to be removed "by government decree."
Sitting in a downtown hotel, sipping a cup of coffee, Kupcis generously relives that explosive piece of Hydro history, which, as it turns out, is hotly relevant to the mess that Hydro nuclear remains in to this day. He has a smooth face, clear blue eyes and a welcoming ability to explain Hydro issues in clear, penetrable terms.
Time has passed. Six years, in fact, since Al Kupcis delivered what one in attendance recalls as a spare, unemotional resignation speech.
Six years since Al Kupcis took what he describes as "kind of a sad elevator ride down" from Hydro headquarters on University Ave.
Six years. And yet the sole individual to pay the ultimate price for what would turn into a multibillion-dollar boondoggle, would appear to be … Al Kupcis.
Within hours of his departure, Hydro would announce the lay-up of seven of its nuclear reactors, including the oldest in the nuclear fleet, the so-called A units at Pickering. Now, more than three years after its initial projected restart date of June, 2000, Unit 4 is at last refurbished and approaching full power. The initial price tag of $800 million to $900 million to refit and restart all four units is a distant, foggy memory, that figure having been widely outpaced by Unit 4 alone. The cumulative to-date expenditures were tallied at $1.4 billion as of June 30. Using the long-ago prediction of a $400 million cost to get each of the other three units up and running takes the all-in projected cost to $2.6 billion. Given the explosive cost overruns to date, no one’s putting any faith in that $400 million figure. Nor is there any sign of conviction that the other three reactors, initially scheduled to restart in January and September of 2001 for Units 2 and 1, and June, 2002, for Unit 3, will ever come back on line.
Ontario Power Generation, which as recently as March announced it would complete a cost reassessment for the remaining three units by summer’s end, is now pledging to deliver detailed project schedules and expenditures some time early next year.
An investigation promised by the provincial government last November into why the project has so persistently run off the rails, was finally launched at the end of May.
The three-person committee chaired by former federal energy minister Jake Epp is to deliver an interim report some time after the provincial election, possibly to a new government that won’t know what hit it. Its final recommendations are to be made by year’s end. Included in the committee’s terms of review is a mandate to "make recommendations to the minister on means of improving the management of the project to restore the Pickering A generating station to full operation, including measures to ensure the cost effective and timely completion of the project."
That, need it be said, was the essential question posed by the Hydro file six years and hundreds of millions of dollars ago. Which leads one to ask, as a Hydro executive did recently, "What the hell happened?"
Head across the park from Hydro HQ, into the pink palace and up a couple flights of stairs and you’re in the legislative office of Sean Conway, Liberal MPP for Renfrew-Nipissing-Pembroke. Until Thursday that is.
After 28 years Conway is calling it quits. And it’s fair to say that the Hydro file won’t be the same without him. Conway has lived every inch of it. In part, because his constituency encompasses the research facility for Atomic Energy of Canada Ltd., which helped put him in the camp favouring the nuclear option as a key element in the province’s energy mix. In part because he’s a bit of an electricity egghead with a drop-dead memory for every inch of the Adam Beck legacy. In part because he can infuse Hydro talk with alluring references: he likens the Queenston-Chippawa generating station to Michael Ondaatje’s In The Skin Of The Lion waterworks. And in large part because he sat on an all-party select committee formed in September, 1997, to determine whether Hydro’s Nuclear Asset Optimization Plan, or NAOP, as drafted by the U.S. engineers retained by Al Kupcis, should be adopted. After months of testimony from 95 witnesses, the committee agreed that NAOP, which included the Pickering restoration, should be adopted to "return its nuclear plants to world class standards in terms of performance and safety in a fiscally and environmentally responsible manner."
Conway is still boyish at 52, still schoolboy eager to talk about meeting the bedevilling problem of meeting the province’s energy needs, and still pretty much wide-eyed about the way in which the refurbishment of Pickering will outlast his political career.
It wasn’t supposed to be this way. "It just infuriates me that we are sitting here six years since we launched NAOP and we’ve just empanelled three people to tell us why the thing has been bogged down in such a hopeless mess," he says. "I think there are some accountabilities here that have to be meted out. I don’t think of myself as a vengeful person, but I think there is a point at which the electricity ratepayer and taxpayer has every reason to expect that heads should roll."
In the alternative, a soft warm blanket of immunity seems to have settled over Hydro executives. Though rumours have periodically buzzed through the corridors of the Legislature and business community of pending executions, it hasn’t happened. Bill Farlinger remains as chair, a position he is contracted to fill until mid-2005. Ron Osborne, who joined the utility to wrestle it toward privatization, or not, remains CEO.
Sean Conway, who has been around long enough not to be surprised by nuclear surprises, considers the way in which Hydro has failed to respond to the ministrations of so many doctors. "Was there never a point where you said," he says, addressing an invisible ‘you,’ "it’s time to check in with the shareholder and say this can’t be done in the way we suggested and we better now start looking at Plan B?"
Al Kupcis was a natural Hydro recruit. He had done his undergraduate work on the "nuclear option." When he joined Hydro in ’73, the nuclear generating station at Pickering, the province’s first major commercial reactor, was just being brought on line. "If you wanted to stay sort of energy/fuel independent in this province, nuclear was the option," he says. "I recognized that the hydroelectric resources that were still available in the province were minimal, and that the coal stations in our system were used for peaking loads. … Those were the days when supply side was it."
Hydro then was the largest design and construction company in the country. The utility’s monopoly mandate demanded that it pre-plan and pre-build for anticipated growth. Beyond the first four units at the Pickering A station, the government would go big, big, big on nuclear. There would be an A station at Bruce and follow-up units. A doubling of the size of Pickering. And the unfathomable cost overruns at Darlington. The huge additions to the system, including the coal-fired plant at Nanticoke, would demand hundreds of transmission lines to connect the pieces. "Oh, I would describe it as a wonderful place for an engineering discipline," recalls Kupcis. "It was known around the world for its problem-solving skills in operating systems and it was just an exciting place to work."
Pickering, a first-generation Candu, was a learn-as-you-go experience. The bulk of Kupcis’s work revolved around developing remote inspection technologies. He supervised an engineering team that developed some of the initial tooling for pressure-tube inspection and in-reactor components that maintenance crews couldn’t get at. Among the early surprises at Pickering were cracks in the reactor’s rolled joints. The joints, which attached the pressure tubes to the face of the reactor, had started to stress. There were corrosion issues and the need to replace improper materials.
Technology problems weren’t exclusive to the nuclear group, says Kupcis, talking brightly about cracking in rotors at Nanticoke. But Pickering had chronic engineering issues: valve problems, steam generator problems. A pressure-tube rupture in the summer of 1983, forcing the shutdown of one of the reactors, raised the spectre of a multimillion-dollar repair bill. In a speech to the Empire Club in November, 1983, Hydro chair Milan Nastich defended the enormous cost of retubing the four Pickering reactors. "No one would scrap an otherwise sound car just because the piston rings need replacing," he said. The "sunk costs" at the reactors would repeatedly be used against those who likened increased funding at Pickering to throwing good money after bad.
Certainly Ottawa had a vested interest in keeping the "car" tuned. Atomic Energy of Canada Ltd., the vendor for the Candu, needed to secure a reactor sale annually if it hoped to make the business case for the Candu as a world-beating technology with a strong export market, an objective it would consistently fail to meet.
In 1988 Kupcis became director of corporate business planning for Hydro, heading up the executive ladder with a mandate to assess on a year- to-year basis the company’s capital programs going forward. Briefly, the province’s demand/supply equation followed its seemingly predetermined course, with demand growing at roughly 4 per cent a year. Then, in 1990, it stopped.
It couldn’t have been a more tumultuous time at the utility. Under CEO Maurice Strong, brought in by then premier Bob Rae, the utility launched its Task Force on Change. The concluding observations were damning: Hydro appeared as a calcified, arrogant monopoly with mounting debt woes and seemingly no levels of accountability.
Kupcis played a key role in the resulting cultural upheaval – and massive downsizing – that was launched upon the organization. Hydro’s very DNA was permanently altered with the decimation of the company’s design and construction group. Culturally, it was an incalculable upset. Design and construction were seen as the glory jobs, the sexy jobs, steering massive construction projects. What would turn out to be Hydro’s last 25-year demand/supply plan, which called for major investments in new nuclear, was scrapped.
‘I don’t think of myself as a vengeful person, but I think there is a point at which the electricity ratepayer and taxpayer has every reason to expect that heads should roll.’
Sean Conway, Liberal MPP
"The whole demand/supply plan, basically, within the context of the capital plan disappeared," says Kupcis.
The group had been headed by Lorne McConnell. "It doesn’t affect my ego at all, but the plan was thrown into the garbage," says McConnell, long since retired but still sharp as a tack on Hydro issues. "Now, of course, things are coming home to roost. … Today we have a situation in Ontario where there is no planning, and that’s what we’re suffering from. I know and I’m absolutely positive that within the next 15 to 20 years some politicians will come along and they’ll find that it would be a revolution to have a utility in Ontario that does some planning."
The prevailing view was that Hydro should be transformed into an operating utility. Maurice Strong set a long-term vision for the company – championing competitive generation and attempting to pry the doors open to the private sector – while in-house implementation of the new Hydro culture was left to Kupcis. In this, the utility’s nuclear operations were key. Attitudinally, nuclear had operated as a company within a company. Kupcis focused on a "let’s pay for performance, not for attendance" mantra within the significantly slimmed down organization. "It was," he says in his understated way, "a fascinating time." Roughly a third of Hydro’s staff was shed across an 18-month period, settling in the vicinity of 21,000 bodies. Attrition within the nuclear group would prove later to be devastating.
The nuclear reactors themselves had a less than lustrous record. Servicing more than 60 per cent of the province’s energy needs, the units throughout the system had proved to be chronic underperformers. While the industry regards an 85 per cent capacity factor as an indicator of excellent performance, Hydro’s nuclear system was operating in a range between 60 per cent and 74 per cent. Kupcis ordered that the plants undergo peer reviews by WANO, the World Association of Nuclear Operators, and its U.S. equivalent, the Institute of Nuclear Power Operators. "What we were getting on subsequent peer reviews were repeat findings, significant repeat findings," he says. Maintenance procedures and training were two areas that showed up time and again, without signs of improvement. Compared against its nuke peers, the Ontario reactors remained firmly in the middle of the pack, as opposed to their American counterparts, which, says Kupcis, had recorded significant improvements. "We seemed, as an organization, to be unable to fix our own problems." Hydro repeatedly launched management-improvement or quality-of-work programs, which resulted in little or no improvement. As much as Hydro resisted the ministrations of various doctors, it resisted management fads, too.
Late that summer of 1996, Kupcis hired a U.S. nuclear expert named Gregory Kane to dig down into the Pickering operations. Kupcis thought the operation’s problems were managerial, and he started to look to replace senior staff. Kane’s assessment was far more dramatic: The design safety margins at the plant were becoming unknown. "That to me was such an incredible statement," says Kupcis, "that I basically had no option but to find another mechanism to see what the issues were."
That "other mechanism" materialized in the form of Carl Andognini, an American ex-marine with a rep for turning around troubled plants south of the border. Andognini offered up a team of like experts who claimed similar successes. "It wasn’t Carl Andognini so much as it was the team I was interested in, the people he brought with him," recalls Kupcis. The group included Rick Machon and Gene Preston, both of whom had worked at the Tennessee Valley Authority (TVA) nuclear operations. That Andognini could deliver this package deal of nuclear experts was, says Kupcis, the "essential feature." He directed the group to undertake a six-month review of safety, managerial and cultural issues throughout the entire nuclear operation.
As CEO, Kupcis knew the risk he was taking. Reflecting on that time he says, "When I hired Carl we talked at length about possible outcomes, one of which was that I might not survive the process."
That Kupcis had retained Andognini et al. was not widely known among senior Hydro staff. "It was a surprise because Al had done it without relying on his usual lieutenants," says a former Hydro executive. "It was done and announced over Christmas, so that by the time we got back in January of ’97, the game was effectively in place. It was a huge internal blow."
The American crew was quickly dubbed the Magnificent Seven.
"You’ve got seven unknown Mr. Fixits to sit on top of this whole thing and essentially blow out the nuclear operating hierarchy," says the former executive. "I got the impression from the Magnificent Seven that they had been sent to the remotest backwater."
Remote, perhaps. But lucrative. In 1997, Associated Press ran a news item on executive bonuses at the TVA. Seven of the TVA’s nuclear officers were in line to receive bonuses of $25,000 per annum. At the time, TVA expressed the view that low salaries were preventing it from retaining the likes of Rick Machon and Gene Preston. Machon’s salary in 1996 was reported at $205,000 (U.S.)
At Ontario Hydro, million-dollar payouts would become the norm.
The Magnificent Seven made quick work of their review of Hydro’s nuclear operations. In just three months the crew conducted site-specific assessments of Bruce, Darlington and Pickering, ranging from safety-system function inspections to organizational effectiveness. On July 21, 1997, the group delivered its 15-volume Independent Integrated Performance Assessment (IIPA) to senior management, which, in early August, shared the findings with the Hydro board.
By the morning of Aug. 12, the bad news was widely known. Kupcis had insisted that the study be "brutally honest," and that is precisely what was delivered. In their findings, Andognini’s team broadly ranked all of the operating stations as "minimally acceptable." For all three stations, and across eleven rubrics including chemistry, engineering and maintenance, no single element drew an "excellent" or "satisfactory" rating. The predominate rating
– minimally acceptable – was defined as "substantially below industry standards." Over-all, said the report in its executive summary, "performance of (Ontario Hydro Nuclear) is well below the level of performance typically achieved by the best nuclear utilities. Immediate attention is needed to improve performance so that the value of OHN’s assets does not depreciate beyond recovery."
But what did that mean?
On Aug. 13, Bill Farlinger made a rare media appearance as Hydro chair alongside Carl Andognini to disclose the study results to the public. Famously, Farlinger likened the nuclear group to a "nuclear cult." The sound bite proved irresistibly delicious, and was quoted widely in the media. There was a crisis in nuclear.
But the predominate issue was not safety. "(T)he IIPA team confirmed that all of the plants were being operated in a manner that meets defined regulations and accepted standards related to nuclear safety," said the report. That wasn’t the message that got out. "The interpretation was that they were on the verge of self-destruction, and that’s not the right interpretation," says Kupcis.
While Kupcis’s chief concern going into the process was the red flag raised by Gregory Kane – impaired safety margins – he says what he got back from the Andognini group was that there were "some issues with material conditions … but there are straightforward technical solutions. … They said accountable and authoritative leadership is the biggest issue." The report put the then state of the nuclear group in a historical context. "Clearly," said the report’s authors, Hydro "did not make a smooth transition from its original and highly successful design and construction phase to the next stage, focused on operating and maintaining its 19 operating units."
Repeatedly, the report’s authors cited problems with processes over and above engineering concerns. Work processes were defined as "cumbersome and ill-defined," employees lacked a "questioning attitude," and the effective workings of the nuclear group were stymied by "organization walls." Many of the remedies could have been plucked from managerial recommendations common to any manner of industry: work functions needed to be aligned, integrated and cross-functional in order to be effective.
Additionally, appropriate processes and procedures were inconsistent at best. There were, said the report, "bad practices," such as allowing shift supervisors to approve "on-the-spot hand-written procedures." Procedural compliance, said the authors, must be standardized. Much of what Andognini’s people highlighted were boilerplate issues. At Pickering, weaknesses were identified in the areas of "maintenance leadership," work control, qualification and training
The Pickering assessment was performed in an 11-day period in the spring of ’97. The team was led by Gene Preston and its conclusions echoed the broad findings that applied to all the stations. Plant organization was found to be unnecessarily complex. "Lack of managerial leadership has reduced the effectiveness of over-all station maintenance and is demonstrated by the lack of a comprehensive maintenance plan and strategy."
Material issues that did surface – "An unlabelled Unit 4 8-inch extraction nonreturn valve near solenoid valve 4-64311-SV380 has an operating arm hinge pin leak" – did not, in Kupcis’s view, appear to be substantial, but rather demanded "straightforward technical solutions."
There were many cues in the report that the responsibility for the management disarray would be laid at the feet of Hydro’s executive team. "The generally relaxed atmosphere at headquarters (except in specific reactionary situations) does not reflect the level of intensity that the IIPA Team expected to find, given the reported conditions that exist at the stations."
"I should have understood the interaction of the angst in the organization as it was going through change and the angst at the employee level, of what the hell that means to them in a nuclear organization," says Kupcis. There was, he concluded, "a huge accountability issue, and I had been preaching accountability for 15 years."
As Al Kupcis made his way out of Hydro headquarters on that last, fateful day, the board turned its attention to Carl Andognini and other members of the Magnificent Seven who stood before them and recommended a radical remedy: that the Pickering A and Bruce A units be laid up.
Minutes of that board meeting are revealing of the cloistered environment in which the board scrutinized the information put before them. "Mr. Andognini," the minutes say, "stated that the entire senior nuclear management team is confident that, not only is this plan of action the best program for Ontario Hydro, it is a program that can and will be successfully managed to produce results." Andognini was supported in this view by Warren Peabody, himself a member of the Magnificent Seven and brought into the fold by Andognini. "Mr. Peabody," according to the minutes, "expressed his belief that the results of the IIPA are accurate." Gene Preston reported on the "existence of a high tolerance for complex and inefficient work processes."
The issues before the board were broader than Pickering. Performance woes at Darlington and at the B units had been documented in the same fashion. In his presentation to the board Rick Machon recommended that due to staff shortages, a state of affairs that harkened back to the massive downsizing years before, the optimal strategy would be to put existing resources into improving performance at the B units and at Darlington and to lay up the A units. There would, he recommended, be a phased-in recovery for the Pickering A units, starting in June, 2000, with the recovery of Unit 4, and concluding in June, 2002. While the board had been provided with details of the Andognini study, it wasn’t until the actual board meeting that the proposed recovery plan was put before them.
Not all of the directors immediately agreed with the initiative. According to the minutes, Jim Bullock, for one, "expressed the opinion that while the safe operation of Hydro nuclear facilities is of the highest priority, he is concerned that the Board has not had sufficient information and opportunity to exercise the due diligence required for approving the … decisions which will have very wide-ranging impacts." Eleanor Clitheroe concurred with Bullock and additionally pointed out that the board had been offered no comparative analysis. Were there other options, and if so, what were they?
On the 11th, then energy minister Norm Sterling had written to Hydro chairman Bill Farlinger urging that all options "be thoroughly evaluated and assessed by the board before decisions on a full recovery strategy are taken." That letter wasn’t distributed to the board until 4 p.m. on the 12th, after the board had voted on implementing NAOP.
‘No one would scrap an otherwise sound car just because the piston rings need replacing.’
Milan Nastich, then Hydro chair, in 1983
Fewer than 24 hours later Bill Farlinger stood before a throng of media and announced the immediate lay-up of seven of the utility’s 19 reactors – the four A units at Pickering and the three A units at Bruce.
The team charged with leading the restoration project at Pickering would be none other than the Magnificent Seven. The auditors had become the operators.
"It was a blitzkrieg," says Sean Conway. "It was yes, we’ve got a big mess and we’ve got to go, go, go." Conway smacks his hands together. "It was lickety split. Basically that day the board decided, we’re going ahead."
"There was a sense that this was the dream team," says one Hydro insider. "Al picked Carl and Carl picked the other boys. To diagnose issues, to identify flaws, problems, they probably did a good job … but could they manage one of the largest nuclear fleets in North America?"
It’s an interesting point. U.S. nuclear stations tend to be built in "two packs," to use the industry vernacular. The four-pack and eight-pack Canadian generating stations were certainly beyond the ken of the American experts size-wise, to say nothing of the distinct Candu technology, which had not been adopted anywhere in the United States, where light-water reactors are the norm. Pickering would be one of the largest nuclear restart efforts ever. And for a company already assessed as culturally dysfunctional, could it survive a massive management exercise under the command and control leadership of Carl Andognini and crew, many of whom were ex-marines who had been cycled through atomic submarines and into U.S. nuclear facilities? "It was a pretty unhappy time," says a former Hydroite.
In early September, the accounting firm Ernst & Young submitted a financial review on the recovery option, which had been requested by the board. The accountants considered the assumptions made by the board in adopting the plan – resolution of labour issues; significant productivity improvement targets; significant capacity improvements. "There are no detailed contingency plans to deal with any failure to achieve these key assumptions," said Ernst.
"As a result, the viability of the NAOP and integrity of financial estimates face significant uncertainty."
That some month, the all-party select committee was struck to review the IIPA and the recovery plan, including costs. "We were told that, yes, these people have the requisite background and training and we think they are the kind of team that can rescue this important but troubled operation," recalls Sean Conway. "They were talking about taking away 4,600 megawatts of baseload capacity … that’s not insignificant … but they were very clear that they would be able to get those four units back in roughly three years’ time."
There was, however, a fair bit of waffling in Andognini’s own appearance before the committee: "It’s been a constant learning curve … and I’m still learning and I’m still finding problems," he forewarned. Later he would describe his team’s report as a "living document (that) has to change to include things that we identify."
Those admissions appeared to have rung a note of alarm with some committee members. "Could I interrupt just a second?" asked MPP Marilyn Churley. "You’re saying that you are still discovering problems, which may add to the cost, that the cost we already have is a ballpark cost because you’re still in the process of discovering physical problems in different plants."
To which Andognini replied, "Ma’am, you’re absolutely right."
The Pickering project, in Andognini’s description, was low hanging fruit, an easy fix. Between 75 and 80 per cent of the problems were personnel-related, he said. The material concerns were not substantial, in part because of the earlier retubing of the reactors.
Jim Bullock, who continued to sit on the board until Hydro was splintered into the five successor companies we know today, was asked to appear before the committee. "When I asked what they wanted me at the all-party committee for, one of the Hydro representatives … said they wanted to know how you could spend $6 billion at your first meeting," he says today. That figure was one that had been thrown on the table as a cost tally to make all 19 units ship- shape. "The point I made … was that really the decision we were making the day of Aug. 12 was to set a course to follow for Hydro to restore its integrity and satisfy itself of the safety of the nuclear facilities."
During his time on the board, Bullock says, the refurbishment program was a standing item on the agenda, and that Andognini made regular representations to the board on his group’s progress. "I was satisfied that the U.S. people coming in didn’t come with a lot of baggage or prejudice from the past and were going to take a fresh look at this thing and set in place a new process here.
