January 23, 2003
RP-2002-0133 Exhibit L, Tab 9
Ontario Energy Board
IN THE MATTER OF section 36 of the Ontario Energy Board Act, 1998, S.O. c.15, Sched. B, as amended;
AND IN THE MATTER OF an Application by Enbridge Gas Distribution Inc. for an Order or Orders approving or fixing rates for the sale, distribution, transmission and storage of gas.
Witness: Norman Rubin
2003 January 23
Q What is the subject of this evidence?
A This evidence addresses Issue 9 from the Board’s Procedural Order #2, “Demand Side Management (DSM) (A7/T1/S1)” and focuses primarily on the following sub-issues:
- Issue 9.2, “Review of the Shared Savings Mechanism (SSM) incentive scheme”;
- Issue 9.5, “Review of the DSM Audit Process”; and
- Issue 9.6, “Recovery of SSM and LRAM for 2000 and 2001(subject of Dec/2002 SSM ADR Settlement Conference)(A8/T3/S1)”.
For convenience of presentation, I would like to deal with these issues in reverse order.
Q What are your qualifications?
A My Curriculum Vitae is attached to this evidence as Appendix A.<!–Don’t forget to do it.–> I have represented Energy Probe at several OEB hearings including EBO 169-III, and on the Demand-Side Management Consultative Groups of both Enbridge and Union since the conclusion of 169-III. In addition, I served as one of three independent members of the Audit Subcommittee that supervised the audit of Enbridge’s 2000 DSM activities, selected by the DSM Consultative Group. I also represented Energy Probe at the ADR settlement conference in the case RP-2001-0032.
Q Didn’t you, acting for Energy Probe, oppose utility-led DSM in EBO 169-III?
A Yes. We still love efficiency, but we still believe that there are better ways to achieve efficiency than with utility-led DSM programs. But since we lost that fight years ago, I’ve remained involved in the DSM process, attempting to ensure that the Board’s prescriptions are carried out as well as possible, in a way that advances the public interest. I have steadfastly resisted reopening the arguments we lost in EBO 169-III. I believe that my selection as a member of the 2000 independent Audit Subcommittee by the members of the DSM Consultative Group represent some public recognition that I’ve been carrying out that responsibility with some success. Indeed, I understand that I recently failed to be reelected to the 2001 Audit Subcommittee only after a tie-breaker vote, at a Consultative Group meeting I was unable to attend, so those views have apparently not materially changed in the past year or two.
Q Beginning with Issue 9.6, “Recovery of SSM and LRAM for 2000 and 2001” – how did this matter come before the Board today?
A In attempting to negotiate an ADR Settlement Agreement to resolve these matters as part of RP-2001-0032, pursuant to Procedural Order #6, settlement was not achieved. A partial Settlement Agreement, not endorsed by Energy Probe or by CME, was submitted to the Board on December 23, ten days following the deadline for settlement proposals specified in that Procedural Order. The Board wrote to the parties on 2003 January 8, indicating that the issues of recovery of SSM and LRAM for 2000 and 2001 will be considered in this case.
Q What are your major recommendations concerning Issue 9.6, “Recovery of SSM and LRAM for 2000 and 2001”?
A The following are my major recommendations on Issue 9.6:
- The Board should ensure that the findings of the independent 2000 DSM audit are incorporated faithfully into the 2000 SSMVA clearance, and not ignored either in whole or in part. Specifically, it is not appropriate for the Company to suggest (or the Board to agree) that the year-2000 SSM — determined by audit to be a near-zero sum of two components, one positive and one negative — should be settled by including the positive part and excluding the negative part.
- The year-2000 SSM clearance should result in $69,262 being returned to the ratepayer for 2000.
- The Board should delay clearing the year-2001 SSM until the independent DSM audit process for that year has concluded. But we urge the Board to clearly endorse audit and SSM principles for 2001 and 2002 that are consistent with the principles of the independent members of the 2000 Audit Subcommittee, and to reject the Company’s principles where they differ. Those recommended principles are outlined below in the detailed discussion of the 2000 Audit results. If the Board establishes clear principles, we expect that the 2001 SSM claim will be easily resolved by ADR settlement according to those principles, following the conclusion of that independent audit process.
- Finally, although the SSM for 2002 is not formally part of this hearing’s Issue 9.6, we note that that year’s DSM activities are complete, although not yet evaluated or audited. We therefore believe that its SSM clearance should also be guided by our principles, endorsed at this Hearing by the Board and carried forward through the Company’s 2002 evaluation report and tested and (if necessary) corrected by an independent audit.
Q How did you arrive at your recommendation for a negative $69,262 SSM for 2000?
A The recommendation flows from a series of adjustments to the Company’s claim, following the findings of the independent Audit of 2000 DSM activity. Those adjustments — mine, and also those of the other two independent members of the 2000 Audit Subcommittee — are laid out clearly in Kai Millyard’s July 31, 2002 Reconciliation Report, Table 2.
Q Does it bother you – and should it bother the Board – that the SSM is negative?
A In one sense no; in another sense, yes.
Q Please explain.
A On the one hand, the SSM was clearly and consciously designed from the start to be a symmetrical incentive, embodying both reward and punishment, the results to be determined by a mathematical formula that compares actual TRC results to “target” results. If putting the true results into the formula generates a negative answer, then that is the correct answer, and anything other answer is wrong.
On the other hand, it is clear that the Company generated significant positive TRC benefits with its 2000 DSM activities, and it seems churlish to penalize shareholders just because those benefits were slightly below some arbitrary “target”.
Putting these two reactions together, I believe the SSM formula itself is to blame, and should be changed in future — pursuant to the discussion in Issue 9.2 — to eliminate its most unfortunate effects.
Q Let’s come back to Issue 9.2 later. What were the biggest changes to the SSM from the Audit’s findings?
A According to all three independent members of the Audit Subcommittee, the largest single change flowed from the Audit’s revelation that roughly half of the gas savings that the Company claimed from large, one-of-a-kind, hand’s-on projects — so-called “custom” projects — were actually improperly attributed to the Company’s DSM activities, because the savings would have occurred with or without the DSM program.
Q Am I correct that these customers who would have made the savings without the program are called “free riders”?
A That is correct.
Q How much difference did the revelation about free riders make?
A The answer varied somewhat among the independent members of the Subcommittee, because we differed in the way we tempered justice with mercy. For Mr. Rowan (of CME) and myself, the net difference was a deduction of $4,499,694 from this one adjustment alone. For Mr. Neme (of GEC), the net difference was “only” a deduction of $3,588,762.
Q So even Mr. Neme’s smaller adjustment here eliminated over half of the $6,547,660 SSM claim the Company took from the Audit Subcommittee process to the ADR?
Q And why was Chris Neme’s adjustment smaller than yours and Malcolm Rowan’s?
A When the Auditors presented their raw results from statistically sampling the Company’s customers for “custom” DSM projects — a job that’ s normally considered evaluation, rather than auditing, and only fell to the Auditor because it became clear during the audit that the Company had not done it in its Monitoring and Evaluation Report — they were even more brutal than the results in Kai Millyard’s chart. On average, only 42% of the claimed savings survived the screen of free ridership!
After we all saw these raw results, we frankly started thrashing around, along with the chief auditor, trying to find ways in which the sample might have been artificially harsh for the Company, and trying to devise what we all called “generous alternative scenarios” and calculations. The auditor proposed a number of these “generous alternative scenarios” in Section 5 of the Audit Report, beginning at p. 5-16 and culminating in Table 5-18, “Suggested Ranges for Effective Free Rider Assumptions”. Given the discussion in the audit report, and my general intention to follow and implement the Audit Report’s findings, I adopted the middle of that table’s three alternative scenarios, or 49% overall average free ridership for custom programs. So did Malcolm Rowan. Chris Neme decided to give the company even more generosity than the Audit Report did, by treating the custom-projects free riders who dealt with Energy Services Companies (“ESCOs”) differently from the others, on the principle that the Company had less (or no) control over customers lined up by ESCOs. Specifically, I believe he recalculated the results assuming that these “ESCO-custom” projects showed the same rate of free ridership as the Company had budgeted, rather than the rate the Auditor had discovered. I believe it is within the power of the Company to require its ESCO partners to control free ridership — to line up customers for DSM subsidies who are not free riders — just as it can do so with its own staff. But we did not reach agreement on the point, and came up with different calculations as a result.
Q The Company says that the results of the Audit Report should not be used to revise actual free ridership rates for the year that’s under audit, but only used to revise budgets and plans for future years. How do you respond?
A First, I think it is perverse to use the SSM to reward miscalculations, rather than utility-induced gas savings and TRC benefits. Those free-rider savings are not attributable to the Company’s DSM programs, according to the independent auditor, and it is a misuse of ratepayer funds to pretend that they are, and to give the company a financial reward for having found so many of these free riders. Whether the mistake was accidental or on purpose, ratepayers should not be asked to reward it. Indeed, the SSM formula calls upon the company to “reward” the ratepayers for custom projects, because they fell short of their target.
Secondly, freezing these custom free rider rates at budget values significantly undermines the primary value of the audit and the audit process, and is inconsistent with the very purpose of the audit. Here are some quotes from the final revised Terms of Reference of the Audit, as distributed to candidates for the job by an Audit Subcommittee chaired by the Company:
“The ultimate goal of the independent audit is to make a determination of whether the Company’s claimed SSM amount is accurate and appropriate, and to give confidence to all stakeholders that the claim – as amended by the audit process if necessary – will be fair and justified.
. . .
“Possible Audit Activities:
“Review and verify the accuracy of all calculations leading up to the proposed SSM amount, and verify that the calculations are consistent with the OEB-approved prescribed methodology;
“Review Enbridge Consumers Gas’ procedures for tracking program participants and determine whether they lead to accurate counts, particularly for programs that do not provide customer rebates;
“Determine whether Enbridge Consumers Gas’ reported values for participation, costs, energy savings, free ridership rates and other key assumptions are accurate and adequately documented by program records, evaluation studies and other relevant data;
. . .”
I find it frankly impossible to read these terms of reference while trying to hold the Company’s stated view – that the auditor’s findings of inaccurate reported values for free ridership should not be used to amend the SSM claim, so that it is ” accurate and appropriate”. How can such an unamended claim, based on inaccurate values, possibly be “fair and justified”, and how can all stakeholders have confidence that it is??
Q Was it primarily the terms of reference that convinced you that the Audit results would generally change the SSM claim?
A No, also in our initial meetings with Dr. Goldenberg, the principal auditor for Xenergy, we continued to make audit suggestions that were consistent with our understanding – and the Terms of Reference’s understanding of – “the ultimate goal of the independent audit”. For example, we asked Dr. Goldenberg to concentrate on auditing and testing those parts of the DSM program that if changed would be most “sensitive” in affecting the SSM claim. As I recall, we all decided that attribution and related issues of free ridership in custom programs should be one priority.
Q Can you cast any light on the rest of the audit findings that led the independent intervenors to revise the Company’s SSM claim for 2000?
A The two other significant changes both involved Domestic Hot Water heating in residential markets. These involved significant changes to some of the biggest DSM programs — the biggest contributors to TRC and the biggest contributors to the SSM. Again, the specific numbers differed a bit between Chris Neme on the one hand and Malcolm Rowan and me on the other — not because of differences in “generosity” in these cases, but because of different positions on retroactivity in prescriptive programs.
Q What is “retroactivity in prescriptive programs”?
A Prescriptive programs are those programs that aren’t custom — like distributing or subsidizing low-flow showerheads, or getting service employees to lower the thermostats on customers’ hot water tanks, across the board. The measures are standard, and generally mass distributed. And retroactivity here means how we deal with revelations that one of our assumptions needs to be revised. Basically, there was no disagreement among the four of us that most of these factors are essentially out of the Company’s control, and therefore that they should not be penalized or rewarded for these changes. But there are two ways to approach such a change with that principle in mind: you can pretend that the revelation didn’t occur, “freezing” the assumption at budgeted levels. Or you can reflect the best number in the actuals, but pretend that you had the new number when you set the budget and the target, revising it, too. The former approach we called “no/no”, and the latter was the “yes/yes”. At least this year, the Company and Chris Neme generally prefer the “no/no” approach, and Malcolm Rowan and I prefer the “yes/yes”.
Q What do you mean, “at least this year”?
A Apparently both the Company and Chris Neme and his organization (GEC) have supported the “yes/yes” position in the past, but now prefer “no/no” for prescriptive programs.
Q Can you summarize the arguments for each?
A Yes. I believe this is certainly an area where reasonable people can disagree, because there are important competing principles. On the one hand, the “no/no” proponents argue that the purpose of the SSM is to give the Company incentive to do what we all — including the Board — agreed the Company should do, i.e., carry out the agreed DSM plan and achieve more than its targeted value of savings and benefits. If the world changes, or a new study shows that important assumptions were wrong, that should be changed for future years, but the Company should still be rewarded for over-achieving at its assigned task. The “yes/yes” proponents (including Energy Probe) like to remind people that SSM stands for Shared Savings Mechanism, and that it’s important that we “share” real savings and not just somebody’s misperception, or something that would have been a saving if only the world hadn’t changed. (In addition, “yes/yes” proponents often feel that the Company is in the best position to know what the right numbers are, and will have more incentive to do so under “yes/yes”.)
Q How does this differ, exactly, from the discussion we had earlier about the custom projects?
A In the custom projects, because the variables can be controlled by the Company, they must logically be part of the Company’s incentive scheme. Here, on the other hand, the variables are generally out of the Company’s control, so all four of us agreed that the Budget and the Actuals must use the same values for these variables. The disagreement here is only over whether those “same values” should be the new, best estimates or the old, now-obsolete values.
Q Now that we understand why the numbers differ a bit, can you go through these two examples?
A Yes. The larger of the two involves a stupid mistake. During the exercise, it was discovered that the calculation of savings from “DHW tank set point” – getting service people to turn down the thermostats on existing hot water heaters — stupidly assumed that the savings persisted for 15 years, the assumed lifetime of a brand-new hot-water tank! Since the tanks in people’s houses are, on average, half-way through their lifetimes, this mistake overstated the savings by a factor of two (really somewhat less because of discounting). Correcting that mistake — which even Chris Neme thought should be corrected on a “yes/yes” basis, because it was always stupid, and the Company should have fixed it — dropped $990,099 from the Company’s claim, for the independent members of the Subcommittee.
Q But not for the Company?
A No. We’re only discussing difference between the Company’s post-audit “revised” SSM claim and the recommendations of Energy Probe and the other independent members of the Audit Subcommittee. The Company apparently believes it is entitled to 35% of the TRC benefits that the over-budget hot-water tanks will not save in their 8th, 9th, 10th, 11th, 12th, 13th, 14th, and 15th years! The rest of us all disagree. And because the Auditor found that 12.6 years is a far better documented tank life expectancy than 15 years, Malcolm Rowan and I also reduced that 7.5 years to 6.3, on a “yes/yes” basis, so we came up with a slightly bigger reduction than Chris Neme did.
Q And the second one?
A The second one involves the distribution of low-flow showerheads. The Company changed the program — from one that gave low-flow showerheads to customers whose existing showerhead tested high-flow, to one that gave low-flow showerheads to virtually every customer visited. Since all or virtually all the showerheads for sale in Ontario now and for several years are low-flow, the market has begun to become saturated. The Audit Report gave a range of estimates for the saturation rate, with 32.5% as the middle estimate. Malcolm Rowan and I applied this change on a “yes/yes” basis, as if the Company had budgeted to distribute showerheads freely into a 32.5% saturated market. Chris Neme applied the change on a “no/yes” basis (the same as all three of us did for custom programs), on the basis that it was the Company’s change of program design that caused the change. This time Chris’s deduction was bigger than Malcolm’s and mine — at $1,142,805 vs. our $605,368.
Q How do you answer the Company’s clailm that revising the SSM as the result of the audit — e.g., by correcting the share of free riders in custom projects — creates unacceptable business risks?
A There’s no doubt that being audited is a threatening to experience, in principle, and so it should be. I think that’s the purpose of audits, when they’re done properly. If they find mistakes, there should be consequences. I would be happier in a world without risks — at least to me – and I’m sure the Company would be, too. But removing these risks from the Company means imposing risks on ratepayers — and as innocent as the Company may claim to be, the ratepayers are more so, and deserving of protection.
Q Aren’t you tempted to show some mercy to the Company? After all, they did accomplish a lot of DSM, and created very large TRC benefits by doing so.
A Yes, I think there is a case for mercy, and that is why all the independent members of the Audit Committee showed mercy — e.g., in directing the auditor to examine “generous alternative scenarios” for free riders in custom projects and endorsing various moderate compromise scenarios among those generous alternatives that restored millions of dollars to an SSM that otherwise would have run into an enormous penalty. I believe ratepayers have also been generous in allowing the Company to pass through an amount of DSM costs estimated at over $790,000 in 2000, in the costs attributable to savings that either did not happen or would have happened without those expenditures. (See I/9/24, table at “C”).
Also, I believe that hindsight has made it clear that allowing the company to claim a 1999 SSM of ********* based only on the Company’s Monitoring and Evaluation report and an audit by an auditor hired and directed by the Company, was excessively generous to the Company by approximately $3 million (See I/9/24, table giving answer to sub-question D).
Finally, I believe we should show mercy by redesigning the SSM so that it is less volatile and more transparent, for the good of all involved.
Q Do you have anything to add to the discussion of principles of retroactivity concerning Issue 9.6?
A Yes. I believe there are two points that must be added about the principles of retroactivity. First, on prescriptive programs, there is ample room for compromise between “yes/yes” and “no/no”, both of which are defensible and principled views. Indeed, the Audit Subcommittee discussed several possible compromises, including the following: (a) treat the assumption as “no/no” but cap the SSM reward / penalty in order to avoid distributing non-existent benefits; (b) use an SSM mechanism with upper and lower bounds, i.e., an “S” curve; (c) use a set of “rolling averages” for high-leverage variables, especially avoided costs of gas (discussed below), or (d) use a simple arithmetic compromise between the calculations made using the two approaches. (Ref. I/9/26/p.41 f.) While I still generally favour “yes/yes” for prescriptive programs, a well-designed compromise might be preferable to either “pure” case.
The second point is this: a large change in the avoided costs of gas presents an extreme “acid test” of any SSM design that depends on TRC — and of either of the two “pure” positions outlined above. For example, applying the “yes/yes” approach when avoided costs fluctuate widely during the year introduces the real risk of unwarranted “windfall” rewards and penalties through the SSM — especially with the current high-leverage “35% of delta-TRC” design of SSM. On the other hand, applying the “no/no” approach in a year when avoided costs fell sharply and DSM performance was above target runs a real risk of distributing huge “savings” to the Company through SSM, when ratepayers actually did not realize any TRC savings at all!
The problem arises because the avoided costs of gas is a variable that is both absolutely beyond the control of the Company’s DSM activities and hugely “high-leverage” in its impact on the calculation of TRC benefits and therefore of the SSM. Unlike changes in other variables which are usually confined to a single program or even a single measure, the effect of a change to avoided gas costs is across the board, increasing or decreasing the value of every unit of gas saved.
This issue obviously has serious implications for Issue 9.2, “Review of the Shared Savings Mechanism (SSM) incentive scheme”, but it is also crucial for the years 2001 and 2002, already completed and (at least 2001) the subject of Issue 9.6. If the SSM mechanism were to continue unchanged for many years, usig our preferred “yes/yes” approach to prescriptive programs, years of “windfall” increases in avoided costs might well be balanced out by years of painful drops, in which the SSM formula might generate millions of dollars of penalties. Of course, that result might be unacceptable in its volatility, even if the average over many years was acceptable.
But in our situation, where it appears that the SSM mechanism may well be significantly changed after roughly 4 years, the volatility from a sharp rise in avoided costs may not have a chance to average out. Moreover, it is worth noting that changing avoided gas costs did not significantly affect the 2000 TRC or SSM calculation, but did significantly affect the 2001 calculation. As a result, the 2000 Audit Subcommittee was not compelled to resolve the issue, and avoided costs were not a factor that led to discrepancies between the supporters of the “yes/yes” position (CME and EP) and the supporters of the “no/no” position (GEC).
In addition to the principles and the possible compromises outlined above, Energy Probe believes there may well be merit in treating avoided cost as a unique variable, to be “frozen” (“no/no”) while other variables are not (“yes/yes” for prescriptive programs, and “no/yes” for custom). Indeed, we joined a large number of parties in advancing several possible settlements that reflected that compromise during the ADR settlement conference (although we certainly do not endorse the December 23d non-consensus proposal).
Q Can you summarize the principles you urge the Board to endorse, at least for the years 2000-2002?
- In general, the audit results must be incorporated into the audited year’s SSM and LRAM calculations. Otherwise, the Board will be giving the Company a durable incentive to make errors in its Evaluation Report.
- For “custom” programs, all input values except avoided gas costs should be changed in the actuals and not in the budget (“no/yes”).
- For “prescriptive” programs, we believe that all input values except avoided gas costs should be changed in both the actuals and the budget (“yes/yes”), although we are receptive to compromises between “yes/yes” and “no/no”.
- For avoided costs, we believe that “no/no” is probably the best approach, again, without ruling out a form of compromise like rolling averages or “smoothing”.
Q As long as you’re giving principles of SSM, how would you change them going forward — that is, what are your major recommendations concerning Issue 9.2, “Review of the Shared Savings Mechanism (SSM) incentive scheme”?
- It is important that the SSM, past and future, give the Company an incentive to create net societal benefits through DSM, and not an incentive to profit from mis-reporting benefits that are either fictional or not reasonably attributable to the Company’s DSM activities. <!–. In the case of the 2000 evaluation and audit, from “custom” projects that are largely within its own control, that are subsequently found to be either fictional or not reasonably attributable to the Company’s DSM activities. –>We believe that an independent audit, whose findings immediately affect the SSM, must be part of such a structure.
- We are sympathetic to several of the Company’s stated principles in the design of its proposed SSM, but we nonetheless urge the Board to reject that design.
- We are especially supportive of the Company’s desire to diminish the importance of the annual target-setting, but we would eliminate it entirely.
- We are also supportive of the Company’s intention to substitute volumetric savings for part of the new mechanism (Ref. A7/2/2/pp.8-9) — which would be immediately and directly meaningful to the public and the Company’s staff as well — for the relatively obscure and indirect TRC. We also note that basing SSM on saved gas volumes is similar in effect (though not identical) to freezing avoided costs with “no/no” — an approach we have endorsed.
- Nonetheless, we urge the Board to reject that design and to direct the DSM Consultative Group to devise an SSM that eliminates the annual negotiated target-setting exercise entirely, in favour of a more transparent and “self-regulating” mechanism, along the following lines: the Company should receive as an SSM a fixed share of allTRC benefits generated, and should cover its DSM costs out of that SSM. The percentage should be chosen to reflect an appropriate level of reward for expected performance, and may be reviewed from time to time before the Board.
- The Consultative Group should attempt to devise rules for the treatment of avoided costs as a special variable, and entertain the notion of incorporating a volumetric component in the formula.
Q What’s so bad about having the SSM calculated as 35% of the difference between the target and the actual TRC?
A I believe that that present “high-leverage” formula has a number of unfortunate effects: it places incredible value on the target-setting exercise, which I believe is an ill-informed and poorly designed exercise; it creates incredible complexity in the evaluation and audit process; and it clearly and openly contemplates large negative SSM settlements (penalties), when it is not clear that the Board, the Company, or many of the intervenors actually have the stomach to see a large negative SSM.
Q Why do you call the target-setting exercise “an ill-informed and poorly designed exercise”?
A In my experience, these sessions involve two sides: one (the Company) claims that the markets are saturated, that last year’s savings were special and cannot be duplicated. The other side (led by the most aggressively pro-DSM environmental groups) heaps praise on the Company for its ingenuity, points out the Company’s history of exceeding its targets, and suggests ways the company could expand its efforts or its scope. In addition to being unseemly and unpleasant events, to my taste, they also bring together two parties with very different levels of access to the facts of the situation — not to mention very different levels of incentive to “win” the competition. I believe that it is worthwhile to “flatten” the incentive curve to a significantly lower percentage of TRC in return for eliminating this exercise, making the exercise much more transparent, and giving the Company an incentive to limit DSM costs where possible, rather than a “cost-plus” incentive to augment them or at least ensure that they are all spent.
Q What is your concern about the lack of transparency of the current mechanism?
A During the discussions of the Audit Subcommittee, we often had the services of Kai Millyard, the designer of an Excel spreadsheet that duplicates the Company’s DSSStrategist software, to calculate TRCs and SSM. Several times, we got into conversations where Kai would kindly point out that the effect of some audit-driven change we were discussing would actually be in the opposite direction than the one we’d all assumed. For example, once or twice when a program became controversial, the Company representative offered to just remove it from the budget and the actuals. That sounded generous on the Company’s part, until Kai pointed out that the program had fallen a bit short of its target, and its withdrawal would actually increase the SSM! Similarly, our discussion about “yes/yes” and “no/no” principles in prescriptive programs was constantly complicated by not knowing which was the “generous” approach and which the “punitive”, because the answers kept changing. I believe that a linear mechanism, that rewards based on every saving, but must bear its own costs, would avoid most or all of these problems.
Q What are your major recommendations concerning Issue 9.5, “Review of the DSM Audit Process”?
A My major recommendations concerning Issue 9.5 are as follows:
- The Board should ensure that the substantial benefits to ratepayers from the independent 2000 DSM audit exercise are carried forward to future years, by ensuring that future audits are independently supervised and high-value like the 2000 audit, and not Company-controlled and of limited value like the 1999 audit.
- The Board should insist on a timely resolution of each year’s DSM evaluation and audit activities, not only to avoid unnecessary retroactivity in rates, but so that important lessons learned in the evaluation and audit process may be incorporated into the DSM process of future years as quickly as is reasonably possible.
Q What was so good about the 2000 audit?
A The 2000 DSM audit — the first supervised by an independent Audit Subcommittee — represented a quantum leap in useful findings and in value-added for ratepayers. Indeed, the difficulty of the discussions and negotiations around “retroactivity” or “the retrospective treatment of changed assumptions” is a measure of how many changes — corrected errors, in my view — the 2000 audit made.
Q Wasn’t the 1999 audit good, too?
A No. Most of us in the DSM Consultative Group were pretty disappointed with it at the time. It seemed as if almost everything that was discovered that year was discovered by Kai Millyard’s replication of the calculation, and not by the auditors. But as disappointed as we were with the audit at the time, that disappointment has mushroomed enormously now that the 2000 audit – as well as the independent Subcommittee discussions – has discovered some serious problems with the Company’s Monitoring and Evaluation work. And unfortunately, there are no solid reasons to believe that these problems weren’t there in 1999 as well. (See I/9/24, answer to sub-question F)
Q What do you think we should learn from this comparison?
A The disappointing audit of the Company’s 1999 DSM activities was done by an Auditor selected and hired and supervised by the Company, with only advisory involvement by other parties, primarily through the DSM Consultative Group. In contrast, the audit of the Company’s 2000 DSM activities was done by an Auditor selected and hired and supervised by an independent Audit Subcommittee composed of one representative of the Company and three independent members selected by the Demand-Side Management Consultative Group. Therefore, the Company’s current proposal to reject the audit-governance structure that prevailed for the impressive 2000 audit in favour of those that prevailed for the far less valuable 1999 audit, should be rejected as not in the best interest of ratepayers.
Q But the Company is concerned about the time and money spent on independent audits, and the loss of control by the Company. How do you respond?
A Although the audit subcommittee process was time-consuming, it is incorrect for the Company to suggest that the time was poorly spent — or to ignore the fact that the lion’s share of the delay was because of the Company’s acts, or omissions. For example, the Company introduced significant delays of its own during the process, several of which involved the failure to meet agreed-upon commitments and timelines. Perhaps most notable of these was the series of delays before a version of the Company’s 2000 Monitoring and Evaluation Report was available, on December 27, 2001.
Q Was that the Company’s main contribution to the delay?
A Probably, but another very significant source of delay was that the Monitoring and Evaluation Report, almost 15 months after the end of the fiscal year, still hadn’t actually monitored or evaluated the custom-project customers. Specifically, the Company had not actually contacted or sampled them to determine (a) if they had installed the measures that Company staff had reported, or (b) if they were actually influenced to do so by the Company’s DSM activities. Of course, if the audit’s initial samples had found that the Company’s claims were essentially valid, we could have proceeded more quickly. But they showed the opposite, so we had no choice (in my view) but to increase the scope of the Audit so the auditor could go back to the customers for a statistically significant Evaluation. Of course, all of this is time-consuming, as the timetable and the minutes show (Ref. I/9/26). But it is evidence of the independent audit’s value in filling embarrassing data gaps, not evidence against its value.
Q Wouldn’t the Audit Subcommittee have contributed more value to the process if it could have reached consensus on the issues that affect the SSM claim?
A Yes, of course. But we had a very large job to do, and we had to stop at the end of July (by ADR agreement) whether or not we had finished the job. Nonetheless, the three independent members of the Subcommittee — despite some important philosophical divisions — reached substantial consensus on the vast majority of important issues, and nearly reached agreement on the “bottom line” of the SSM. This substantial consensus – which to us seems consistent with the direction of the discussions during the meetings of the Subcommittee – stands in stark contrast to the Company’s final position, which to us seems extreme, unsupported by the Subcommittee’s discussions, and without merit.
Indeed, given the accommodation and rational discourse among the independent members in the last few meetings of the Subcommittee, it is my opinion that the independent members would likely have reached agreement within two or three additional meetings, except for an ADR-agreed deadline of July 31 for a final report. That agreement would, I believe, have been within the range of the final positions of the three independent members, and extremely remote from the position of the Company.
Q Weren’t the meetings difficult and time-consuming?
A Yes, they were. In addition to the complexity of some of the principles, and the enormous quantity of data involved, and the need to supervise a team of auditors, and the significant range of beliefs represented at the table, we had sharply conflicting personal styles, which occasionally caused difficulties and delays. Personally, I would have glossed over many details and principles and focused much sooner and much more on those decisions and principles I considered most important. My sense is that Chris Neme would have joined me — even though we actually disagreed on many of those decisions and principles — while Malcolm Rowan wanted to make sure that everything was done formally and properly and no steps were skipped. But even if we had the time to reach a consensus, I’m very sure that that consensus would have been among the three independent members of the Subcommittee, not including the Company. And the “bottom-line” SSM would also have been along the lines of our various final positions, as reflected in Kai Millyard’s Reconciliation Report — and nowhere near the Company’s final position.
Q Can you sum up the division between the Company and the independent members of the Subcommittee in a nutshell?
A Sure. In my view, we all agreed that the Company shouldn’t be held responsible for things that aren’t within its control. But I believe we couldn’t get the Company to agree that it should be held responsible for things that are within its control.