Mohsin S. Khan

This selection was excerpted from an article published in IMF Staff Papers in June 1990. Mohsin Khan is the director of the IMF Institute, a department of the IMF that provides training in macroeconomic analysis and policy for officials of member countries.

A question that is frequently raised in connection with fund-supported programs is whether such programs have been effective in achieving their macroeconomic objectives. Some writers have argued that, at best, fund programs do little in the way of improving the economic picture, while others have gone as far as to say that programs worsen the situation by inducing stagflation. Providing a clear-cut answer to this question turns out to be no easy task. There is at present little agreement in the profession either about how to estimate the macroeconomic effects of programs or about what impact past programs have actually had on the principal macroeconomic variables the fund typically is concerned with. Although there have been a number of studies on the subject over the past decade, one cannot say with certainty whether programs have "worked" or not.

Before turning to a discussion of the alternative methodologies used in the evaluation of programs, it is important to define how the effects of programs ought to be measured. At the most general level, it has been argued that macroeconomic performance under a program should be compared to the "counterfactual"--defined as the macroeconomic performance that would have taken place in the absence of a program (see Guiti n 1981). The counterfactual is perhaps the most appealing yardstick against which to assess program performance and the standard most widely employed in economics to measure the impact of government policy interventions. What would have happened in the absence of a fund-supported program is by no means the only standard against which to judge the outcomes of programs, but in many cases it is the most appropriate one. However, the counterfactual cannot, by definition, be observed and must be estimated or approximated. The various approaches used in evaluation studies should thus be judged in terms of how good they are in providing estimates of the counterfactual.

The literature suggests that four distinct approaches have been applied to the evaluation of fund-supported adjustment programs: (1) the before-after approach, which compares macroeconomic performance during a program and performance prior to the program; (2) the with-without approach, which compares macroeconomic performance in countries with programs and performance in a "control group" of countries without programs; (3) the actual-versus-target approach, which compares actual macroeconomic performance under the program and the performance specified in its targets; and (4) the comparison-of-simulations approach, which compares the simulated performance of fund program-type policies and simulated performance with some other set of policies.

In the discussion that follows, the studies are grouped according to the approach that was employed by them.

BEFORE-AFTER APPROACH

In the literature on the effects of fund-supported programs, the before-after approach has been the most popular. The first study to use this approach was by Reichmann and Stillson (1978). These authors examined a total of seventy-nine fund-supported programs implemented during 1963-72 and compared the behavior of the balance of payments, inflation, and growth during the two-year periods before and after the implementation of the program. Using nonparametric statistical tests, they found that there was a significant improvement in the balance of payments in only about one-fourth of all programs. In a majority of cases (over 70 percent) there was no significant change in the balance of payments. Of the twenty-nine programs involving countries with high inflation during the program period, the rate of inflation fell in six of eleven programs for which there was a notable deceleration in the rate of domestic credit expansion; in the nine programs involving a devaluation, there were five in which inflation was higher. Finally, growth performance was examined for seventy programs, and it was concluded that, on balance, fund-supported programs did not exert adverse effects on growth rates. In 40 percent of programs the growth rate declined after the inception of the program relative to the previous two years' average rate of growth, but at the same time growth was higher in 47 percent of the programs.

A similar procedure was followed by Connors (1979), who examined a total of thirty-one programs in twenty-three countries that were adopted during the 1973-77 period. He compared periods of one year before and after the programs. Also using a nonparametric rank test, Connors concluded that fund programs had no discernible effects on the ultimate targets--growth, inflation, and the current account deficit--or on important intermediate targets, such as the ratio of the fiscal deficit to gross domestic product (GDP).

The relationship between fiscal variables and fund program performance was examined in more detail by Kelly (1982). The methodology was primarily of the before-after variety, and in order to take into account possible lags in adjustment, comparisons were made over both one-year and three-year periods. In a sample of seventy-seven programs (covering thirty-three countries) during 1971-80, Kelly observed that when a one-year comparison was used, the fiscal deficit was reduced in 56 percent of the cases and that the current account and fiscal deficits moved in the same direction in 62 percent of the programs examined. This last result was also supported by regression analysis that showed a positive and statistically significant relationship between changes in the ratio of the fiscal deficit to gross national product (GNP) and changes in the ratio of the current account to GNP. Furthermore, in about half the cases there was a decline in the average growth rate over a three-year period and an increase or no change in the other half.

A study by Killick (1984) also attempted to capture the effects of lags by comparing the behavior of the balance of payments, the current account, growth, and inflation a year before the program with the behavior both one and two years after the program. Killick employed nonparametric statistical tests to gauge the effects of thirty-eight programs covering twenty-four countries during the period 1974-79. In contrast to other studies, Killick found that the balance of payments and the current account deteriorated irrespective of the time period over which the comparison was made. However, the difference between the preprogram and postprogram values of these variables was not statistically significant. Inflation was reduced, but the effects on growth were ambiguous. In the first year after the program the rate of growth was higher, but by the second year the positive effect had eroded and the net effect was zero.

Zulu and Nsouli (1985) also constructed before-after measures of program effects in their study of thirty-five programs implemented in 1980-81 for twenty-two African countries. They found that growth was lower or the same in the year after the program in about 60 percent of the cases. For the current account and inflation targets, the split was even--with as many programs showing an improvement in the current account and inflation performance as those showing a worsening or no change.

The most recent study using the before-after approach was that by Pastor (1987) for eighteen Latin American countries during 1965-81. Using one-year comparisons and on the basis of alternative statistical tests, Pastor concluded that fund programs led to a significant improvement in the balance of payments but that apparently there was no effect on the current account, inflation, or the rate of growth of nominal GDP.

WITH-WITHOUT APPROACH

The with-without approach is designed to overcome the inability of the before-after approach to distinguish between program and nonprogram determinants of macroeconomic outcomes. The basic reasoning behind this approach is as follows. Assume countries with and without programs are subject to the same nonprogram determinants--that is, they face the same external environment. Then, so the argument goes, by comparing before-after changes in outcomes in program countries to those in the control group of nonprogram countries, the effects of nonprogram determinants will cancel out--leaving the difference in group performance to reflect only the effects of fund-supported programs. Put in terms of the counterfactual, the idea is to use the observed performance of nonprogram countries as an estimate of what the performance of program countries would have been in the absence of a fund-supported program.

The with-without group approach was first used in two studies by Donovan (1981, 1982), which analyzed a sample of programs implemented from 1970 to 1980. The control group of countries without programs was taken to consist of all nonoil developing countries, and the comparisons were carried out over one-year and three-year time horizons. In the first study (Donovan 1981), which covered a sample of twelve programs (for twelve countries) implemented during 1970-76, the improvement in the rate of growth of exports was consistently higher for program countries than for all nonoil developing countries. The increase in the rate of inflation for program countries was about half that of the control group during the first year, and although it rose when three-year comparisons were undertaken, it was nevertheless well below the average increase in the rate of inflation of all nonoil developing countries. The outcome for growth was not as clear-cut. In the one-year comparison there was a sharp improvement in growth in program countries relative to the control group, but in the three-year comparisons growth in countries fell by more than it did in nonprogram countries.

In Donovan (1982) the sample of programs was expanded to seventy-eight, covering the period 1971-80, and the same analysis as in the first study was undertaken. The balance of payments and the current account positions of program countries were found to improve relative to the control group in both the one-year and three-year comparisons. The increase in inflation in program countries was about half that of nonoil developing countries in the one-year comparisons and fell to one-third in the three-year comparisons. However, in contrast to the results of the first study, the rate of growth of real GDP fell by more than the average decline experienced by nonoil developing countries in the one-year comparisons but by less in the three-year comparisons.

Loxley (1984) applied the same types of tests as Donovan (1982) to a group of thirty-eight least-developed economies (defined as countries with per capita incomes of $690 or less in 1980) that had programs with the fund during 1971-82. His results, however, were less definitive than those obtained by Donovan, in that the least-developed countries with programs did no better, on average, in terms of current account, balance of payments, and growth performance relative to other least-developed countries without programs, the program countries examined by Donovan, and all nonoil developing countries. The improvement was statistically significant only in the case of inflation and then only in the three-year comparisons.

Gylfason (1987) also used a version of the with-without approach in his study of thirty-two programs implemented during 1977-79. The reference group included the developing countries that had experienced balance of payments difficulties during 1975-77, and nonparametric statistical tests were used to determine if the behavior of the macroeconomic variables for program countries over a three-year period was significantly different from that of the reference group. First, there was an improvement in the balance of payments in program countries relative to the outcomes in the control group. Second, inflation in program countries did not fall but rather remained approximately the same as the average rate of inflation in the group. Third, the growth rate was not significantly affected by the program.

This pattern of results was basically replicated by Pastor (1987) for Latin American countries. Employing the same statistical methodology as Gylfason, Pastor also found that the balance of payments performance of program countries was significantly better than that of nonprogram countries and that the differences in inflation and growth performance were not statistically significant.

While the with-without approach copes with some of the problems of the before-after approach, it is by no means ideal. The problem is that program countries differ systematically from nonprogram countries prior to the program period, and this difference matters for performance evaluation. In short, program countries are not randomly selected. Instead, they are adversely selected in the sense of having relatively poor economic performance prior to the program period. This is not surprising since, after all, a basic requirement for fund financial support is that the country have a balance of payments need. This requirement alone suggests that program countries would be expected to have weaker than average external positions when the program was implemented. In any case, nonrandom selection of program countries means that simple with-without estimates of program effects may be biased. Intuitively, the bias occurs because, under nonrandom selection, the with-without estimator attributes differences in outcomes exclusively to program status, when in fact the difference in starting positions itself is a cause of differences in subsequent performance between the two groups. Furthermore, the direction of the bias can go either way. If past economic difficulties signal less-serious current difficulties, even in the absence of a program, then the with-without approach will overstate the beneficial effect of a fund-supported program. Conversely, if past difficulties signal even more serious present difficulties, then the effect of programs will be understated. There are ways, however, as discussed by Goldstein and Montiel (1986), to modify the with-without approach to reduce some of the biases.

Goldstein and Montiel (1986) applied the generalized evaluation estimator to a sample of sixty-eight programs for fifty-eight developing countries implemented during 1974-81. These authors found that program countries systematically demonstrated weaker performance--that is, higher inflation, slower growth, and larger current account and overall balance of payments deficits--than nonprogram countries in the preprogram period. Adjusting for these preprogram differences in performances and taking into account the effects of policy instruments on targets, Goldstein and Montiel used regression analysis to estimate the program effects. Two interesting sets of results emerged from this study. First, there were no statistically significant effects of programs on the current account and balance of payments, on the rate of inflation, or on the growth of real output. Second, the estimated program effects under the generalized evaluation estimation were quite different from those obtained with the standard with-without estimator. The latter indicated an improvement in the current account, a slight worsening of the balance of payments, a reduction in inflation, and a rise in the growth rate associated with programs. When the generalized evaluation estimator is used, the improvement in the current account ratio disappears, the deterioration in the balance of payments ratio is magnified, and the favorable outcomes for inflation and growth are reversed.

ACTUAL-VERSUS-TARGETS APPROACH

Another strand in the literature on program effects compares actual outcomes for certain key macroeconomic variables to the targets for such variables specified by the authorities and the fund at the inception of the program. This approach has not been as frequently used as the other two approaches. Reichmann (1978), for example, studied twenty-one programs for eighteen countries that were in effect during 1973-75 and compared the outcomes to targets for the balance of payments, inflation, and growth. He found that the balance of payments targets were met or exceeded in nearly two-thirds of the programs. The targets for inflation were, however, exceeded in over half the programs. Greater success was achieved in the case of the rate of growth, with 62 percent of programs meeting the targets.

In a similar vein, Beveridge and Kelly (1980) surveyed the fiscal content of all 105 programs supported by the fund that were approved during 1969-78. Their focus was on intermediate targets--domestic credit expansion, government revenues and expenditures, and deficit financing--rather than on the final macroeconomic objectives. Nevertheless, the results for the intermediate targets are informative since achieving these is generally a necessary condition for meeting the ultimate targets for the balance of payments, inflation, and growth. Beveridge and Kelly showed that almost all programs contained government revenue and expenditure forecasts and that both actual expenditures and revenues tended to differ from these forecasts. A shortfall in revenues occurred in about 40 percent of the cases, while expenditures were above projected values in nearly 60 percent of the programs. Consequently, the overall fiscal deficit targets were achieved in only about half the programs, as were the domestic credit ceilings. Finally, governments were more successful in meeting domestic nonbank financing limits than foreign financing targets. The former were satisfied in almost 70 percent of the programs, but foreign financing of the fiscal deficit exceeded the target in over 60 percent of the cases.

Zulu and Nsouli (1985) also analyzed actual outcomes and targets in their study of programs in African countries approved in 1980-81. They found that the current account targets were met in 38 percent of programs, and the inflation targets in about 48 percent of the cases, but growth targets were only achieved in less than 20 percent of the programs.

COMPARISON-OF-SIMULATIONS APPROACH

Unlike the other three approaches, the comparison-of-simulations approach does not infer program effects from actual outcomes in program countries. Instead, it relies on simulations of economic models to infer the hypothetical performance of fund-type policies or policy packages and alternative policy packages. If the aim of the exercise is to evaluate the results of a specific fund-supported adjustment program, then the use of actual program outcomes is indispensable. However, if the purpose is to evaluate the design and effectiveness of fund-supported programs in general, then examining the likely effects of alternative policy packages can be quite useful and revealing.

Khan and Knight (1981), for example, constructed a small dynamic econometric model and estimated its parameters on a pooled cross-sectional, time-series sample of twenty-nine developing countries, most of which had programs with the fund. They then investigated the hypothetical effects of a stabilization program that pursued an external balance target using policies that figure prominently in fund-supported programs--namely, domestic credit restraint and reductions in government expenditures. The simulation experiments showed that such a program produced a sharp price deflation in the first year, followed by a temporary burst of inflation as prices rose back to their equilibrium level. Output, however, contracted sharply in the first year, then rose temporarily above its full-employment level, approaching that equilibrium level gradually over a period of several years.

In a further study, Khan and Knight (1985) extended their simulation analysis to a comparison of alternative policy packages. Specifically, they compared the results for the balance of payments, inflation, and real output growth of a package of demand-management policies (that is, a onetime reduction in the rates of growth of nominal domestic credit and nominal government expenditures, plus a devaluation) with a combined package of demand-management and structural policies (that is, the above-mentioned demand-management policies, plus a set of structural policies that would gradually raise the rate of growth of capacity output). The demand-management package improved the balance of payments almost immediately but at the cost of a temporarily higher rate of inflation and a short-run reduction in growth. The simulations of the combined package showed that structural policies could help to partially offset any short-term adverse effects on growth that might result from demand restraint as well as the inflationary consequences of devaluation. Furthermore, the longer-run effects of fund-type policies on the balance of payments, inflation, and growth were more favorable than the short-run effects.

A summary of the results obtained by the various studies that have evaluated the effects of fund-supported adjustment programs on the principal macroeconomic objectives is contained in table 3. Overall, these studies yield three conclusions. First, there is frequently an improvement in the balance of payments and the current account, although a number of studies show no effects of programs. Second, inflation is generally not affected by programs. Finally, the effects on the growth rate are uncertain, with the studies showing an improvement or no change being balanced by those indicating a deterioration in the first year of a program.

Table 3
Summary of Studies on the Effect of IMF Programs
	1981					effects onb
Study	Time Period	Number of Programs	Number of Countries	Methoda	Balance of Payments	Current Account	Inflation	Growth
Reichmann and Stillson (1978)	1963-72	79	...	Before-after (2-year)	0	...	0	+
Reichmann (1978)	1973-75	21	18	Actual-versus-target	+	...	+	+
Connors (1979)	1973-77	31	23	Before-after	0	0	0	0
Donovan (1981)	1970-76	12	12	With-without	...	...	-	+
Donovan (1982)	1971-80	78	44	With-without	+	+	-	-
Killick (1984)	1974-79	38	24	Before-after	0	0	-	0
Zulu and Nsouli (1985)	1980-81	35	22	Actual-versus-target	...	0	-	-
				Before-after	...	+	-	0
Goldstein and Montiel (1986)	1974-81	68	58	Before-after	-	-	-	-
				With-without	-	+	-	+
				Generalized evaluation	-	-	+	-
Gylfason (1987)	1977-79	32	14	With-without	+	...	0	0
Pastor (1987)	1965-81	...	18	Before-after	+	0	0	0
Khan and Knight (1981)	1968-75	...	29	Comparison of simulations	+	+	-	-
Khan and Knight (1985)	1968-75	...	29	Comparison of simulations	+	+	-	-
Loxley (1984)	1971-82	38	38	With-without	0	0	-	-
aComparison over one-year periods, unless otherwise noted. bDirection of change; (+) indicates positive effect, (-) indicates negative effect, 0 indicates no effect.