Measuring the effectiveness of health care systems as a result of health policy and regulations

As the world's economies faced the COVID-19 pandemic and difficulties in providing access to medical services, the issue of health system efficiency became particularly important. The current situation faced by countries around the world, although triggered by COVID-19, is the result of the design of health systems over the long term. The characteristics of a country’s national health care system can affect its levels of economic growth, labour resources, and labour productivity. Healthier workers are both physically and mentally more willing to work, and this willingness is reflected in their higher productivity and correspondingly higher wages^[1]. At a time when many societies are growing older and becoming more urban, the functioning of the health care system can have a significant impact on a population, including on its general health. Many factors related to the provision of medical services are playing an increasingly large role in the distribution of morbidity and mortality, particularly the types of services covered by the health care system, the quantity and quality of the medical services provided, the approach to redistribution, and the availability of these services^[2].

The method of health care financing is extremely important, as it determines the quantity, distribution, and quality of the medical services provided. It also has a significant impact on the operational efficiency of the health care system, as the ability of the system to provide medical services depends on the demand, and, to a lesser extent, on the financial capacity of the entities involved. Therefore, governments play an important role in the health of the population through the decisions they make about the amounts and the methods of health care financing. In line with Evans et al.^[3], we assume in this paper that the healthy life expectancy (HALE) indicator can be used to aggregate information on the method of health care financing, health care institutions and management methods, and the general availability of medical services. As this indicator is in turn related to the data on the amount of health expenditure in each of the surveyed countries, it can be used to estimate the frontier function of health production.

The aim of this article is to calculate the effectiveness indexes of health care systems across the world; to determine how the scope and the structure of health care financing, as well as the organisation of the health care system, affect the health resources of a given population; and to show how these values varied in the analysed period. The study examines the effectiveness of national health care systems, initially measured based on the World Health Organization convention as the health-adjusted life expectancy (HALE); i.e., the average number of years of healthy life that a new-born individual can expect have. This rate is estimated by adding estimated birth rates and death rates from age-specific conditions to each population age cohort, and life expectancy is adjusted by years spent in poor health. Medical care is considered based on the amount of public expenditure on health care per capita (maintaining PPP). The results of the research for 191 countries carried out by WHO forms the basis for our analyses of the effectiveness of health care systems.

The study also includes a detailed verification of the applied research methods, and an attempt to select the most accurate method. The frontier function of health is determined using non-parametric and parametric econometric methods. How this function developed between 2000 and 2016 is then analysed.

The nature of the subject matter this paper covers determined the kinds of research methods we have chosen to use. Our decision to implement the cognitive goal formulated above was based on the previous literature and quantitative analyses in which both parametric (FHD, DEA) and non-parametric methods of approximation of the frontier production function (COLS, stochastic approach) were used.

The study uses statistical data from the databases of the World Health Organization and the OECD.

The largest databases on health care systems are built and maintained by the World Health Organization (WHO) and the Organization for Economic Cooperation and Development (OECD). These databases contain a great deal of information, including indicators and quantities that describe both the health resources and the health expenditures of different populations, as well as the institutional characteristics and medical statistics of various health care systems. Almost all countries in the world collect information related to the life expectancy of their populations, including on their demographic characteristics and social profiles. This information is used by both research centres and state institutions, particularly by those responsible for making the most important health care-related decisions in a given country.

In the following, we present selected indicators used in research that provide information on the life expectancy and the health-based quality of life of populations, and on the factors that devalue the latter. We also present the statistical data used in our analyses.

In 2001, WHO formally replaced the DALE index with life expectancy in good health^[4]. The healthy life expectancy index expresses the average number of years that an individual from the study population will survive in various health states. In other words, HALE includes more than just the state of full health. Thus, this indicator varies strongly both over time and between countries.

The methodology for calculating the healthy life expectancy index is not simple. Using data reflecting the global burden disease, the intensity of the incidence of various diseases is estimated, while taking age and gender into account. Estimates of the severity of incidence are then obtained. Ultimately, the incidence frequency is calculated using the World Health Survey and the WHO Multi-Country Survey Study, and the HALE value for the analysed country is obtained based on the survival tables^[5]. The HALE indicator attempts to capture the effectiveness of the health care system in reducing the burden of disease in the population. It is possible that the disease duration or the suffering associated with the disease is reduced, but mortality is not affected. HALE can be used to capture such progress, even when the mortality rate has remained unchanged.

This paper attempts to verify the research methods used by Evans et al. in their paper entitled The Comparative Efficiency of National Health Systems in Producing Health: an Analysis of 191 Countries. The study was based on the HALE data published by WHO for 2000 and 2016, and on general health expenditure per person according to the purchasing power parity for a given year. The original DALE data from 1997 on which the authors based that paper are not available. Moreover, the nomenclature of this indicator was changed in 2001. Hence, the DALE indicator can be seen in the figures presented in the analytical chapter, while the same value is called HALE in current international statistics. To ensure consistency with the above-mentioned study, we use health expenditure data for 1997. In brief, however, when specifying the date in the charts, the following year is used (i.e., for HALE).

Even among countries with similar income and education levels, there are significant differences in the health resources of the population. These differences are partly due to the varying levels of effectiveness of national health care systems.

Murray and Frenk^[6] attempted to define the effectiveness of health care systems using an approach that enables both international comparisons of effectiveness and the observation of changes in this effectiveness over time. The authors measured the efficiency of a health care system by comparing the observed health resources with the services that a given system ought to provide. In their 2001 article, Evans et al.^[7] pointed out that in the absence of expenditures on the health care system, or of any health care system at all, the level of health will not be zero, as the entire community will not immediately die as a result.

It should be noted that the efficiency of the health care system, as examined in this paper, is not based simply on the system’s technical efficiency. In our study, health care expenditures are equated with outlays, while the results are based on the population’s health resources. Thus, efficiency as such reflects more than just whether a system’s solutions or reforms are implemented at the lowest possible cost. Rather, it is also based on whether the health care system selects the most cost-effective intervention programs for a given level of expenditure.

Assuming that the maximum achievable production volume (or frontier production function) is not observable, there are several options for approximating it. The first approach involves defining an acceptable set of interventions that can be implemented in the system, identifying their costs, and checking whether they are effective; and then selecting a combination that maximises the objective function with the resources available. However, such a study would require a very large amount of data, and these data are not available for many countries.

Another way to approximate the frontier production function is to estimate it based on econometric methods using input and output data at the country level.

Examples of a deterministic approximation of the frontier production function are based on nonparametric methods, which primarily include:

• data envelopment analysis (DEA), and
• free disposal hull (FDH).
  
  

Due to the significant shortcomings of non-parametric methods, we decided to carry out the analysis using parametric methods, which are based on the production function, and enable an analysis of the relationship between outlays and effects. The main challenge that arises in the econometric analysis of effectiveness is estimating the parameter values ​​of a given frontier production function and the indicators reflecting the level of effectiveness. To model the frontier production function, both classical econometric solutions and linear and quadratic programming are used, which in turn have significant drawbacks, such as a high sensitivity to unusual deviations, and the failure to include the distribution of the ineffectiveness-reflecting variable^[8].

The parameters of the production function are estimated using standard econometric estimation tools, and the production function itself sets the efficiency curve. The deviations from the efficiency curve are caused by ineffectiveness and random errors. As part of the parametric approach, many methods have been developed that differ mainly in the assumptions they use to describe the distribution of random error and ineffectiveness^[9].

Results


I. The determination of the frontier function of health production for 2000, including the verification of methods

The frontier set for the HALE data from 2000 based on the Free Disposal Hull method takes the form of a step function. The horizontal axis shows the value of the logarithm of health expenditure per capita from 1997, while the vertical axis shows the logarithm of HALE 2010. The cloud of observations in this study is similar to that in the initial study. The largest clusters can be observed within the same points. The production limit curve is stretched over the outermost observations, while all the points below show ineffectiveness as represented by the distance of a given point from the frontier. The differences in the size of outliers may be related to the database update by the WHO.

Source: author’s own calculations based on WHO data.

Data envelopment analysis

Another method used is data envelopment analysis (DEA). The horizontal axis represents the 1997 logarithm value of health spending per capita, and the vertical axis represents the HALE 2010 logarithm. There are differences between the graph in this study and the graph in the initial study, although the main focus points are in analogous positions.

Source: author’s own calculations based on WHO data.

As expected, we can see that the frontier production curve determined by the FHD method is within the frontier determined by the DEA method.

Subsequently, an attempt is made to estimate the frontier of the production function using the corrected ordinary least squares (COLS) method. In the first step, the residuals are determined using the classic method of the least squares estimation. After estimating the parameters, and in line with the assumptions of Evans et al.^[10], the estimated model takes this form:

In the next step, the maximum residual value from the above model is obtained. It is 0.31. The next stage is to divide the range containing all empirical observations into 100 parts within each unit of the axis, with the outlays marked in the form of health-related expenditure. In total, 620 values ​​of the logarithm of the expenditure on health per person according to the purchasing power parity in 1997 are obtained, increasing on the scale by 0.01, starting from 2.12 up to 8.32. For each of those values, the theoretical value of HALE is determined, adjusted by adding the value of the maximum residual component from the model estimated by the classical least squares method (CLSM) to the intercept. This enables us to determine the production function frontier presented in the diagram below using the COLS method.

In relating these results to those of Evans et al.^[11], it should be noted that the dispersion of empirical values ​​is similar, and the production function frontier intersects the same observation. Nevertheless, the shape of this function is different; i.e. the production function is non-monotonic in the original study, which means that it decreases in the range from seven for the lnEXP variable. Therefore, it is interesting to consider what method for determining the frontier Evans et al.^[12] adopted, because the article did not contain a description of it. Our calculations in this paper, indicate that the course of the production function is classical; i.e., it is a non-decreasing function.

Source: author’s own calculations based on WHO data.

Following the work of Evans et al.^[13], we then attempt to determine the frontier production function for the HALE data for 2000 and for the data on health expenditure per capita from 1997 by adopting precise assumptions as to the u_i distribution expressing the technical inefficiency of a given country. The authors were of the opinion that the type of distribution imposed does not affect the final results, and that the following distributions can be used: half-normal, truncated-normal, exponential, or gamma. This assumption will be verified below by determining the frontier production function for all the above-mentioned distributions, and, additionally, by showing the impact of the declared parameters on the final effect.
The first distribution we analyse will be the gamma distribution. The probability density function in this distribution is described by the following formula:

for        and .

The chart below presents variants of the gamma distribution, assuming different parameters and using the actual data used in the calculations. As in the COLS implementation, each unit of the interval containing the observed values ​​of lnEXP is divided into 100 parts, increasing on the scale by 0.01, starting from 2.12 up to 8.32; and n probability density function of the gamma distribution is determined in this interval. In particular, the presented range does not include values ​​between zero and 2.12, as there is no country for which lnEXP would fall within this range. Similarly, values ​​above 8.32 are not included.

Source: author’s own calculations.

From the above examples, the gamma distribution is chosen for the values ​​k = 2 and θ = 2. Theoretical values ​​are then calculated by adding the estimate value u_i from the gamma distribution to the lnHALE theoretical value, according to the formula:

In this way, the values of the frontier production function are obtained, assuming that the u_i distribution, which expresses the technical inefficiency of a given country, adopts the gamma distribution.

Source: author’s own calculations based on WHO data.

The next step is to carry out a procedure analogous to that described above, assuming that the u_i distribution, which expresses the technical inefficiency of a given country, will be described by the exponential distribution; i.e., a special case of the gamma distribution, where if X ~ Gamma (1, λ) then X ~ Exp (λ). The density distribution of the probability function is given by the formula:

As a result of the conducted experiments, the value of λ = 0.25 is selected, for which the limit production function has been obtained in the form presented in the diagram.

Source: author’s own calculations based on WHO data.

The probability density function in a semi-normal distribution is described by the formula:

for x > 0.

Source: author’s own calculations based on WHO data.

The figure above presents the production frontier curve, assuming that the u_i distribution, which expresses the technical inefficiency of a given country, takes a half-normal distribution.

The probability density function in the truncated-normal distribution is described by the following formula:

Source: author’s own calculations based on WHO data.

Contrary to the authors’ statement^[14] that the assumption of a specific form of the u_i distribution expresses the technical inefficiency of a given country, our results indicate that the obtained frontier productivity curves differ significantly in the dependence on the residual component imposed in the distribution model.

The chart below displays the values ​​assumed by the residuals of the model in different distributions in the analysed range. Clear differences can be observed between them, which obviously translates into the form of the frontier production function.

Source: author’s own calculations.

Assuming the truncated-normal distribution, the frontier production function obtained in this study takes exactly the same form as the frontier function presented in the article by Evans et al.^[15] This result confirms that we have accurately recreated the course of the analysis carried out in the above-mentioned article (despite using slightly different data and completely different computer programs to support our calculations), and that the differences observed between the forms of the production function, which result from the assumptions imposed on the u_i distribution, are not the result of an error. The obtained results have very important implications for efforts to measure the levels of inefficiency, which differ significantly depending on the method used and the distribution assumed.

The analysis carried out above helps to confirm that the frontier production function with a truncated-normal distribution superimposed on u_i actually corresponds very well with the empirical data. Nevertheless, this does not mean that making specific assumptions as to the u_i component has no effect on the frontier function of production.

Source: author’s own calculations.

In the next part of the research, we conduct an analysis of the latest empirical data; i.e., the HALE index for 2016 and the WHO data for 173 countries on health expenditure per person in 2016, expressed as purchasing power parity. The purpose of the following calculations is to show how the overall effectiveness of global health care systems changed over the 16-year period.

The FDH and the DEA analysis carried out with the use of non-parametric methods shows new shapes of the frontier production function curves after 16 years. The horizontal axis shows the logarithm value of health expenditure per capita from 2016, while the vertical axis shows the logarithm of HALE 2016.

It is easy to see that many observations have moved away from the origin of the coordinate system, which indicates that in these countries, both the level of expenditure on health per capita and the expected healthy life years increased. The dependency between the frontier production function curve determined by the FHD and DEA methods is the same as it was in 2000. However, after 16 years, there was a clear shift of the frontier production curves to the right, which means that there was an increase in per capita health care expenditure in many of the surveyed countries. However, no upward shift can be seen in the frontier production curves determined with the use of nonparametric methods, which means that the increase in health expenditure was not reflected in an increase in the value of HALE for the previously effective countries.

Source: author’s own calculations based on WHO data.

Source: author’s own calculations based on WHO data.

The results obtained using the COLS method are very interesting in light of the changes that took place between 2000 and 2016. The figure below shows the frontier production function determined by the corrected least squares method. The frontier curve based on the 2016 data is located at a much shorter distance from the observation cloud than it was based on the 2000 data. There is also a concentration of observations parallel to the frontier, which proves that there were differences between countries in the effectiveness of the funds spent on health, and that the origin of the coordinate system was moving towards the right, which in turn indicates that spending on health was increasing in many countries. It is interesting to note that the frontier production curve determined on the basis of the data from 2016 is below the curve for the data from 2000. When interpreting this situation literally, it is possible to conclude that the efficiency of spending decreased. However, it should be noted that this finding may be attributable to the method used and the reduction of variance in the 2016 data.

Source: author’s own calculations based on WHO data.

As before, using WHO data from 2016, estimates of the frontier production functions are made based on the stochastic approach. The parameter values ​​are assumed to be identical to those in the previous examples. Only the output data (and other values resulting from them, such as their natural logarithm values, mean, standard deviation, etc.) have changed. Presented below is the frontier of the production function, assuming that the distribution expressing the technical inefficiency of a given country takes a gamma distribution.

Source: author’s own calculations based on WHO data.

The stochastic frontier production functions for the exponential, semi-normal, and truncated-normal distribution are presented below.

Source: author’s own calculations based on WHO data.

Source: author’s own calculations based on WHO data.

Source: author’s own calculations based on WHO data.

The study using the HALE data for 2016 helps to confirm the results of the study based on the HALE data for 2000 in terms of conclusions regarding the imposition of the distribution on u_i. The frontier productivity curves differ significantly depending on the distribution imposed on the residual component in the model. Again, these differences can be illustrated with a diagram. The diagram below shows the values ​​assumed by the model residuals in different distributions in the analysed interval.

Source: author’s own calculations.

As a result of these differences, different estimates of the structural parameters of the models are obtained (adding the values ​​dependent on the adopted u_i distributions to the theoretical values ​​). Thus, the frontier production functions differ from each other depending on the assumptions made regarding the stochastic specification of the distributions.

Source: author’s own calculations.

To illustrate the shifts of the frontier production curve between 2000 and 2016, the results of the paired estimates of these functions are presented below. Regardless of the selected distribution imposed on the u_i expressing the technical inefficiency of a given country, an upward shift of the frontier production curve can be observed in each case. This shift expresses an improvement in the overall global effectiveness of health care systems.

 Source: author’s own calculations based on WHO data.

Discussion

Discussions about how the performance of health care systems should be measured started to play an important role in public debates as early as the 1980s, when many countries were seeking to reform their systems. The aim of these reforms was to improve all aspects of the functioning of health care systems, such as their financing and management, the prices and quality of medical services, and the availability and universality of health care.

In this paper, we analysed changes in the efficiency of these systems caused by the above-mentioned reforms and improvements, and by any other changes that affected the levels of health expenditure and of life expectancy in good health over a period of 16 years.

The aim of the article was to verify the assumptions that were arbitrarily made by the authors of the original article (Evans et al.^[16]), and to conduct the study again using data from 2016 in order to observe shifts in the frontier production function.

Our results indicate that the frontier productivity curves differ significantly depending on the u_i distribution imposed on the residual component in the model. As a result of these differences, different estimates of the structural parameters of the models were obtained. Hence, the frontier production functions differed from each other depending on the assumptions made regarding the stochastic specification of distributions.

In the analysed period, there was also a concentration of observations parallel to the frontier, which proves that the differences between countries in the effectiveness of the funds spent on health were decreasing, and that the origin of the coordinate system was moving towards the right – which, in turn, indicates that spending on health was increasing in many countries. On the other hand, the distance between the frontier efficiency curve and the origin of the coordinate system was small.

The obtained results allow us to confirm the assumption that worldwide, spending on health has been increasing over time. However, the results also raise questions about this process, because when we used the methods discussed in the paper, the frontier efficiency function actually moved away from the origin point of the coordinate system, thus showing an increase in expenditure on health production; but it did not change its position relative to the lnHALE variable. This observation may, for example, indicate that the cost of health care has been increasing over time, or that the group of people on whom these funds are being spent has been growing. It can certainly serve as a starting point for further research in the field of health care systems.

Most of the previous research conducted on the effectiveness of health care systems was based on life expectancy. There are no exhaustive analyses and no unequivocal answers to questions about the channels through which various approaches in the organisation of national health care systems influence economic growth and labour productivity. Indeed, the possible impact of health care systems on economic growth has barely been mentioned^[17]. The existence of this research gap is an interesting starting point for further analysis.

The effects of the current pandemic situation will be observed in the years to come, and the conclusions and methods presented in this article may provide an interesting starting point for further research in the area of health care efficiency measurement, as well as for discussions on decisions regarding health care financing methods made by state governments.