The Nature of Evidence
Show Me the Evidence was a popular book by Ron Haskins and Greg Margolis published by Brookings in 2014. The central idea was that economic and social policy should be based on credible data rather than just opinion and advocacy. This seems reasonable, although ideology has of late disrupted these intentions.
Can this idea be reasonably generalized to a wide range of domains? The answer clearly depends on the existence and accessibility of relevant data. There are also issues of types of data.
One class of data concerns “what is.” How many people live where, work where, have what levels of education, and what levels of income? How many cars, appliances, books, etc. sold last year? These types of data are increasingly available.
Another class of data concerns “what if.” You cannot measure what has not yet happened. If we implement policy X will outcome measure Y increase? There are many examples of this type of question.
Will school vouchers increase enrollment? Will tax reductions lead to greater investment? Will increased fuel economy regulations decrease global warming? These types of questions require methods for predicting the future consequences of current actions.
One approach to prediction is extrapolation. One assumes that the past measured relationship between X and Y will continue even if the values of X and Y are outside the ranges previously measured. This can be a tenuous assumption.
Another approach is to develop a mathematical or computational model of the phenomena underlying the relationship between X and Y. This requires an understanding of these phenomena, which sometimes can be gleaned from published studies of these phenomena.
Usually this involves much more work, which may be justified if the problem of interest is important. It may also involve convincing stakeholders in the problem that your assumptions are valid and computations correct. This can be a challenge for stakeholders without technical backgrounds.
Additional challenges include difficulties when the underlying phenomena are not understood — by anybody — or there is no consensus on the nature of the underlying phenomena. Developing multiple models and comparing their predictions can address this. A good example of this is the use of multiple hurricane models to predict their paths.
One can also use sensitivity analysis to assess the impacts of uncertain parameters within computational models. Quite often, a few parameters strongly affect predictions while others have minimal impacts. One can then focus on refining estimates of the most impactful parameters.
Scenario analysis is often employed to explore different strategies rather than just trying to fine-tune one scenario. This can include varying initial conditions and, in general, identifying the conditions under which each scenario is superior to the other scenarios.
These techniques enable one to explore what “might” happen. However, this exploration rarely results in knowing what “will” happen. One ends up with a set of well-reasoned possibilities and insights into leading indicators of how these possibilities may be manifested.
Are such results “evidence-based”? In the context of pursuing “what if” rather than “what is,” this is about as rigorous as one can be in terms of addressing what might happen. What will happen will have to wait until it happens.
Can all decisions be addressed this way? No. Some decisions are based on what one believes to be right. This is the realm of values and ethics. Consider the Golden Rule — Do unto others as you would have them do unto you.
What is the evidence that this rule is correct? Many religions advocate a version of this rule. So, it is popular. However, lots of things are popular. That cannot be sufficient evidence.
Another rationale is that it makes sense, at least for humans if not for lions and gazelles, or wolves and rabbits. Perhaps it makes sense if we are all going to get along together.
Thus, getting along together is a value. One reason we value this is that everyone benefits and does not feel marginalized. Another reason is that it inhibits resistance, confrontations and possibly violence.
Let’s consider the Constitution. All people are created equal and everyone has inalienable rights to life, liberty and the pursuit of happiness. What is the evidence that this is right? We do have evidence of the consequences for people living in societies that do not prescribe to these values.
However, that is negative evidence. What is the positive evidence that these values are good and useful? Having aspired to live by these values, however imperfectly, for 200+ years has resulted in a dynamic, thriving country. And, we aspire to live less imperfectly in the future.
But the conundrum remains. We have no evidence of how the country might have progressed under a different value system. There have been no randomized clinical trials.
Of course, we have no evidence for alternative college majors, alternative mates, and all the jobs we did not take. We likely believe that we made good choices, particularly if the consequences were good. However, we have no evidence that these were the best choices.
Interestingly, we sometimes have evidence that decisions were poor because the consequences are sufficiently negative to know that we would rather have avoided these choices. The type of models discussed earlier can help with this. While we cannot predict exactly what will happen, we can often predict that undesirable outcomes are very likely and get rid of bad ideas quickly.
So, everything cannot be evidence based. It would be unwieldy, impractical, and often impossible. Nevertheless, when it makes sense, evidence-based decision making is a good practice.
