Clues for Evaluating Research Studies

Introduction

One form of authority that relies a great deal on observation and often carries special weight is the research study: usually a systematic collection of observations by people trained to do scientific research. How dependable are research findings? Like appeals to any authority, we cannot tell about the dependability of research findings until we ask lots of questions.

Society has turned to the scientific method as an important guide for determining the facts because the relationships among events in our world are very complex, and because humans are fallible in their observations and theories about these events. The scientific method attempts to avoid many of the built-in biases in our observations of the world and in our intuition and common sense.

What is special about the scientific method? Above all, it seeks information n the form ofpublicly verifiable data—that is, data obtained under conditions such that other qualified people can make similar observations and see whether they get the same results. Thus, for example, if one researcher reports that she was able to achieve cold fusion in she lab, the experiment would seem more credible if other researchers could obtain the same results.

A second major characteristic of scientific method is control—that is, the using of special procedures to reduce error in observations and in the interpretation of research findings. For example, if bias in observations may be a major problem, researchers might try to control this kind of error by using multiple observers to see how well they agree with one another. Physical scientists frequently maximize control by studying problems in the laboratory so that they can minimize extraneous factors. Unfortunately, control is usually more difficult in the social world than in the physical world; thus it is very difficult to successfully apply the scientific method to many questions about complex human behavior.

Precision in language is a third major component of the scientific method. Concepts are often confusing, obscure, and ambiguous. Scientific method tries to be precise and consistent in its use of language.

While there is much more to science than we can discuss here, we want you to keep in mind that scientific research, when conducted well, is one of our best sources of evidence because it emphasizes verifiability, control, and precision.

Problems with Research Findings

Unfortunately, the fact that research has been applied to a problem does not necessarily mean that the research evidence is dependable evidence or that the interpretations of the meaning of the evidence are accurate. Like appeals to any source, appeals to research evidence must be approached with caution.

Also, some questions, particularly those that focus on human behavior, can be answered only tentatively even with the best of evidence. Thus, there are a number of important questions we want to ask about research studies before we decide how much to depend on their conclusions.

When communicators appeal to research as a source of evidence, you should remember the following:

Research varies greatly in quality; we should rely more on some research studies than others. There is well-done research and there is poorly done research, and we should rely more on the former. Because the research process is so complex and subject to so many external influences, even those well-trained in research practices sometimes conduct research studies that have important deficiencies; publication in a scientific journal does not guarantee that a research study is not flawed in important ways.
Research findings often contradict one another. Thus, single research studies presented out of the context of the family of research studies that investigate the question often provide misleading conclusions. Research findings that most deserve our attention are those that have been repeated by more than one researcher or group of researchers. We need to always ask the question: “Have other researchers verified the findings?”
Research findings do not prove conclusions. At best, they support conclusions. Research findings do not speak for themselves! Researchers must always interpret the meaning of their findings, and all findings can be interpreted in more than one way. Thus, researchers’ conclusions should not be treated as demonstrated “truths.” When you encounter statements such as “research findings show…” you should retranslate them into “researchers interpret their research findings as showing . . .”
Like all of us, researchers have expectations, attitudes, values, and needs that bias the questions they ask, the way they conduct their research, and the way they interpret their research findings. For example, scientists often have an emotional investment in a particular hypothesis. When the American Sugar Institute is paying for your summer research grant, it is very difficult to then “find” that sugar consumption among teenagers is excessive. Like all fallible human beings, scientists may find it difficult to objectively treat data that conflict with their hypothesis. A major strength of scientific research is that it tries to make public its procedures and results so that others can judge the merit of the research and then try to replicate it. However, regardless of how objective a scientific report may seem, important subjective elements are always involved.
Speakers and writers often distort or simplify research conclusions. Major discrepancies may occur between the conclusion merited by the original research and the use of the evidence to support a communicator’s beliefs. For example, researchers may carefully qualify their own conclusions in their original research report only to have the conclusions used by others without the qualifications.
Research “facts” change over time, especially claims about human behavior. For example, all of the following research “facts” have been reported by major scientific sources, yet have been “refuted” by recent research evidence:
Prozac is completely safe when taken by children.
It is important to drink eight glasses of water a day.
Depression is caused entirely by chemical imbalances in the brain.
Improper attachment to parents causes anti-social behavior in children.
Research varies in how artificial it is. Often, to achieve the goal of control, research loses some of its “real-world” quality. The more artificial the research, the more difficult it is to generalize from the research study to the world outside. The problem of research artificiality is especially evident in research studying complex social behavior. For example, social scientists will have people sit in a room with a computer to play “games” that involve testing people’s reasoning processes. The researchers are trying to figure out why people make certain decisions when confronted with different scenarios. However, we should ask, “Is sitting at the computer while thinking through hypothetical situations too artificial to tell us much about the way people make decisions when confronted with real dilemmas?”
The need for financial gain, status, security, and other factors can affect research outcomes. Researchers are human beings, not computers. Thus, it is extremely difficult for them to be totally objective. For example, researchers who want to find a certain outcome through their research may interpret their results in such a way to find the desired outcome. Pressures to obtain grants, tenure, or other personal rewards might ultimately affect the way in which researchers interpret their data.

As you can see, despite the many positive qualities of research evidence, need to avoid embracing research conclusions prematurely.