Research Validity

nternal validity is the result of an experiment being crafted in such a manner that the outcome of the experiment matches its teleology. That is to say it in fact does what it purports. In order to have internal validity the experiment must demonstrate that it was the variable manipulation and not some other extraneous cause that brought about the effect. External validity is the ability of the experimental results to be extrapolated to the general population or to another sample group.

Threats to validity are important considerations when designing studies because first the most effective response to validity threats is in the original design, specifically the design of the sampling procedure. The choice of participants must be made in such a manner as to maximize the similarity both between groups and between the groups and the population at large in order to minimize the threats to validity. There should be similarity between groups to increase internal validity; similarity between both groups and the general population to increase external validity.

Because studies take place over time the elapse of time itself can be a confounding variable especially in the study of children wherein the elapse of even a small amount of time can effect drastic changes in the operating parameters of the subjects. Also because experiments take place over time, attrition is a threat to validity not only mortality proper, but attrition due to relocation and simple dropout. These changes in the selected groups may alter the parameters carefully chosen at the beginning of the study.

During the time interval between the beginning and conclusion of a study, events may occur outside of the experiment which have a confounding effect even if irrelevant to the study. For example, a study of seasonal affective disorder in New York City conducted from August through December of 2001 might have confounded results through no fault of the study design. Outside influences can also threaten external validity, wherein a subset of a group receives an experience that enhances or detracts from the experimental treatment. Whereas the threat to internal validity is to the attribution of cause, the threat to external validity is to generalizability.

Naturally occurring developmental changes to the sample population may also introduce confounding causes. Activities of the sample population outside of the experiment may also have an adverse effect on the end result for example; a new exercise or diet fad amongst the general population may spoil the results of a concurrent weight loss study.

Pretesting may sensitize participants to not only the content of the instrumentation but also to the process of testing. Improved scores on tests may reflect improvement on what is being measured but it may merely measure the aspect of practice makes perfect. Test-pretest sensitivity is a threat to internal validity that threatens external validity as well in that the experiment repeated without the pretest may yield different results not attributable to any other factors in the repetition thereby adversely affecting generalizability.

When the administration or scoring of a test is inconsistent, the interpretation of the results has no sound basis. Validity can not only be affected by the interpretation of the statistics, but also by the statistical model itself. While regression toward the mean may be ameliorated by dropping outliers out of the equation, it is a trifle naïf to assume that the choice of the use of the mean or the median has no effect on the analysis of the data. Also the experimenters themselves may have a deleterious effect on results, whether in the administration of the test or by unintended non-verbal cues of the tester. The threat to external validity labeled reactive arrangement is also known as the “Hawthorne effect” which has been explained as an effect of participants being aware of the experiment. However there is a simpler explanation. Elton Mayo’s observations were of worker productivity at a factory (Bell, Paul A., Green, Thomas C., Fisher, Jeffery D., Baum, Andrew 2001). Anyone who has worked on a production line would be able to explain the increase in productivity when an experimenter changed the independent variable not so much as the result of condition changes but as a result of worker fatigue having a smaller effect on worker productivity when the worker is under supervisory or other observation. It is also trifle naïf to assume that observation has no effect upon the thing observed.