February 13, 2001
Assignment #2: Critique of an Experimental Study
Reference: Supinski, S. B. (1999). An experimental comparison using cooperative learning and interactive video: Lessons for the interactive classroom. The Journal of Interactive Learning Research, 10 (2), 123-148.
Principle: This study compares two instructional strategies using cooperative learning (CL) and interactive video instruction (IVI) and measures the effect of these strategies on achievement and attitude.
Type of Design: It is a quantitative study using a posttest only design (no control group). It compares two types of treatments with each other. The one independent variable, type of treatment, has two levels, sequential and simultaneous. There are two dependent variables: achievement and attitudes. Achievement was measured with three dialogue construction worksheets and by performance on an instructor developed exam. Attitude was measured with a self-report questionnaire given at the end of the study. This design does not fit any of the ones described in Campbell & Stanley. (It does not fit Design 6: Posttest-Only Control Group.) Therefore, an intuitive diagram of this design type is:
R X1 or 2 O1 & 2
Factors Jeopardizing Internal Validity: Internal validity is also termed main effects. Campbell & Stanley lists eight factors that may affect internal validity. These are history, maturation, testing, instrumentation, regression, selection, mortality, and interaction. It seems that the primary threats to internal validity in this study are in the selection, testing and interaction phases of the study. The author himself points out that the "lack of purpose and structure [when using both CL and IVI] may explain why gains in achievement were not realized." There are several confounding variables that may include prior level of computer knowledge and learning styles.
Factors Jeopardizing External Validity: External validity, also called interaction effects, influences the generalizability of a research study. They include the interaction of testing and X, the interaction of selection and X, reactive arrangements, and multiple-X interference. The author of this study states "In terms of supporting the continued use of CL with IVI, definitive conclusions and generalizations cannot be made due to the lack of a control group...[T]he lack of a control group...prohibits teasing out the variables attributable to specific levels of achievement." This conclusion seems to have been reached because of the numerous interactions that occurred during the course of the experiment as well as the fact that the study used no control group against which to measure the treatment. Reactive interactions were documented as anecdotal comments.
Adequacy of Statistical Procedures Used: The simplest statistic for this design is the t-test. The author of this article used the t-test to look for differences between means on the attitude measure. He used a mix of the t-test and the Wilcoxon rank sums method, which is also an appropriate statistic for this design, to look for differences between means on the achievement measure. He found no significant differences on the achievement measure and only one significant difference on the attitude measure. That difference indicated the students preferred the CL form of instruction to the IVI form.
Brief Summary of Logic (Inductive/Deductive): The line of reasoning that ties the empirical results obtained in this study to the principle being investigated seems to be the conclusion reached by the author that the "observed success of both treatments does provide support to those researchers who advocate the use of cooperative learning and interactive video instruction to enhance second language acquisition." The logic for conducting this study appears to lie in two statements made by the author: "Logic dictates that when CL and IVI are combined, achievement in language acquisition should improve" and in the observation that "[s]imilar results [ie, no significant differences] in numerous studies in other educational content areas suggest that the question of whether one can achieve improved educational benefit by combining CL with [IVI] has yet to be resolved."
Design Improvement: This study does not appear to represent a true experimental design. Since there is no control group against which to measure the treatment, accurate and reliable information from this study is difficult to identify even though the individual students were randomly assigned to a treatment group as a true experiment requires. A better design would be to use a posttest-only control group design (Design 6 in Campbell & Stanley). This design adequately controls for the elements of internal validity and controls for the interaction of testing and X as an element of external validity. A diagram of design 6 is:
R X O
Extension of the Study: Additional studies that could extend the idea presented by this study include further research to determine how mature learners are best grouped and how to most effectively use two-person workstations in language laboratories. A design that seems to fit these recommendations is the Solomon Four-Group design, specifically because it parallels Design 4 and controls for the external factors of validity that influences generalizability to other populations. A diagram of the Solomon Four-Group design is:
R O1 X O2
R O3 O4
R X O5