Principle: This study attempted to investigate the effect of two independent variables on achievement in organic chemistry. The first variable was a concept map as a navigation interface in a computer aided instruction program . A second independent variable was creating a concept map as a learning activity.
Design: This experiment was a posttest-only control group design
The two different treatments were 1) using partial concept maps in CAI and constructing concept maps and 2) using complete concept maps in CAI and taking notes. The control was using menu selection in CAI and taking notes. The posttest was a chemistry achievement test which was 60 % high-level questions and 40 % low-level questions as classified by Bloom's taxonomy. In addition, each student was tested on their ability to create a concept map.
Factors jeopardizing internal validity: Most of the threats have been considered in this design, however the possibility of non-equivalent groups is always present in small samples. Also in rating the concept maps only one person judged the maps so there might be a problem with instrumentation because there was no check for inter-rater reliability.
Factors jeopardizing external validity: The main threat to external validity is the interaction between selection bias and the experimental variable. The sample was taken from a special school which serves some of the top 10% of the students in Singapore. Children with high IQs might react differently to concept maps than the average student.
Adequacy of statistical procedures used: The author correctly used an ANCOVA to statistically control for the effects of the extraneous variable, mid-year chemistry achievement. In addition, the Bonferroni adjustment technique to keep the Type I error in line with the stated alpha level was correctly used to determine the pairwise effects.
Logic summary: The author's logic is flawed. With the design combining the two independent variables, it is impossible to demonstrate whether it was the partial concept map, the concept mapping activity or the interaction between the two that improved achievement.
Design improvement: The author apologized for the lack of a fully crossed design (3 types of CAI interfaces X 2 types of learner's activities), but that would be the only way to accurately determine what is causing the increase in achievement. A more even ranking of the aspects of creating the concept map would also improve the operationalization of the variable.
Extension of the study: A fully crossed design should be used to accurately determine the effects of the two independent variables. Another way to test the use of concept maps on the CAI interface would be to have the links on the relationships instead of the concepts and see if that increased the achievement. A more representative sample would also increase the external validity.