Lying About and Cheating with the Evidence

Teaching Reading – The Lies and Deceit in Studies of Evidence-Based Research

In Jean Stockard’s recently published book, The Science and Success of Engelmann’s Direct Instruction, the author reviews how educational research is chosen and publicized and how agencies lie to us about scientific research in education.

An Example of Good Science – Project Follow Through

• The largest educational research study ever done
• Compared 22 different teaching methods
• Studied hundreds of thousands of at-risk students in the primary grades (K-3) for many years
• Direct Instruction was the only method  with positive gains for all outcomes
• Direct Instruction produced the highest scores on all outcomes, academic and affective.
• Direct Instruction higher long-term positive effects for students (e.g. graduation rates)

A Non- Example of Good Science – Project Star

• A study of limiting class size to 23 or fewer in California
• Involved 1650 students for four years
• Classes in the same school were randomly assigned to the control or experimental group with the experimental group having no more than 23 students per class.
• Students in the experimental group (smaller classes) had higher achievement scores than those in the larger classes. (.21)
• The results were below the level typically used to qualify as “educationally important.”
• Results for Project Star were only a fraction of the results for Direct Instruction students in Project Follow Through (1.41)

How the results were handled

Project Star

• The results of the Star Project were widely publicized. As a result he state of California enacted a bill limiting class size.
• In the 1994 National Assessment of Academic Progress, California placed second from last in reading with 18% of its students rated as “proficient” or “advanced’ in reading.

Project Follow Through

• The What Works Clearinghouse which was founded by the Department of Education to gather and disseminate research on effective methods disqualified Project Follow Through as being outdated with no explanation as to why these data were no longer acceptable.
• Fewer than 10 % of all studies are published.  Only 5% of the hundreds of Direct Instruction studies qualified for publication under WWC guidelines.
• The WWC has established research criteria that virtually assure that field studies will be rejected and only limited, highly controlled studies can be considered for publication.  Studies that meet these criteria despite their weak results are publicized; those with robust results that violate their view of “scientific research” are tossed into the trash can. By setting a model for science that precludes studies in the field, WWC biases the educational approaches like Direct Instruction, despite its consistently strong results for over half a century.

Like beauty, skin-deep scientific evidence is in the eye of the beholder. Real science requires an acceptance of the methods commonly used in the rest of the social sciences, not an arbitrary determination of a particular research model adopted by WWC. Such a limitation is a disservice to science and is a huge cost to the 15,000,000 currently at-risk students for whom we need better academic practices built on sound applied science.