"The evidence-based methodological standards adopted by the Panel are essentially those normally used in research studies on the efficacy of interventions in psychological and medical research" (pp. 1-5).
The Panel noted that the review of reading research should be no less rigorous than those noted above but that "such standards have not been universally accepted or used in reading education research" (pp.1-5).
"Unfortunately, only a small fraction of the total reading research literature met the Panel's standards for use in the topic analyses" (pp. 1-5).
Topics and subtopics (described in Course #2) were developed based on a review of the Snow et al. (NRC, 1998) report as well as public hearings conducted across the country. Next, these topics and subtopics became the subject of the work of subgroups comprised of one or more Panel members.
There were about 100,000 research studies on reading published since 1966 and about 15,000 before that time identified through a variety of public databases. Thus, a screening procedure was absolutely essential. The Panel had to determine common procedures for how to include (and exclude) studies for consistency across Panel members. These common procedures were grounded in scientific principles. Various criteria were established for selecting articles, such as the period of time covered by the searches and how many articles could be retrieved in this time period; research had to focus directly on reading development from preschool to Grade 12, and the studies had to be published in English in refereed journals.
PsychINFO and ERIC databases, among others, were searched. Once articles were identified, they were examined to determine their relevance (title and abstract review). If studies were not relevant, they were excluded. If they were relevant, they proceeded to the next evaluation. In this evaluation the following were reviewed:
· Study participants must be carefully described
· Study intervention must be described in sufficient detail to allow replicability
· Study methods must be described to ensure that instructional fidelity was maintained
· Studies must include a full description of outcome measures
Again, if studies did not provide this information, they were excluded; if they did, they were further evaluated. Any time a study was excluded, careful notes were provided as to why this was done. If a set of studies was deemed insufficient to address the topic (e.g., too few studies), additional search procedures such as manual (hand) searches and the inclusion of less recent studies were done.
The Panel focused exclusively on studies published in peer-reviewed journals; however, those studies not meeting this criterion were treated as preliminary or pilot data and became the focus of areas of future research.
The Panel included only experimental or quasi-experimental studies as well as multiple baseline designs. If there were insufficient numbers of these studies for a topical area, descriptive or correlational studies were used (no claim could be made on these studies alone, however).
Characteristics of these "core" studies were coded and analyzed. The following categories were included (note: sub-categories are not provided for brevity’s sake):
(a) Reference
(b) Research question
(c) Sample of student participants
(d) Setting of the study
(e) Design of the study
(f) Independent variables
(g) Dependent (outcome) variables
(h) Nonequivalence of groups
(i) Result for each measure
(j) Coding information.
When appropriate, effect sizes were calculated for each intervention or condition in experimental and quasi-experimental investigations. If these could not be completed, a literature analysis was done. The Panel, in its entirety, reviewed and approved each decision for completing a meta-analysis (use of effect sizes) or literature analysis, again to ensure consistency. Overall effect sizes were calculated. Further, the Panel compared the magnitude of the treatment’s effects under various identified conditions such as program contexts and outcome measures used.
Studies included for review appear in the NRP Report in various appendices that outline the studies in some detail. Results of the NRP analysis appear in some detail in Course #3.