Criteria for evaluating the innovative “Project Learning” projects
Cuban (1998) remarks that in evaluating the success or failure of a reform, two major sets of criteria, namely one set of standards related to effectiveness, popularity and fidelity as well as another set of standards related to adaptiveness and survival (or longevity), needs to be addressed. The effectiveness standard usually refers to what students have learned in school by using proxy measures such as student test scores and other measures to determine the success of change outcomes. The popularity standard refers to the extent of the spread of an innovation and/or to the extent of public receptivity to the innovation. The fidelity standard refers to the correspondence between the initial design, the formal policy, the subsequent program(s) it generates, and its implementation. The adaptiveness standard refers to the way in which teachers adapt the innovation and make the innovation work with respect to their own school context. The survival standard refers to how long an innovation can sustain and become influential (Lee, 2004).
On the other hand, in the field(s) of medicine and public health, the RE-AIM framework, which is compatible with systems-based or social-ecological thinking, provides valuable information that may facilitate the translation of research to practice (e.g., Dzewaltowski, Glasgow, Klesges, Estrabrooks, Brock, 2004; Glasgow, Vogt and Boles, ). The RE-AIM evaluation framework/model refers to interactions of five separate evaluative dimensions: Reach, Efficacy/Effectiveness, Adoption, Implementation, Maintenance. While Reach and Efficacy/Effectiveness are related to individual-level indicators and Adoption and Implementation are related to organizational levels of impact, Maintenance is assessed at both an individual and organizational level of impact. Table 1 below broadly compares the five dimensions of RE-AIM modified for educational projects and settings with Cuban’s (1998) criteria for evaluating educational reforms and innovations.
Table 1 RE-AIM framework modified for educational projects and settings and Cuban’s criteria (based on Dzewaltowski, Glasgow, Klesges, Estrabrooks, Brock, 2004, p.76; Glasgow, Vogt and Boles, p.38).
| Research evaluation issue |
Dimensions and level of analysis of RE-AIM | Definition of RE-AIM dimensions modified for educational projects/settings | Larry Cuban’s (1998) Criteria |
|---|---|---|---|
| Representativeness of participants | Reach (individual level) | What is the number or percentage of eligible participants (i.e. students) (a) took part, (b) eligible but excluded and (c) how representative were they? | Popularity (within one school or project schools) |
| Outcomes under optimal and real world conditions | Efficacy / Effectiveness (individual level) | What impact did the project have on (a) all participants who actively took part the program and (b) on both (intended and unintended) positive and negative (unintended) outcomes? | Effectiveness |
| Representativeness of participants | Adoption (setting / agent level) | How many organizations and other participants within these settings (e.g., teachers, parents) were willing to participate and how representative were they? | Popularity (within the project school(s) and beyond the project school) |
| Outcomes under real world conditions | Implementation (setting / agent level) | To what extent were the various components delivered as intended (in the design) or adapted when conducted by different members in applied settings? | Fidelity versus adaptiveness |
| Long-term maintenance of behavior or practices | Maintenance (setting level) Maintenance (individual level) |
What is the extent to which a program or policy becomes institutionalized or part of the routine organizational practices and policies? What are the long-term effects of a program on outcomes (individual) after the most recent/ last implementation? |
Survival or longevity |
There are some similarities and differences in dimensions between Cuban’s criteria and RE-AIM framework but it must be cautious that:
- There are differences between education innovations (e.g., Project-based learning) and public health interventions in terms of nature of “innovation” versus “intervention”, target population to which each individual project/intervention intends to generalize;
- The outcomes of the Project Learning may be much more difficult to be gauged both “objectively” and “accurately” as compared with those clinical trials from health interventions;
- The “Project Learning” projects supported by QEF especially those in the school sector may not have adopted rigorous measures of different dimensions of project effectiveness as well as evaluation of implementation; and
- As Project Learning is one of the key tasks of curriculum reform, the selected “Project Learning” projects may have quite a progress over the years or adaptation in operation after the QEF sponsorship is completed.