The Relevance of “RealWorld Evaluation” to Autism Research
Tamara Daley, Nidhi Singhal, Merry Barua, Tom Weisner, Rachel Brezis
In some scenarios, a researcher may encounter an autism intervention that has already been implemented within a community setting, and it may or may not have been developed in a university setting or through a Random Clinical Trial. This can occur when an administrative unit such as a district or county has implemented or scaled up an intervention across a broad population of children (e.g., Nicholson, Berthelson, Williams & Abad, 2010; Anderson, Birkin, Seymour & Moore, 2006). It is also very common in low and middle income countries, where interventions for autism have almost exclusively emerged from within the community rather than a research laboratory. For these situations in particular, there is value in conducting an effectiveness study of existing programs prior to or in conjunction with introducing and adapting models from other settings. Borrowing from the broader field of international evaluation, autism interventions can be examined through the lens of RealWorld Evaluation (RWE; Baumberger, Rugh & Mabry, 2006). This approach offers an opportunity to expand the knowledge base about available and effective interventions while working within the constraints typical of community settings.
The current study presents a systematic evaluation of a parent training model for autism at Action for Autism, India, following the steps evaluation using a RWE approach.
The first six steps of the RWE method were used to frame and conduct the evaluation. First, the program theory was used to develop a logic model for the program. Budget, time, data, and political constraints were examined and incorporated into the design. The evaluation took place over three consecutive sessions of a 12-week parent training program developed in India. After extensive testing of the measures, a total of 40 participants were administered pretest and posttest measures using a set of questionnaires tied directly to the program theory. Assessing and addressing the strengths and weaknesses of the evaluation design has taken place in consultation with the organization operating the intervention.
Using the RWE approach, this evaluation demonstrated significant pretest – posttest comparisons across multiple measures of parent knowledge, empowerment, acceptance, and skill. The use of a post-test comparison group further strengthened the validity of these findings. The greatest challenges to the evaluation were the need to limit data collection primarily to quantitative measures to reduce burden on staff and participants, and accommodating and accounting for naturally occurring variations in the intervention.
The current study is the first use of the RWE approach for an autism intervention. Resulting in both formative and summative data, this study was successfully conducted with minimal changes to the existing structure, content and procedures of the program. As an approach, RWE was developed for those working under budget, time, data and political constraints. As we seek to understand more about effective and culturally appropriate intervention approaches for children in diverse contexts, a RWE process has particular salience, particularly where there is little infrastructure and limited expertise to support more rigorous research designs.