“Because change takes time, successes are not always recognized when they occur. Communicating to others exactly what you are trying to accomplish and how you will know that you are making progress can also be difficult” (Reisman & Gienapp, 2004, 1).
Developing a programme theory allows stakeholders to explore the causal relationships that explain why and how a change will occur as a result of an initiative. It allows them to recognise when change is taking place, as well as communicate their intentions to others.
Sharpe states that “program theory should be developed prior to the commencement of the program. However, even if the program is underway, it is important for a program theory to be developed” (2011, 72). This is the situation we found ourselves in. Our school had adopted a new math programme, Numicon, but there was no shared understanding about the assumptions we were making as to how this programme would improve students’ success in mathematics. As such, it was difficult for us to know whether or not the programme was achieving its goals.
Programme rationale (theory of change) or the “why” of the programme: Developing our programme rationale through backcasting – articulating our desired future and working backwards to the present – clarified the leadership teams’ underlying assumptions and allowed us to then communicate this with other stakeholders.
Our theory of change is based on the social constructivism theory of learning. We believe that children must construct their understanding of the world by engaging with it. Children develop mental models, theories of how things work, that change and expand over time as they encounter new experiences.
In the context of learning mathematics, we think that all students must understand foundational number concepts. Students who develop robust mental models of abstract number concepts will be more successful in understanding and applying these concepts. The question is, how do we support students in developing these mental models? Constructivist theory says we must give students opportunities to engage with these concepts in meaningful ways. Given that these concepts are abstract (number, place value, etc…), we posit that providing a concrete model that they can use to represent these concepts will help them build these mental models.
Specific need: School data indicates students are struggling with abstract number concepts. Number concepts are foundational to all subsequent mathematical learning.
Specific intervention: Use a math programme (Numicon) which employs concrete models to represent abstract number concepts so students can develop mental models of these concepts. Using these concrete models also allows teachers to “see” student’s thinking, therefore providing teachers an opportunity to identify and correct students’ misconceptions.
Target population: All students within the school. Given that students need multiple opportunities to interact with these physical models in a variety of contexts in order to develop robust mental models, we think all students within the school will benefit from using this programme.
Intended outcome: Students scores on a variety of mathematical assessment tasks improve.
Identifying goals allow stakeholders to monitor/evaluate the effectiveness of their programme. In our context we identified our short term goals as:
- students learn the value of each Numicon resource (concrete models)
- students use the Numicon resources to talk with peers about their mathematical thinking
- student engagement in mathematics lessons increases
- students use the Numicon resources to solve mathematical problems
- students use the Numicon resources to prove their theories, so teachers can “see” their thinking
- teachers use the Numicon resources to model solutions to mathematical problems
- formative assessment results improve
Our longer term goals are those we would expect to see once the students have been using the programme for a few years. These longer term goals are:
- summative and standardised assessment results improve (intended)
- older students become less reliant on or use the Numicon resources less frequently to solve previously taught mathematical problems (unintended desirable)
- students show a greater degree of resilience when confronted with challenging math problems (unintended desirable)
Programme plan (theory of action) or the “how” of the programme:
To clarify our thinking about how the programme theory would work in practice, we developed a series of if/then statements. This was an effective way of seeing the causal links. It also helped to identify some of the implementation issues we would need to address, specifically capacity building and communicating clear expectations to teachers.
The Numicon programme is new to all of our teachers, so we will need to invest in training to build their knowledge base and capacity. To do this we have identified the following strategies:
- share training videos with year level teams during staff PD sessions and have them engaged in visible thinking routines, such as Connect/Extend/Challenge to determine how to implement this new learning,
- share best practice with each other during weekly planning meetings – teachers can talk about a lesson that has gone well, how it was done, and how students responded,
- hold vertical articulation meetings each term for teachers to share strategies, identify issues in implementation, share assessment data, and explore possible solutions,
- providing teachers access to other teachers with a strong mathematics background for knowledge-building support on an as needed basis,
- ask questions of the leadership, and
- engage in a year level lesson study cycle during term 3.
Additionally, we want the teachers to implement the programme as it is laid out in the teaching guide. We think this will allow us to gather reliable data later on and ensure the programme is being taught using the methodology set out by the researchers who designed the programme. This needs to be more clearly communicated to teachers and will be accomplished by:
- holding a professional development session to present the completed theory of change and theory of action models to staff, review the components of the programme and the timeline for implementing them, review the role of the teacher and assessment guides, and answer questions,
- working with the curriculum coordinator in term 3 to review long-term and medium-term planning in preparation for the next school year,and
- outlining how the programme will be evaluated and how teachers can contribute to it when it commences.
Here is the action model:
As there are two parts to our theory of action, our evaluation will also have two parts.
As the ultimate goal of any educational programme should be an improvement in student learning, an outcome evaluation is essential. To evaluate whether the Numicon programme is achieving its aim to help students develop robust mental models and an improved understanding of mathematical relationships, we will begin by conducting a short-term outcomes evaluation. This evaluation will be conducted by the school leadership in a non-participatory manner and the results will be shared with the teaching staff. An additional outcomes evaluation can be completed again in a couple years time and include gathering data from other sources, such as middle school math results, to see if our Numicon cohort fared better than the non-Numicon cohort.
To understand how well teachers are implementing the programme and using the Numicon programme to see and correct student misconceptions, a process evaluation will be most beneficial. The Numicon resources are new to our teachers and students, so we want to understand how we can best support them in implementing the programme. The design of this part of the evaluation will be collaborative, with the aim to increase teachers’ understanding of the programme, develop shared knowledge, increase their commitment to teaching the programme and increase their sense of power in being able to respond to implementation challenges. We think this type of evaluation also supports the constructivist social learning theory that underlies the programme, as teachers will explore evidence that can reaffirm or challenge their mental models and allow school leadership the opportunity to see teacher thinking and address misconceptions.
Chen, Huey-tsyh. Practical program evaluation: theory-driven evaluation and the integrated evaluation perspective. Los Angeles, SAGE Publications, 2015.
Donohoo, Jenni. “Theory of Action.” Newsletters, Learning Forward Ontario, 2014, learningforwardontario.ca/resources.html. Accessed 21 Feb. 2017.
Lawton, B., Brandon, P.R., Cicchinelli, L., & Kekahio, W. (2014). Logic models: A tool for designing and monitoring program evaluations. (REL 2014–007). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Pacific. Retrieved from http://ies.ed.gov/ncee/ edlabs.
Reisman, Jane , and Anne Gienapp. “Theory of Change.” The Annie E. Casey Foundation, Organisational Research Services, 2004, http://www.aecf.org/resources/theory-of-change/. Accessed 21 Feb. 2017.
Rogers, Patricia. Develop Programme Theory. Better Evaluation, betterevaluation.org/en/plan/define/develop_logic_model. Accessed 21 Feb. 2017.
Sharpe, Glynn. “A Review of Program Theory and Theory-Based Evaluations.” American International Journal of Contemporary Research, vol. 1, no. 3, 0ADAD, pp. 72–75., Accessed 20 Jan. 2017.
“Numicon Staircase.” About Numicon, Numicon New Zealand, Oxford, http://www.numicon.co.nz/177849/. Accessed 21 Feb. 2017.