NJCIE 2021, Vol. 5(3), 53–73

http://doi.org/10.7577/njcie.4155

The I’m Learning! intervention methodology for quality learning environments in developing country contexts

Cameron Ryall[1]

InformEd International

Lisa Zook

InformEd International

 

Copyright the authors

Received 16 November 2020; accepted 23 June 2021

Abstract

Within the context of the global learning crisis and calls to focus on the quality of education, one international organisation took a systematic and holistic approach to improve learning, wellbeing, and development in schools. Known as the Quality Learning Environment (QLE) Framework, it conceptualised the quality of the learning environment in schools with four guiding principles: emotional and psychological protection of learners, physical wellbeing of children, active learning processes, and close collaboration between school and parents/community. From 2013 – 2017, the framework was piloted in three countries of Cambodia, Uganda, and Zimbabwe to improve learning environments and outcomes while documenting the process, methods, and results of the QLE pilot in their country. The interventions in the three countries were closely tracked by a longitudinal research study. This paper explores the ambition of global and national research firms to carry out rigorous cross-country research alongside contextualized and evolving school interventions.

 

Keywords: international development; SDGs; global governance; school governance; quality education

The historical context that led to the I’m Learning! project design

In the post-2000 period guided by the Education for All (EFA) goals, the Millennium Development Goals (MDG), and now the Sustainable Development Goals (SDG), the global education community has developed and emphasised a focus on the quality of education. The six EFA goals, MDG Goal 2, and SDG Goal 4 encourage a broad range of interventions to improve the quality of education and to guide actions of organisations supporting access to and quality of education. Consistent with this direction, one of the world’s leading independent international development organisations, Save the Children, believes that education is a fundamental and enabling right and must be equitably realised for all children. The organization has existed for more than 100 years, starting up in Europe as a response to World War I, it now works in more than 100 countries to ensure every last child survives, learns, and is protected, and quality education is necessary to achieve this vision.

Save the Children’s Education Strategy for 2012-15, Moving ahead on education, had the following first strategic objective for basic education:

Schools and informal learning situations supported by Save the Children are Quality Learning Environments (QLEs) (Save the Children, 2012, p. 10).

This objective built upon a global movement for raising educational outcomes, where school enrolment increased significantly, but where the investment in human, material, and financial resources did not keep pace with the demand of many more children enrolled and attending school. Schools struggled to educate children with the provided classroom infrastructure and availability of trained teachers. There were simply insufficient resources to ensure a good learning environment. Furthermore, the children who were newly enrolled in school were often those who came from poor families, who possessed even fewer resources in their homes and communities to succeed. The combination of these factors among others resulted in a learning crisis, where many children left school without the necessary qualifications and skills to succeed in the real world, as demonstrated by a recent estimate that sixty percent of 6–14-year-old children are not achieving minimum proficiency levels in mathematics and literacy (UNESCO, 2017).

Moving forward in pursuit of all children learning from quality basic education, Save the Children sought to understand what circumstances helped promote children’s learning and wellbeing while taking a broad perspective on learning and development. Save the Children aimed to not only improve the cognitive learning outcomes from school, like numeracy and literacy but aimed to apply a rights-based holistic approach to the quality in education which sort to establish:

i)                the physical and psycho-social protection of children in school,

ii)              improved knowledge, skills, resources, and practices for school-based teaching and learning

iii)            a collaborative approach between parents, community members, and teachers to help all students succeed.

With this, Save the Children developed a global Quality Learning Environment (QLE) framework, encompassing guiding principles and indicators that sought to define a good quality learning environment, in this way setting guidelines for interventions. Also, the framework served as a monitoring tool to measure progress and results in efforts to improve basic education quality. Thus, the QLE Framework was the foundation for an assessment tool, a monitoring instrument, and a planning guide for Save the Children’s work to improve the quality of education. 

From 2013 to 2017, Save the Children implemented the pilot of I’m Learning!, which operationalized the QLE framework, in 3 countries: Cambodia, Uganda, and Zimbabwe. Funded by Norad from 2013 – 2017, the project aimed to improve the quality of the learning environments in target schools and strengthen enabling factors for the improvement of children’s learning and development. 

Overview of the I’m Learning! approach

The foundation for the I’m Learning! pilot project is the QLE framework, designed as a comprehensive tool that allows for systematic and nuanced measurements of the quality of the learning environment based on four guiding principles (GP):

GP1: Learning environments ensure children’s emotional and psycho-social protection

GP2: Learning environments are physically safe

GP3: Teachers encourage an active, child-centred learning process

GP4: Parents and communities actively support the children’s learning process

The pilot project explored and tested how Save the Children’s QLE framework and associated QLE data could be used to inform programme planning and management. Each of the three Country Offices developed its own contextualized interventions and activities to assist schools in achieving the QLE framework’s four guiding principles, creating a unique country-specific programming model. To provide further guidance for this, each of the four guiding principles has a set of indicators that determine whether the guiding principle has been achieved or not. The indicators are displayed in Table 1. The QLE framework is discussed more in the following section.

Simultaneously, each country office had a research initiative to explore, monitor, and document the correlation between improvements in the learning environment and children’s improved learning and development. 

By running the project in three separate contexts, the pilot of I’m Learning! aimed to develop and test a QLE-based programming approach that could be replicated and applied widely across Save the Children and partners to improve the quality of learning environments. In doing so the project challenged conventional international development approaches which are designed before project commencement, are often dictated by implementing organisations, funders, and donors, thereby constraining the responsiveness of interventions to the implementation context. Rather, the QLE-based programming approach embodied by I’m Learning! provided a framework that prompted locally generated solutions to achieving a quality learning environment. This is elaborated on in the following sections.

Table 1: Quality Learning Environment Guiding Principles and Sub-Indices

Guiding Principle 1: Emotional and Psychological Protection	Guiding Principle 2: Physical Protection	Guiding Principle 3: Active learning process, improved learning outcomes	Guiding Principle 4: Close collaboration between school & parents/community
1.1 Child Safeguarding Policy/Code of Conduct for learners and teachers ensuring wellbeing is in place	2.1 An area of space for learning exists that is safe for all learners	3.1 Teachers are present for their classes	4.1 School Management Committee / Parent Teacher Association includes representatives from a cross-section of the community
1.2 School has mechanism in place for receiving and responding to complaints	2.2 Safe drinking water is available for learners and staff	3.2 Teachers have specialized training and national qualifications where they exist	4.2 Teachers and parents collaborate on key issues affecting the children’s learning process
1.3 Learning environments are free of discrimination, violence, intimidation, bullying, and harassment	2.3 Adequate sanitation facilities are available for all learners	3.3 Teachers are provided continuous support to improve their practice in key areas specific to their role	4.3 Parents and communities are trained in how they can support the children’s learning processes
1.4 The teacher interacts with all learners in a positive and respectful manner regardless of their background	2.4 The play area is safe for all learners	3.4 Learning is supported through the use of relevant visual aids and other teaching materials
1.5 Teachers are trained in psychosocial support to detect cases of abuse or trauma among their students and provide support	2.5 Learning environments are accessible to the populations they serve in terms of hours, locations, and fees	3.5 Teachers develop, follow and adapt lesson plans to the needs and abilities of learners in their classes.
	2.6 Learners participate in health-promotion programs	3.6 Teachers use mother tongue of the majority of learners to further explain key concepts and support learning
	2.7 A minimum of health services is provided to learners (health-promotion programs, such as deworming, nutritional supplements, and health education)	3.7 Teachers ask individual questions and interact with the learners
	2.8 A School Disaster Management Plan, address disasters with the strongest likelihoods, is in place.	3.8 Teachers use some form of informal or formal learning assessment
		3.9 Teachers are trained on child rights and child protection
		3.10 Learners’ participation is ensured during development and implementation of teaching and learning activities
		3.11 Learners participate actively in decision making activities in their schools
		3.12 The learning environment encourages expression of child right and learners are knowledgeable about their rights.

Source: Christensen, 2016, p. 4

 

The Quality Learning Environment (QLE) Framework

The four guiding principles and twenty-eight indicators of the QLE framework are displayed in Table 1. The QLE framework was developed over two years by Save the Children International for the organization to have a common frame of reference for defining and conceptualizing quality education. The process for development included a literature review examining what works to achieve quality education in various contexts (Briseid & Osnes, 2013), formal review of the education work being carried out in similar organizations, and actively involving education professionals and practitioners across and beyond the organization, both in development and humanitarian contexts (Heijnen-Maathuis & Christensen, 2017).

In the work leading to the creation of the QLE framework, Save the Children discussed and explored the concept of quality education. At the forefront of these discussions was the awareness that most countries define quality education by test scores and cognitive competencies. Save the Children recognized that doing so only addresses one of the “pillars of education” as defined by the Delors Commission, which acknowledged the fundamental role of education to build knowledge and skills, but that “it is also – perhaps primarily – an exceptional means of bringing about personal development and building relationships among individuals, groups, and nations” (Delors, 1996, p. 10) depicting this vision in their four pillars of learning to know, learning to do, learning to be, and learning to live together.

The resulting QLE framework takes a holistic approach to education, recognizing the importance of the broader learning environment to foster education. Based on this framework, a monitoring instrument was designed, tested, and ultimately used in more than forty of Save the Children’s programming countries.

The QLE Framework for Monitoring

The QLE facilitates monitoring of a quality learning environment through the twenty-eight indicators displayed in Table 1 (Christensen, 2016). Monitoring is carried out using a 4-point rubric scale developed for each indicator labelled as exceeding expectation (4), meeting expectation (3), less than expected (2), much less than expected (1). Each indicator has a criterion for each level of achievement, enabling a scoring scale from 1 to 4. To illustrate this point, Annex 1 shows an example of a country contextualised version of the QLE framework.

The definitions related to the rubric as well as the data collection methodology are contextualized by each country, ensuring local relevance and use of culturally appropriate and understandable concepts. In all three pilot countries, this was done through a consultative process involving the Ministry of Education and local stakeholders. Based on the contextualized QLE framework, QLE data collection tools were developed to guide interviews, focus group discussions, and observations of the learning environment.

The scoring of the items within each indicator was informed by quantitative and qualitative evidence collected through a thorough assessment of the schools as well as a reference to government standards, review of existing procedures, and assessing the technical strength of the practice. In-depth and structured qualitative consultations with key stakeholders such as learners, parents, teachers, and school management were also conducted. These data are triangulated to provide a holistic picture of the learning environment and final scores on each sub-standard.

As such, the QLE framework and its accompanying assessment tool(s) aimed to provide a holistic perspective on the quality of education in the target schools, supporting these schools to identify strengths and gaps in the learning environment. To improve on QLE in target schools and achieve impact, school stakeholders were mobilized and supported with capacity development initiatives to assess, analyse, and find solutions to problems in the schools. Based upon the QLE assessment results, interventions were planned, designed, and implemented in an integrated manner across sub-themes and in conjunction with other sectors. Through a high level of participation and consultation across stakeholders at all levels and steps of the assessment, planning, implementation, and monitoring – the school communities, local education authorities, and other local government entities (e.g., health centres) felt ownership and responsibility for the project and achieving results in line with their own identified priorities and solutions.

The I’m Learning! pilot project explored and tested different and extended uses of the QLE framework beyond project monitoring, including project management and planning, and the use of the QLE data to inform these areas. The approach operationalised the QLE framework at the school level, creating a QLE-based programming approach to school improvement. The project emphasized locally driven solutions and strong local ownership through close collaboration with the government at national and local levels. 

The I’m Learning! pilot goal and objectives

The I’m Learning! pilot aimed to establish a QLE-based programme approach that could be replicated and taken to scale within Save the Children and externally by national governments and partners. The project sought to provide empirical evidence that holistic project design in education is effective in achieving an impact on children’s lives, and positive effect on their learning and development. The objectives of the pilot were to:

(1)            develop and document an effective and sustainable QLE-based programming approach for improving the quality of basic education which can be replicated and taken to scale. 

(2)            develop, test, and document specific programme interventions that are responding directly to the indicators in the QLE framework.

(3)            advocate with Ministries of Education about the importance of the quality of the learning environments and its impact on children’s learning and development, based on the experiences from the pilot project and findings from the I’m Learning! research.

It was recognized that to do so required robust evidence-based on the effectiveness of the approach and documentation of interventions, implementation models, costs associated with these interventions, and tools that have been developed to implement and monitor QLE-based interventions which are why the pilot programme undertook rigorous research as described in following sections.

Selection of pilot countries

Three pilot countries were selected to explore, monitor, and document the assumed positive interrelationship between a school’s learning environment and improved student learning outcomes during 2013-2017. According to Save the Children’s documentation, selection criteria were mainly based on four areas: 1) teachers must be competent and well-trained; 2) officials must have the capacity to support; 3) children, parents, and communities are essential participants in the educational process; and 4) schools must have the necessary physical elements, such as infrastructure and materials (University of Oslo, 2018). These criteria were established so there were similar starting points between countries to cater for the research design. However, in practice, the selection was done based on countries already receiving funding as part of SCNs Programme Agreement with Norad, and where the country office was willing and able to participate in a longitudinal pilot and research study. Furthermore, Save the Children country offices had to commit to programme implementation being done “in collaboration with communities, governments, and research institutions to ensure local ownership and sustainability and to strengthen civil society and the voice of children” (University of Oslo, 2018, p. 1).

Piloting the I’m Learning! project allowed Save the Children to additionally test the hypothesis that a broad global framework could be applied in three different contexts, leading to different solutions all working toward the same goal of achieving quality education. To adjudicate this hypothesis, it is necessary to interpret the results presented in the I’m Learning! research considering the education context within each country.

Cambodia

Education Context

Traditionally, the Cambodian education system targeted boys and was managed by the local government. The education system, therefore, served the community and acted as a social training centre. From the 1920s to 1953, during the French protectorate, the educational system was based on the French model integrating arithmetic, French, and geography into traditional subjects (Clayton, 1995).

After its independence from France in 1953, Cambodia saw a rapid increase in school attendance. Most of this progress however was undone under the Khmer Rouge regime from 1975 to 1979. During this time, most schools were closed or destroyed. Educated people and teachers were targeted by the Khmer Rouge and schools sought to indoctrinate children into the regime’s political and ideological beliefs. By end of the Khmer Rouge regime in 1979, the entire educational system was in shambles leaving Cambodia with 2,481 primary schools and 13,619 teachers to educate the 724,058 enrolled students (Sophoan, 1997). Rebuilding the educational system was further challenged by the occupation of Cambodia by Vietnam.

Since the new government elections in 1993, Cambodia has been working to rebuild its education system. To accomplish this, Cambodia has made the international Education for All (EFA) goals its national policy in the late 1990s. In 2007 the country also adopted the UNICEF designed Child-Friendly Schools (CFS) approach as a national policy. Both initiatives have a rights-based approach to education and the CFS takes a holistic approach to primary education. Continuing the focus, in the post-2015 SDG period, Cambodia has continued to emphasise inclusive and equitable quality education while also building the effectiveness of leadership and management at all levels in the education system to achieve these outcomes (MoEYS, 2019).

Programming in Cambodia

I’m Learning! cultivated a holistic approach to education, emphasizing the democratization of education and empowering teachers, communities, and learners. Save the Children selected 15 schools for programme interventions, five from each of the provinces of Kampong Cham, Kampong Chnang, and Kratie. Criteria for the school selection included enrolment (small, medium, large), location (rural, semi-rural), socio-economic status, and the number of grade levels which determined whether the school was complete (grades 1-6) or not complete (offers less than grades 1-6).

Uganda

Education Context

Uganda gained independence in 1962. At this time there were three main challenges that Uganda faced after independence, all of which had close ties to education (Ssekamwa, 1997). Firstly, was around the number of qualified people to work within government departments and companies. Under colonialism, few people were afforded the rights to school education, and fewer still qualified from university. Thus, there was a small and insufficient pool of qualified people to staff the bureaucracy. Secondly, the government needed to develop economic activities in the country to increased production so that government services could be carried out. The third challenge was the need to build Ugandans confidence in their ability to solve their own problems after colonial rule. To address these challenges, the government put significant resources into building and strengthening Uganda’s education system.

Unfortunately, in Northern Uganda, much of the progress in education was lost as a result of the Lord’s Resistance Army (LRA) and the ongoing guerrilla campaign that began in 1987. From 1987 to 2007, the LRA conflict led to the destruction of schools and disruption in education. Additionally, the LRA inflicted trauma on Ugandans, the effects of which are still being experienced by many today.

Starting in 2008, the LRA presence had weakened and efforts began to strengthen the education system once again. While there have been advancements through policy developments and rebuilding the infrastructure and capacity of the formal school system, including significant investments by non-governmental organizations, big disparities in education results between Northern Uganda and the rest of the country continue. The lingering effects of the conflict, resource disparities in schools, gender inequality, and changes in the social fabric have stagnated improvements to enrolment and learning outcomes (Atim et al., 2019; Spreen & Knapczyk, 2017).

Programming in Uganda

Save the Children’s I’m Learning! programming in Uganda was piloted in ten schools in the Gulu District, part of the Acholi sub-region in Northern Uganda. Of note, the Gulu District was one of the most heavily impacted areas of Uganda during the LRA campaign. The region also has the lowest learning achievement in Uganda.

Given the history of violence in northern Uganda, the I’m Learning! project in Gulu and Omoro emphasized psychosocial support by partnering with a local organization, THRIVE. The project worked with learners, teachers, parents, and community members to address ongoing mental health concerns and challenges. Additionally, the project focused on infrastructure and capacity building within the school and community environment.

Zimbabwe

Education Context

Before Zimbabwe’s independence in 1980, its education system was inequitable. At the time, most of the Black population only had access to primary education in poorly funded schools with inadequate educational resources, whose main purpose was equipping for labouring jobs in areas like agriculture and construction (Kanyongo, 2005). Furthermore, the colonial Rhodesian government made European education compulsory and universal and spent as much as 20 times more per European child than the African child (Shizha & Kariwo, 2011).

The first twenty years following independence ushered in a period of expansion and development in the education sector, under the government’s driving principle of ‘Growth with Equity’. Primary school was free to all and in the early 1990s, Zimbabwe’s education results were among the best in sub-Saharan Africa (Majgaard & Mingat, 2012).

In 2000, economic turmoil that resulted from a land redistribution effort led to a mass exodus of educated professionals from the country, including thousands of teachers, negatively impacting the education sector (Kanyongo, 2005). By 2013, Zimbabwe’s education system was a poor shadow of its former self.

Programming in Zimbabwe

Zimbabwe implemented I’m Learning! in two districts: Rushinga (Mashonaland Central province) and Matobo (Matebeleland South province), with each district having three different types of school (A = well-resourced and high enrolment, B=average size and resourcing; C=satellite school with low-resourcing and enrolment). Throughout the six schools, the project estimates a total of 3,591 students (1,752 girls) benefitted from the project. The I’m Learning! project in Zimbabwe was characterized by inclusive education, emphasizing infrastructure that provided access to children with disabilities. Similarly, Cambodia and Uganda, additionally emphasized child-centred teaching methodologies and child participation.

The I’m Learning! pilot research components

The I’m Learning! pilot research is best thought of in two sections:

(1)            Longitudinal Research

(2)            Programme Evaluation

At the start of the I’m Learning! pilot project, SCN contracted the Department of Education at the University of Oslo (UiO) to facilitate longitudinal research that was to

accompany the implementation of interventions in selected schools in the three countries in order to provide evidence for SCN in their decision-making concerning multiplication and up-scaling of the programme … [and] to test the underlying assumption of SCN that there is a positive relationship between learning environments (as understood in the QLE framework) and learning outcomes and child development (University of Oslo, 2018, p. v-vi).

The study was designed to run three school years in each country, being conducted from 2014-2017. In this role, UiO oversaw, coordinated, and compiled the longitudinal research carried out by local research organizations in each of the pilot countries. The local research teams included the Kampuchean Action for Primary Education (KAPE) in Cambodia, the University of Gulu in Uganda, and the University of Zimbabwe to complete the study in Zimbabwe. These teams tracked the project throughout three years, conducting primary and secondary data collection at regular points. Each country then produced a final research report, from which UiO produced a Final Comparative Research Report.

After the close of the I’m Learning! project, InformEd International was engaged to pair this longitudinal data with additional data gathered through site visits within each country, reflection exercises, key informant interviews, and project monitoring reports to provide additional perspective to the pilot project. This work produced a final programme report for each pilot project. Additionally, InformEd produced a Summative I’m Learning! Report which compiled the longitudinal research reports and the final programme reports to inform Save the Children programming moving forward.

Longitudinal Research

UiO worked closely with the local research institutions to develop longitudinal research designs within each country. There was an intentional effort to keep research and programme monitoring and evaluation activities separate to preserve the objectivity of the research. Programme monitoring and evaluation were kept separate in Uganda and Zimbabwe but the research partner in Cambodia had dual roles of implementing the longitudinal study as well as support the regular project monitoring.

The research component aimed to systematically study how and to what extent the quality learning environments as defined by the QLE framework noted before, contribute to improved learning outcomes and child development. It was also the objective of the research to find evidence if the I’m Learning! programme model can be a commonly accepted intervention approach to improve learning/child development outcomes. Finally, the research component intended to support evidence-based education policymaking and planning about the sustainability and scalability of the intervention programme (Ryall & Zook, 2018a). These research objectives were developed by SCN.

All three countries undertook a quasi-experimental design with mixed methods to examine the same set of research questions that were jointly developed between UiO and the local teams:

1)     How do learning environments change over time in intervention and comparison schools?

2)    How do pupils in intervention schools perform in terms of learning outcomes and child development compared to those in comparison schools?

3)     How does the psychosocial environment relate to learning outcomes and child development in intervention and comparison schools?

4)     How does the physical environment relate to learning outcomes and child development in intervention and comparison schools?

5)     How does active, child-centred learning relate to learning outcomes and child development in intervention and comparison schools?

6)     How do school-community and community participation relate to learning outcomes and child development in intervention and comparison schools?

(University of Oslo, 2018, p. 17)

While all research initiatives aimed to answer the same research questions, the roles and responsibilities of stakeholders, definitions of research constructs, data collection tools, sampling strategies, and timelines all differed across the three countries. These differences occurred for a few reasons, but with the foundation being four different research teams originating from four different institutions, with different motivations, and each with a differing level of capability. Complicating this arrangement were local dynamics which included the ease of accessibility to schools by researchers, the level of support from the Save the Children country office, and the differing levels of funding allocated to the research institution by the project. Moreover, these dynamics created conditions for potential validity issues with the research study, issues that were recognised by the research teams and navigated to the extent they could be by the coordinating researchers at the University of Oslo. The I’m Learning! Summative Report outlines the variations in each country, as synthesised below (Ryall & Zook, 2018a).

In Cambodia, four intervention and four comparison schools were selected by the research team using a non-random purposive sampling technique. The schools were all located in one of the three provinces the project operated in, with there being four different school typologies, “defined on a sliding scale based on standardized criteria identified in collaboration with Save the Children (kind of school (core, satellite or annex school), school size (enrolment), location (rural, non-rural), teacher background, student: teacher ratio, and socio-economic status)” (University of Oslo, 2018, p. 20). The team followed two cohorts of students for three years, the first cohort started in Grade 1 and the second cohort started in Grade 4. Save the Children’s monitoring and evaluation team designed and carried out the literacy and numeracy assessments (based on EGRA), the data of which was shared with KAPE for the research. KAPE carried out the life skills test. These assessments were carried out at the beginning and end of each academic year in both the intervention and comparison schools. The QLE assessment data was also carried out by Save the Children’s monitoring and evaluation team in both intervention and comparison schools. Additional qualitative data on learning environments and child development, informed by quantitative data, was carried out sporadically during the research period.

In Uganda, the study carried out purposive, convenience, and stratified random sampling techniques to select headteachers, teachers, parents, and learners in Grade 3 and Grade 5. The first round of data collection (carried out in 2014) was dropped from the final analysis due to very small sample sizes. Learners were resampled in 2015 and followed from 2015 to 2016. The literacy and numeracy tests were developed by the research team and were based on the UWEZO tool although the final report suggests significant deviations from the UWEZO tool processes. The life skills assessment was developed based on the World Health Organization (WHO), the United Nationals Children’s Fund (UNICEF), and the Uganda National Examination Board (UNEB). The assessment was considerably altered and updated throughout the research period. Focus group discussions and key informant interviews were carried out concurrently with quantitative data collection.

The University of Zimbabwe carried out life skills assessments, QLE assessments, and qualitative stakeholder interviews in all six intervention schools as well as six comparison schools. Two cohorts of children were tracked, the first cohort started in Grade 3 (exiting the research study in Grade 5) and the second cohort started in Grade 5 (exiting the research in Grade 7). The team also identified case study learners within the intervention schools which were followed more closely than others throughout the research. Qualitative data collection included lesson observation, school observation, and focus group discussions with pupils and teachers. Save the Children carried out student literacy and numeracy assessments, providing the data to the University of Zimbabwe for reporting purposes. The University provided annual reports on programme impact as well as a comprehensive research report upon programme completion.

Programme Evaluation

The programme evaluation was carried out retrospectively by an external international development monitoring and evaluation firm, InformEd International. The purpose of the programme evaluation was to summarise the results of the I’m Learning! project, draw conclusions, and provide consolidated recommendations to inform future Save the Children programming for quality education. To do so, the evaluation team produced a final programme report for each country and a summative I’m Learning! report. This was done by reviewing all existing data (longitudinal research, programme monitoring and evaluation data, financial records, and programme reports), carrying out action research and reflection workshops, key informant interviews, and site visits.

Figure 1: I’m Learning! Summative Report Development Process

Source: Ryall & Zook, 2018a, p. 14.

 

The programme evaluation aimed to answer the following key questions:

1.     Did the pilot achieve its goals/objectives? Why or why not?

2.     What best practices emerged from the three countries? Why?

3.     What did not work well across the three countries? Why?

4.     What areas need further investigation or development?

5.     What is recommended for Save the Children’s continued programme development toward a common approach for participatory school improvement linked to the Quality Learning Framework?

(Ryall & Zook, 2018a)

Desk Review

The programme evaluation commenced with a desk review, examining documentation covering global and country-specific information on the I’m learning! project. Table 2 outlines the specific sources of information contributing to the desk review.

Table 2: I'm Learning! Programme Evaluation Desk Review

Source: Ryall & Zook, 2018b, p. 6.

Key Stakeholder Interviews

Key Stakeholder Interviews were conducted in each of the three countries, as well as key informants of Save the Children Norway. Interviews were conducted in two methods: one-on-one interviews and workshop focus group discussions. Interview notes were recorded and used as a reference for the country and summative reports. Table 3 lists the stakeholders interviewed for the evaluation.

Workshop Design

A reflection workshop was held in each of the three countries, bringing together local stakeholders to reflect upon the pilot programme. Each of the three reflection workshops was conducted over two days. The purpose was to hear directly from a sample of project participants and gather their feedback on the I’m Learning! project. Also, workshop sessions were designed to gather information that was lacking in the available project documentation. Each workshop focused on the following topics:

·       Implementation Approach: How was QLE Framework contextualized for Cambodia/Uganda/ Zimbabwe; what tools were used to support implementation; what were the key interventions across the four QLE principles; How did Save the Children work with partners; How were the interventions monitored?

·       Summary of results: literacy; numeracy; life skills; enrolment; attendance; QLE indicators

·       Advocacy & Sustainability:  capacity to replicate I’m Learning! interventions; District engagement for improved policy implementation.

·       Lessons Learnt: Key learnings from project implementation, including successes as well as areas that did not work well

Throughout each of the workshops, participants were organized into groups for reflection activities. Groups were either be organized by stakeholder type (Headteachers/Teachers, SMC/PTA, MoE Representatives) or by mixed groups (with equal representation of each stakeholder), depending upon the objective of the activity.

Table 3 is a consolidated list of key informants interviewed through either one-on-one interviews or the reflection workshop. 

Table 3: I'm Learning! Programme Evaluation Key Informant Interview List

Source: Ryall & Zook, 2018b, p. 7.

Project Site Visits

A sample of 3-5 schools was visited in each country. During the visits, observation of the learning environment and classroom teaching and learning practices was conducted. Also, interviews of school principals, teachers, school management committee members, and students were conducted to better understand the programme and the impact within the schools. 

Discussion: The I’m Learning! programme approach and research initiative

The I’m Learning! programme was a commendable initiative from Save the Children, challenging common development approaches of top-down, NGO-mandated programming. It explored the role of the large international nonprofit in providing guidance and structure to stimulate local solutions to challenges. Furthermore, it established research partnerships across countries to generate evidence to inform future programming. The pilot programme generated numerous lessons which are explored throughout this Special Issue. For this chapter, the discussion focuses on the design of the programming approach and the research, with the subsequent chapters diving deeper into the nuances of findings.

Although not articulated in the initial programme design, the summative report of the I’m Learning! pilot, the programme approach was described in the following 4 phases:

Figure 2: Emerging I’m Learning! Phases

Source: Ryall & Zook, 2018c, p. 40.

Phase 1 consisted of each country reviewing the Quality Learning Environment sub-standards and aligning to government standards. By way of example, indicator 2.3 measured whether adequate toilet facilities were available to learners. The measurement of this indicator was contextualized to align with the government standard of the toilet to learner ratio. Based on this revised QLE framework, assessment tools were developed. Again, there was room for contextualization at this stage, with some programme staff developing guided questionnaires and surveys others facilitated self-reflection on the rubric itself.

Phase 2 then worked to apply the contextualized rubric and assessment tools through the annual QLE assessment. A best practice identified during this phase was to carry out the QLE assessment with all school stakeholders, an exercise that brought quieter voices in the school community to the forefront and encouraged stakeholder buy-in. Final ratings on the QLE framework were decided upon in committee among all stakeholders, facilitating discussion and dialogue while increasing transparency and accountability. Finally, results were posted in the school community for all to see.

Phase 3 focuses on acting upon the results identified in Phase 2. The self-assessment process carried out with the QLE framework identified strengths and weaknesses in the school, facilitating the prioritization of school improvement planning. School stakeholders took on responsibilities for implementing activities prioritized in plans.

While Phase 2 and 3 happened iteratively throughout the project, by the end of the project some schools were transitioning to Phase 4, which was evidenced in systems and strategies that the school had in place that sustained support for improving the quality learning environment among all stakeholders. One way to consider Phase 4 is to examine the requirement of ongoing Save the Children support. If a school has truly transitioned to Phase 4, then the practice of school self-assessment through the QLE framework and working together through school stakeholders to improve the learning environment within a school would not require the support of Save the Children.

These phases took place within each school, with school improvement designs, interventions, and initiatives emerging from the I’m Learning! process. This ultimately resulted in I’m Learning! looking very different in each school and each country. The I’m Learning! pilot is thus an example of the ongoing effort to decolonize development work by building the voice of communities and empowering local solutions.

The pilot project implemented rigorous longitudinal research alongside the pilot projects. While this provided useful insights to the project implementation and impact, numerous challenges resulted from a project being designed and adapted alongside a rigorous research framework. There was therefore a misalignment between a bottom-up, contextualized project design and a rigid research design that did not effectively capture some of the emergent project developments.

While it is recognized that a defining characteristic and strength of I’m Learning! is the fact that the QLE framework allows for country-specific contextualisation, interventions, and innovations, it also poses a significant challenge for project model development. The project struggled to provide enough structure that ensures rigour and fidelity of project implementation, while also allowing the project to be tailored to the needs of each country.

Thus, the I’m Learning! programme approach and longitudinal research provide a unique opportunity to explore the friction between rigorous research methods, international research partnerships, and the reality of responsive emergent international development programming.

References

Atim, T., Mazurana, D., & Marshak, A. (2019). School’s out: why northern Uganda’s girls and boys are not getting an education and what to do about it. Overseas Development Institute. https://securelivelihoods.org/wp-content/uploads/201908_SLRC-Uganda-education-working-paper_web.pdf

Briseid, O., & Osnes J. (2013). Quality and Learning Outcomes in Education. What works? [Save the Children internal paper]

Christensen, S. D. (2016). Save the Children’s quality learning environment framework, presentation to Save the Children’s inclusive education learning event, Bangkok, December 2016. https://resourcecentre.savethechildren.net/node/10305/pdf/qle_ppt_sine_christensen_iele_bangkok_dec._2016.pdf

Clayton, T. (1995). Restriction or resistance? French colonial education development in Cambodia. Education Policy Analysis, 3(19). https://doi.org/10.14507/epaa.v3n19.1995

Delors, J. (1996). Learning: the treasure within. Report to UNESCO of the International Commission on Education for the Twenty-first Century. UNESCO Publishing. https://www.gcedclearinghouse.org/sites/default/files/resources/%5BENG%5D%20Learning_0.pdf

Heijnen-Maathuis, E., & Christensen, S. D. (2017). Save the Children’s Quality Learning Environment (QLE) framework: What have we learned and how will this inform our future work. [Save the Children internal paper]

Majgaard, K., & Mingat, A. (2012). Education in Sub-Saharan Africa: A Comparative Analysis. World Bank. https://doi.org/10.1596/978-0-8213-8889-1

Kanyongo, G. (2005). Zimbabwe’s public education system reforms: Successes and challenges. International Education Journal, 6(1), 65-74.

MoEYS (2019). Education strategic plan 2019 – 2023, Ministry of Education, Youth & Sport, Cambodia. https://www.moeys.gov.kh/index.php/en/policies-and-strategies/3206.html#.X67h7GhKiHs

Ryall, C., & Zook, L. (2018a). I’m learning summative report, Save the Children. https://www.norad.no/om-bistand/publikasjon/ngo-evaluations/2018/im-learning-summative-report/

Ryall, C., & Zook, L. (2018b). I’m learning inception report, Save the Children. [Save the Children internal paper]

Ryall, C., & Zook, L. (2018c). I’m learning summative report presentation, Save the Children. [Save the Children internal paper]

Save the Children (2012). Moving ahead on education: A focused strategy for achieving our education goals 2012 – 2015. Save the Children. https://resourcecentre.savethechildren.net/sites/default/files/documents/6526.pdf

Save the Children (2013). Quality learning environment monitoring form for basic education programs. [Save the Children internal paper]

Shizha, E., & Kariwo, M. T. (2011). Education and Development in Zimbabwe. Sense. http://dx.doi.org/10.1007/978-94-6091-606-9

Sophoan, P. (1997). Education destruction and reconstruction in Cambodia. In T. Sohbi (Ed.), Final report and case studies of the workshop on educational destruction and reconstruction in disrupted societies, 15-17 May 1997, Geneva, Switzerland. International Bureau of Education & The University of Geneva (pp. 43-49). http://www.ibe.unesco.org/fileadmin/user_upload/archive/Publications/free_publications/Tawil5e.pdf

Spreen, C., & Knapczyk, J. J. (2017). Measuring quality beyond test scores: the impact of regional context on curriculum implementation in northern Uganda. Forum for International Research in Education, 4(1), 1-31. http://dx.doi.org/10.18275/fire201704011110

Ssekamwa, J. C. (1997). History and development of education in Uganda. Fountain Publishers. http://www.nzdl.org/gsdlmod?a=p&p=home&l=en&w=utf-8

UNESCO (2017). More than one-half of children and adolescents are not learning worldwide, Fact Sheet No. 46, September 2017. United Nations Education, Science, and Cultural Organisation Institute of Statistics. http://uis.unesco.org/sites/default/files/documents/fs46-more-than-half-children-not-learning-en-2017.pdf

University of Oslo (2018). “I’m Learning!” longitudinal study: a synthesis report. Department of Education, University of Oslo.

 

Annex 1: Example of Save the Children’s QLE assessment tool

Guiding Principle 4: In Save the Children-supported education programs, parents and local communities are actively involved in planning, decision-making and action to improve education.
Indicators Ratings		Not applicable (NA)	1 = Not at all achieved	2= Almost achieved	3 = Achieved		4 = Exceeded Project Objectives or Expectations
4.1 Teachers and parents collaborate on key issues affecting learners, such as sickness, absenteeism, performance or discipline. (Note: requires consultation with teachers).   Item Score: 1       2       3       4	NA		-Teachers and parents do not communicate about key issues -	-Teachers are able to schedule at least 1 meeting during school year with parents on key issues	-Teachers and parents are able to speak consistently about key issues	-Consistent evidence of successes due to collaboration between parents and teachers (e.g. refurbished classrooms, school feeding programs, reduced absenteeism of students and teachers)
4.2 Parents and community members are provided with training and capacity building to support learner development and education.     Item Score: 1       2       3       4	NA		-No report of training or capacity building activities for parents and community members	-School has been hosting some training or capacity building activities for parents and community members, but not regularly -Parents and community members are rarely informed or rarely attend capacity building activities hosted by school	-Parents and community members regularly attend trainings/capacity building activities -Moderate to high level of participation by parents and community members in workshops -Capacity building activities hosted by the school meet the needs of parents and community members	-Parents and community members work with school officials and teachers to plan capacity building activities -Consistent and high level of participation by parents and community members in workshops and capacity building activities -Parents are aware and supportive of upholding their children’s rights
4.3 A group such as a Parent-Teacher Association (PTA), School Management Committee (SMC), Community Education Committee (CEC), or School Development Society (SDS) includes representatives and a cross-section of the community (e.g. by age, employment, gender, disability). (Note: requires access to and review of attendance records to assess the composition of the committee).    Item Score: 1       2       3       4	NA		-PTA/ SMC/ CEC/SDS does not exist	-PTA, SMC, CEC, or SDS formed but not active - OR the existing PTA, SMC, CEC, or SDS includes learner representatives, but no evidence learners have any real influence.  -Parents and community members are rarely involved in decision making activities and school events	- PTA, SMC, CEC, or SDS is formed and actively involved in 50% or more of decision-making processes, problem solving and planning for school events - Learners are actively involved in 50% or more of decision making and planning in PTA, SMC, CEC, or SDS (e.g. group finds ways to maximize involvement of learners aligned with age and ability) -Parents and community members asked to participate in 50% or more of decision-making activities and school events (either individually or formally through PTA, SMC, CEC, or SDS)	- PTA, SMC, CEC, or SDS play an active role in school management and as advocates for the learner’s education within the school and in the broader community/ local government - Learners regularly suggest ideas for all school events and play significant role in most decision-making activities -Evidence of successful collaboration within PTA, SMC, CEC, or SDS groups advocating for developments that will meet the needs of all learners (e.g. allocation of resources for school and children’s activities that support improvement of learning environment and learner performance/ outcomes)
______ Check here if 50% or more of indicators for Guiding Principle 4 were given ratings of 3 or 4.
Overall Rating: Has this learning environment “achieved” or “exceeded” all four guiding principles? [Check yes if 50% or more of all indicators for all 4 principles were given ratings of 3 or 4.] Yes                  No

Source: Save the Children (2013, pp. 13-14).