Kas M.J. Cascant i Sempere

Meeting the Literacy Evidence Gap Challenges: The International Reflect Evaluation Framework

 

Reflect and the Need for Evaluation

Reflect is an approach to adult learning that aims to link literacy, empowerment and the achievement of rights. Reflect has won 4 UN Literacy Prizes in 2003, 2005, 2007 and 2008. It is presently used by 500 organisations in around 70 countries. The huge diversity of the approach has led to the need of systematising evaluation efforts. Together with this need, there was a strategic intention of pioneering participatory evaluation in the field of adult education.

Evaluating Literacy in Reflect

In response, Reflect practitioners internationally have participated in the development of a new evaluation framework. To initiate the process, ActionAid developed a draft framework that was piloted in South Africa in May 2007. In October 2007, a workshop was held with UK-based research students to critique the early framework. This was followed by a one-week workshop co-hosted by DVV International, ActionAid and SARN (South Africa Reflect Network) in Cape Town in November 2007, where 39 Reflect practitioners from 20 countries convened to engage with and critique the initial framework. In April 2008 SARN took up coordination of the initiative and appointed a Reflect Evaluation Framework Coordinator.

As part of this evolving process, for six weeks during June and July 2008, 88 Reflect practitioners from 42 countries joined an online network and collaborated across four languages (English, French, Portuguese and Spanish) to discuss the evaluation of Reflect and to contribute ideas. A summary document of the online dialogue was then shared in English, French and Spanish with Reflect practitioners who were not available to take part.

 

The Cape Town workshop, 2007 Source: Cascant i Sempere 

The framework was also informed by analysis of Reflect evaluations implemented in 2008 as well as two reviews of past Reflect evaluations carried out by Riddell (2001) and Duffy & Fransman (2008) on behalf of ActionAid. A “zero edition” of the framework (to be piloted and shared at CONFINTEA VI) is the latest output of this two-year collaborative process.

Evaluating Literacy in Reflect: the Framework

The framework provides a broad toolkit for designing and conducting an evaluation which might play a summative (assessing the programme), formative (informing better practice) and pedagogic role (since literacy and communication practices are often developed through the very process of participating in the evaluation process). It is also rooted in the principles of participation which suggest that meaningful and democratic involvement in an evaluation can enhance the ownership of programmes by participants, promoting sustainability and transparency. Individuals are able to reflect on their own learning experience at the same time as the circle as a whole reflects on the collective learning experience. The expectations of participants are therefore just as significant as the programme objectives of implementers and the broader social goals of civil society organisations, governments and donors; all of which should be taken into account by the evaluation.

After carefully explaining the concepts of literacy and evaluation and outlining the principles in which the framework is grounded, the framework then takes the form of a toolkit organised into 16 sections with 80 tools:

  1. Planning your evaluation
  2. Baseline data and methods
  3. Literacy levels
  4. Literacy practices for change
  5. The literacy environment
  6. Supporting organisations
  7. Funding and sustainability
  8. Circles and circle members
  9. Facilitators and facilitation
  10. Trainers, training and support
  11. Learning and materials
  12. Writing your evaluation report
  13. Networking and communication
  14. Documentation and advocacy

Crucially, the framework is not intended to be used as a blueprint but rather a set of guidelines and suggestions which should be adapted to suit a particular concept. A workshop to help pilot the framework in three countries is planned for later in the year.

The evaluation framework as a good practice: the process

The process of creating an evaluation framework can be considered to be a good practice for the following:

  • The tools were gathered globally and sent by the field experience in many countries working with Reflect
  • The participation in process of creation included:
    – A workshop in London with phd students
    – A workshop in Cape Town with 39 practitioners from Asia and Africa
    – An on-line discussion with 42 countries, 88 people, 4 continents and 4 languages
    – The integration of previous Reflect reviews and evaluations

The on-line discussion on “Evaluating Reflect”, 2008  

The Evaluation Framework as a Good Practice: the Content

In the same line, the content of the framework can be considered to be a good practice for the following:

  • Its focus on literacy as the foundation of education and its usefulness for other  organisations.
  • Its new conception of literacy, being:
    – More than something written (evaluating oral literacies, visual literacies…)
    – More than just “skills” (see particularly chapter 5 of the framework on literacy for
       empowerment & change)
    – More than just an individual thing (see particularly chapter 6 of the frame- –work on the literacy environment)
  • Its challenge of keeping the Reflect spirit within evaluation – for example, with visual tools for evaluation, evaluating power, evaluating the creation of materials (no primers), as well as its accessibility (easy-to-use, in four languages).
  • Its efforts in pioneering the international development and coordination of participatory evaluation – for people to be their own evaluators together with external ones (finding a balance) and learn from it.
  • Its integration of gender and age (young & adults), namely chapters of data • (chapter 3), power (chapter 5 & 6) and chapters of circles, facilitators and trainers (chapter 10, 11 and 12).
  • Its efforts to coordinate with other present efforts in adult education: The need to • evaluate literacy and the need to reinforce literacy for empowerment are in line with UNESCO’s effort of: LAMP (Literacy Assessment & Monitoring Programme) and LIFE (Literacy Initiative for Empowerment). In this sense, tool 18 of chapter 4 is entirely based on LAMP and chapter 5 to literacy for empowerment.

 

Why is the Reflect Evaluation Framework Innovative?

The Reflect Evaluation framework can be considered innovative for the following:

 

The Cape Town workshop, 2007 Source: Cascant i Sempere 

1. It represents a pioneering effort to work in four languages simultaneously rather than working first in English and then translating it into other languages for non-English speaking participants to comment. This allowed, for instance for Anglophone, Francophone and Lusophone African countries to work together in the same effort as well as country representatives from Africa, Europe, Latin America and Asia, all at the same time.

2. It represents a pioneering effort to keep an international literacy community together (CIRAC) with an on-line resource center and constant information flows amongst them as well as its aim to link with other networks such as the African Platform, ICAE, etc…

At what stage now?

The next steps of the framework project are as follows:

1. Dissemination, mainly through a workshop in the official programme at CONFINTEA VI in Dec. 2009
2. Training on the evaluation framework
3. Piloting and refinement of the framework

The partnership is presently seeking for more funding in order to achieve these three objectives.

Recommendations on Financing for Evaluation in Adult Education

 


The fish tool, South Africa. Source: SARN

Recommendation 1: To foster and finance a culture of evaluation learning within youth and adult education and literacy organisations.
In particular the following two aspects: the correct gathering of baseline data at the beginning of programmes and the sharing of evaluation results and learnings at the end of them. This includes the devolution of results from external consultants.

Recommendation 2: To support programmes that make an effort to go beyond external-based, donor-led evaluations as the only way of evaluating and that integrate a mixed / multiple evaluation format, through the inclusion of other evaluation methods such as peer reviews, on-going evaluations, internal evaluations and self-evaluations.

                              Reflect Evaluation in El Salvador, 2008

 

Recommendation 3: To promote innovative research on new and/or  improved evaluation methods tailored for literacy and adult education. Methods could range from evaluating literacy levels as a continuum rather than a line to be passed, or for better evaluating concepts such as the literacy environment and the empowerment side of literacy and education.

 

The motivation tree tool, Angola.  Source: dvv / AAEA

Previous recommendations are crucial because it is not a matter of financing more literacy and education programmes regardless their character istics but rather of financing those programmes that have evident developmental impact and those entities with the will to learn from their past actions, to develop the ownership of their evaluation process and that use evaluations as a literacy/education method in itself.

                                                                      The self evaluation tool, DRC.  Source: CEREBA/GOAL

 

 

Editions

Article search

You can search for articles in our article index (sorted by authors, issues, year, regions and countries). It also provides a full text search.

Subscribe

The journal Adult Education and Development is distributed free of charge in English, French and Spanish. If you wish to receive the journal, please subscribe here.