S6050 Tove Nagel. A participatory, illuminative evaluation of the Quality Education Project (QEP), in Gonder Zone, Amhara region, Ethiopia. Sep 2002 – May 2005. Paper at Netreed’s conference at Beitostølen, Dec. 5-7, 2005. Save the Children Norway www.fflh.no/dialog/S6050nagel.html

Summary of an illuminative, participatory evaluation of the Quality Education Project (QEP) in the Amhara region, Ethiopia. By Dr. Tove Nagel (Nov. 2005)

Through introduction of action research the project aims at empowering and raising the consciousness of teacher educators (25), university lecturers (10) and teachers (280), so that they investigate and find solutions to the serious quality problems related to students’ learning.

The evaluation was designed to further empower, enlighten and assist in developing the project. It consists of inputs by ; teachers, who presented their action research write-ups to a big audience of teachers, lecturers and MoE staff, followed by comments and discussions. Their presentations were evaluated by Save the Children education staff from Nepal, Zambia, Mozambique and lecturers from the Universities of Zimbabwe and Bahir Dar. Teacher educators’ view of the project have been included.

Interviews and observations of teachers were carried out by the project leader and teacher educators. The purpose was to identify and describe changes in teachers’ thinking and behaviour and discuss these with them as a part of further illumination.

It was found that teachers have changed to become confident, more aware and motivated for their work. They also acknowledge their own role in children’s learning. Most participants say that QEP has meant much to them, both personally and professionally, as it has opened their eyes to look at their profession in a new way. Asked if the project ceases tomorrow, most of them stated that they would continue doing action research into their practice.

Before we were teaching subjects, now we teach children….”

A participatory, illuminative evaluation of the Quality Education Project (QEP),

in Gonder Zone, Amhara region, Ethiopia

Sep 2002 – May 2005

Paper at Netreed’s conference at Beitostølen, Dec. 5-7, 2005

by

Dr. Tove Nagel,

special adviser to Save the Children Norway

Introduction

The objective of this paper is to present and problematize the recent evaluation of an educational development programme, the Quality Education Project, QEP, as an example of an illuminative, participatory approach to education evaluation. The special feature of the evaluation is that the epistemological and methodological thinking behind the evaluation has developed as an integral part of the project itself and been inspired by the nature of the programme. The evaluation has become multi-faceted and consists of many parts and the opinions of many groups of people, insider participants as well as externals to the proramme.

The major concern of this paper is not to engage itself with the quality problems in education or to describe the project in detail. The purpose of the paper is to analyze and reflect upon what we have experienced through evaluating the project and to show how the evaluation thinking and practice have developed as an integral part of the project, in line with its theoretical foundation as well as value base. Some issues present themselves for further discussion: such as the relationship between practice and research in evaluation, the legitimacy of insiders’ participation in evaluation, the purpose of the evaluation and how it can be used, empowerment, and finally change and reflection of participants as a result of the evaluation.

It has been our experience that evaluations carried out in a participatory manner have assisted in raising issues that would otherwise not have been captured, f.ex. elucidating what changes there have been in teachers’ conception of the teaching/learning process as well as changes in their class-room behaviour. This has been achieved by interviews of teachers and their pupils by the teacher educators and SCN QEP staff, including the author if this paper, who is also the international leader of the project

Before going into the discussion of the evaluation it is necessary to start by a short presentation of the project so as to provide background to the discussion of evaluation of this programme.

The background for QEP

The objective of the QEP project for Save the Children Norway (SCN), is to develop and pilot a model for quality improvement of education and try it out in different contexts across a number of countries. The model seeks to demonstrate that by empowering teachers and lecturers to investigate, reflect upon and change their own practices in schools and teaching institutions, the learning of children will improve. The Quality Education Project (QEP) was launched in October 2002 in Ethiopia by Save the Children Norway as a response to the serious teaching/learning problems encountered in many countries, including Ethiopia.

The quality problems in education and issues related to children’s learning are found all over the world, but most outspoken in developing countries, where it has been demonstrated by national evaluations that children perform at alarmingly low levels across the subjects (SACMEQ 1998, 2000, 2005). The SACMEQ findings from seven African countries are gradually being complemented by similar findings also from other parts of the world (EFA 2004 Global monitoring report).

Save the Children Norway has education programmes in its country programmes world wide. Education is the biggest thematic area of the organization, and in 2001 it was decided to have a thematic evaluation of education activities in 9 of the 16 countries where SCN has programmes. As a result of the evaluation it was decided to focus the organisation’s education work on educational quality. QEP is the organization’s major response to address the quality issue in education.

It was also decided to focus the quality question to issues related to children’s learning.

How the quality problems express themselves is not the place to contemplate here. It is referred to the original documents prepared at the initiation of the project. It can be found at the QEP project web-site at: http://www.rb2.no/qep under the heading Quality Education Project, Ethiopia.

An international conference on Quality of Education was held in Oslo to advocate with authorities in the countries where SCN has programmes, about the seriousness of the situation for children’s learning. Among the 120 participants at the conference, 20 high ranking officials from Ministries of education attended. As a result, QEP was introduced in Mozambique at the request of the MoE. Papers from this conference are found on the same web-site and illustrate the seriousness of the situation for children’s learning in many countries.

What is the Quality Education Project ?

In the original project document it is stated: ”the long-term objective of the Quality Education Project (QEP) is to investigate and contribute to education quality development in order to improve children’s situation in school and their opportunity for learning “.

QEP is a development approach designed by the Save the Children Norway special adviser in education and initiated and implemented in cooperation with partners in Ministries of Education. The major focus is on capacity building in the area of action research and qualitative research methods. QEP aims at bringing about sustainable change through nurturing participants’ empowerment through critical reflection. Using action research as a vehicle to bring about change in schools is not a new approach. It falls in line with several traditions, f.i “the teacher as researcher tradition”, “the reflective teacher tradition” and the broad family of Participatory Action Research (PAR) approaches that have been used not only in education, but in other professions and areas as well, f.i. PLA in agricultural extension and development work.

What does QEP do?

The approach chosen to achieve change in the awareness about the quality problems in schools, is through introducing participants to action research. Since qualitative research is fairly unknown to most professionals in the countries where the project is presently working, it has been necessary to start by teaching basic introductory knowledge in qualitative research, also presenting its philosophical foundations and most common methods. It has been seen as important to combine theory and practice, as this is a recurring problem in professional training, for example in teacher education. This has been done by always taking participants to schools and communities to practice the research methods they have been taught.

The project is inspired by several theoretical traditions. The most important are: Thinking/reflection (Dewey, 1933, Carr, Kemmis,1986), Action/practice + reflection, reflective teaching (Schön , 1983, Liston and Zeichner, 1993), Participatory inquiry/research (McTaggart, 1997) Communication and critical awareness (Habermas, 1971, 1986 etc).

QEP in Ethiopia offers training to teacher educators in qualitative research methods and action research in order to also have an effect on the education system per se in a long term perspective. It was thought that just concentrating on teachers would not change the system. Teacher educators practice the knowledge acquired through the training in systematic and regular investigations in primary schools. Participants’ reflection is prompted by the problems they discover and which they have to investigate and describe. They are encouraged to explore and define the problems encountered in the practical class-room and school reality and in their own teaching. After a period of training of teacher educators, teachers from primary schools are introduced to the same topics, however in a simplified version. They are followed up by the teacher educators, who become their mentors.

The teachers, in turn, seek to identify and investigate quality problems and to find adequate solutions to the problems according to the circular model of action research:

· Identify problem (reconnaisance)

· Plan the research

· Collect data and analyse them

· Research a solution (change)

· Reflect on what you see (observe effect)

· Draw a conclusion

· Revisit problem etc. etc (Kemmis and McTaggart, 1988)

It is a central feature of the project that participants themselves have to discover, uncover and acknowledge their own roles in relation to children’s learning problems. Problems therefore have to be identified by the participants themselves. The fundamental thinking behind the QEP project is leaning towards a constructivist epistemology. Participants themselves investigate and construct knowledge about their professional activities and the context in which they act. Through increased awareness and knowledge they are also empowered.

Within education, and particularly in developing countries it has been a tradition to see the teacher as the one who “has all the answers”, no matter at what level. It is also a tradition that higher levels have the authority and have the right to prescribe what others should do. It is a challenge for many of the project participants not to “tell” (what is wrong) and prescribe (solutions) in all the countries where the project has been introduced. This has particularly been difficult for some national staff, who come from autocratic organizational traditions and sometimes display a moralistic approach to the problems they see in schools. It is acknowledged in wide circles today that just giving teachers short courses to address the problems in schools do not bring about lasting change. This is especially so when the problem is not acknowledged as a problem by the teacher. Therefore, another and more empowering method was needed. The “teacher as researcher tradition” is an approach based on empowering teachers to bring about change in schools. It has been tried out in many countries in the West, but to a very little extent in developing countries.

The organization of QEP

In 2002 the Quality Education Project” (QEP) was first launched in Gonder in Ethiopia, where the organization had been working since 1995 in education. The organizational model chosen for QEP in Ethiopia was to first work with 11 teacher college lecturers, and then gradually introduce 20 teachers in 5 schools into the project. This was done after 1 ½ years. After a year all the teachers in the same schools as the first 20 teachers were invited into the project. It turned out they were very eager, even admitting to be jealous of their colleagues who had initially been selected to be the first to be trained. The same model is not necessarily adopted in the 3 other countries where the project is being implemented.

A partnership was set up between Gonder College of Education and Save the Children Norway-Ethiopia is signed, and the Ministry of education is following the project with great interest

The lecturers have been working with these teachers and at times the teachers have been invited to the so-called “Interventions”, which are professional seminars conducted by the education adviser from Oslo for the participants. In general there have been 4 interventions per year in Ethiopia.

Over time the number of resource persons in the college has decreased due to transfers and scholarships. It was decided in Dec 2004 to invite a new group of college lecturers to be trained in qualitative research methods and action research at the college.

At present the project in Ethiopia has 280 teachers, 25 lecturers (1^st and 2^nd batch) and 10 new participants, lecturers from the regional university of Bahir Dar. The University lecturers were invited to participate in the evaluation of the QEP teachers in may 2005. As a result, they pronounced that they also wanted to become part of the QEP project, and are now being trained to become resource persons for the scaling up of the project in the future. We have understood that it is important to train a number of people at tertiary level for the sustainability of the project. At the beginning of the project, in 2002, attempts were made to interest both the University of Addis Abeba and Bahir Dar to participate in the project, but without any luck. Now it seems the University of Bahir Dar is coming on board.

Other countries have chosen other ways of organizing their QEP project. The objective with the pilot is to scale up the project to 20 schools, 500 teachers and 5 ABECS (Alternative Basic Education for Children out of School) centers to demonstrate feasibility of scale-up and sustainability by training support staff in college and university by 2009.

Heading the project is an international QEP core group which was established in 2004. it consistr of the QEP coordinators from the 4 countries as well as the project leader from Oslo.

QEP is an international, participatory project

In 2003 QEP was introduced in Zambia, 2004 to Mozambique and in 2005 to Zimbabwe. In all countries there is a close cooperation with the Ministries of Education. The organizational models are slightly different.

In the model adapted SCN education staff from other countries are regularly invited to follow the project in other countries to learn and take inspiration back to their own programmes, as can be seen in the illustration above. In Mozambique we have invited Spanish and Portuguese speaking staff and partners from MoEs from other countries, like Guatemala and Angola.

It is only in Ethiopia, Zambia, Mozambique and Zimbabwe that the QEP model with introduction of action research to teachers and teacher educators is beeing implemented. In all the other countries, however, the issue of quality of education has been addressed in different ways, often inspired by QEP involvement.

Evaluation methods developed as an integral part of the project

It was felt by the project management that evaluation of the project should not break fundamentally with the nature of the project. Since the project aims at awareness-raising and empowerment of the practitioners it was seen as important also to have a participatory approach to evaluation of the project.

It has been a conscious decision that the evaluation methods, in order to be useful, need to be seen in line with the broader objectives and practices of the project. Evaluation should be formative and contribute to the project development. Its main target should be to further illuminate all participants as to whether the project is on track and what works and doesn’t work as well as to raise practical and theoretical issues of concern. The decision to make it a participatory illuminative evaluation was both an ethical, but even more so a practical question, since it is teachers and teacher educators who actually run the project and eventually also need both to understand and change it, according to the findings of the evaluation and the needs of the project.

When the question of evaluation of the project came up the first time from the management level in SCN- Ethiopia, it was therefore argued for a participatory, illuminative approach to evaluation. Traditionally, an evaluation is seen as serious only in so far as it is external and “so-called” impartial. It was seen as artificial to have an external person with no previous knowledge of the project’s life and process development to come and decide what was good and bad in the project. Instead it was sought to find ways in which to carry out an evaluation so that the evaluation should become truly formative and benefit the development of the project. The question of impartiality on the part of the evaluator is a contested issue in evaluation theory, but not as controversial as it used to be (Shaw, 1999, p. 20 ff)

In evaluation theory there have for a number of years been discussions whether goal-oriented evaluations are actually the most useful way of seeing evaluation or whether it is rather an outcome-based evaluation that is the most useful way of approaching evaluations. (Scriven, 1996. Stake,1994,1995, Campell,1964,1996, Chronbach et al, 1980). One would think that at least it should be decided what is the purpose of an evaluation.

The QEP core group agreed on a number of issues they though necessary to address in the evaluation, but emphasized that the participants should not in any way feel restricted by this list. It was discussed in the group if it was right to influence the topics of the evaluation in such a way. But as it was also a certain fear in the group that the evaluation would not address the questions that the international core group members thought were necessary to elucidate at this stage in the project, the evaluation could run the danger of getting astray with participants taking the lead in what they wanted to present and how they chose to present it. The list of topics that the QEP core group saw as important to address was given to the external guests, SCN staff from other countries and to the University lecturers from Bahir Dar who were new to the project.

During the QEP’s life time evaluations have been carried out continuously as a normal part of project life. At the beginning participants were asked to make SWOT (Strengths, Weaknesses, Opportunities and Threats) analyses. This was done in smaller groups who presented their view in plenary, followed up by discussions and decisions about how to handle the problems. Weak points and threats were addressed in planning sessions and built into the further development of the project. Such planning is done at the end of every intervention in the project. One such example was the fact that a written partnership contract was missing at the beginning of the project between the SCN local office and the college. This was taken up by the lecturers, who felt that their work in the project was not acknowledged by all parts of the college management as well as higher levels in the MoE at regional level. This made cooperation with the college administration difficult and at times imperilled the project, as lecturers could at all times be sent to do other tasks, and plans to go to schools and work with teachers in the QEP project were jeopardized. When the contract was signed, it was easier for the project participants to carry out their tasks with reference to the contract. SCN staff has also carried out advocacy about the project and the need to address children’s learning in quality education with the regional MoE level and invited them to interventions with the University lecturers in Bahir Dar.

In May 2004 a first minor evaluation of QEP was carried out. It was a midway evaluation of the first phase. There was a discussion and feed-back of results with SCN management in Ethiopia who received the evaluation report. It seems, however, still to be the case that the interest from higher levels in administration are more related to formal, legitimative purposes of evaluation than in a real interest in how the project is developing. However, it has been easy to get approval for other ways of carrying out the evaluations, once arguments have been put forward.

In the half-way evaluation the methods chosen were focus group interview with the college lecturers and SCN support staff. This was done the first time by the project leader carrying out Focus Group Interview (FGI) with the participants. In this way two objectives were achieved; one was to teach and demonstrate FGI and the other was to show how information could be obtained by getting a group to discuss, and how assessment of the status of the project could be carried out by using qualitative research methods. These interviews were tape-recorded and summarized in writing to the participants to agree to whether this was actually what they said, to disagree and/or comment upon.

Organization of the evaluation

The first formal evaluation of QEP was carried out in May 2005 and in September /October. At this stage, the Quality Education Project (QEP) had been working in Ethiopia for 3 years, since Sep 2002. Initially 11 lecturers from Gonder College of Teacher Education comprised the main target group, but only 7 are left due to transfers and scholarships. None have quitted because they did not want to be part of the project.

In this evaluation of the first phase of the project, the major target groups were the “old QEP teachers” and the original group of lecturers. The main focus at the intervention was, however, the teachers. To safeguard bias, different parts of the evaluation have been written by different people. Thus, one part is written by visiting SCN education staff from other countries and a lecturer from the University of Zimbabwe, brought to Ethiopia to see, assess and learn.

Ten staff members from the Faculty of Education, University of Bahir Dar, the regional university of the Amhara region, were also invited to the 10^th intervention which was devoted to evaluating how far the first group of teachers involved in QEP had come in their reflective thinking and practice since September 2003.

A big, two-days’ meeting was convened with invited parties from other municipalities (woredas) in the Gonder Zone. Also the head of education in the zone was present together with 10 invited guests from the University of Bahir Dar, one lecturer from the University of Zimbabwe and SCN education staff from Nepal, who had never been to this project before. The QEP coordinators and education officers from SCN Mozambique and Zambia were also present.

The SCN staff, together with the person from the University of Zimbabwe, were the ones to write up the results of the teachers’ presentations. The explanatory introduction is written by the project leader. Also the ten lecturers from the University of Bahir Dar were invited to form part of the external perspective of the evaluation. Their opinions are presented as they were written by themselves, as well as a tabulation of comments from the district education authorities from other districts who do not yet have QEP are involved, as they were written in the evaluation report.

The accompanying interviews and observations were carried out by “the old” group of teacher educators and SCN staff, the project leader and the QEP coordinator in Ethiopia. Interviews were carried out in Amharic and to a certain extent in English with an interpreter present. All interviews with children were conducted in Amharic.The analysis of interviews and observation were done by organizing a participatory analysis of the findings.

Is a participatory, multi-perspective, multi-voice evaluation a feasible way of carrying out an evaluation?

The present evaluation, which is the topic of this paper, attempts to have a multi perspective and multi-voice approach. The paper will not concentrate on the actual outcomes of the evaluation, but instead try to raise the choices and dilemmas encountered in carrying out an evaluation in such a manner. One result found in the evaluation is that teachers and lecturers indeed have started to question their own teaching and role in the classrooms. This fact begs the explanation as to why they have started to change, and whether this change is real.

The causal question is a central issue in evaluation theory. Some theoreticians claim that it is not the task of evaluation itself to investigate the deeper causes for findings, while others say that there is no difference between research and evaluation in this respect (Hammersley, 1992).

”I am not convinced by arguments to the effect that research is one thing and practice is another and that they should be kept separate. They are not one and the same, but neither can they be neatly disentangled. (In Shaw, 1999 p. 177)

The debate within evaluation theory as to whether there is indeed a difference between research and practice is discussed by Shaw (1999) who refers to Lang (1994, p.265) as one who “considers differences in the data-processing strategies of social work practice and qualitative research and how the two might be integrated. This view is opposed by Hammersley and others.

In the QEP project outline it is stated that:

“People should have to identify, realize and acknowledge the problems by themselves (in this case related to the quality of education) by investigating the problems, finding out what constitute them, exploring and acknowledging their own role within this context. As part of this investigation they also have to find the solution to the problems. In this way the solutions may be of a lasting nature –

i.e. more sustainable. Such changes in mentality are difficult and require long time. (Nagel, T. NFU conference, Bergen, Oct. 2004)

In the evaluation theory it has also been a debated issue whether an evaluation should always have to draw conclusions and suggest remedies (Shaw 1999, p. 23). To me it sounds contradictory that an evaluation should abstain from doing so, at least from drawing conclusions. After all, evaluation is about “deciding what value something has”. The question is therefore what purpose the evaluation has. For QEP evaluation is clearly seen as a two-pronged tool. Firstly it should assist in developing the project, secondly it should also contribute to participants’ understanding of their own roles and development; hence the role of evaluation as a progress catalyst is central. It has not yet been discussed with the participants how the evaluation findings can be brought to bear on the development of the project in other countries as well. It is only by analyzing these data and writing them up that I have come to see the potential for reflection in the leadership of QEP in the other countries as well. For practical reasons (people are scattered) such discussions require a well summarized paper on the different experiences and a forum for sharing them. Such work has just started. There is also the opportunity to use the QEP web-site, but until now this has not been very successful as a discussion forum.

The question we address here is really: why have teachers and teacher evaluators participating in QEP started to question their own role and teaching? Can the evaluation carried out establish a causal link between the project involvement and the change that participants claim to have undergone?

When interviewing teachers and teacher educators the following question has been posed to them: “How can you be sure that it is because of QEP that you now see things differently and have started to change your practice?” The two main factors that an analysis of their responses yield are: 1.By linking theory and practice. One teacher said that: “By observing others I see myself” and one female teacher said after her first observation of another teacher: “I did not know this is the way I teach.” In interviews with teacher educators they admit that they did not know the reality of the schools and classrooms. They were surprised at the difficult situation teachers have and they also were surprised at how little it seemed that what had been taught in teacher education had transpired into practice. At the beginning teacher educators even denied and discussed fiercely whether the teachers they observed “were their own product.” They have later reflected upon this and admitted that it was a denial of their responsibility for the poor quality teaching they saw in schools. 2. By demonstrating that the problems encountered in teaching are common and not individual

A question that has puzzled the project leader and coordinators has been the evident change observed in teachers’ and teacher educators’ confidence. We have asked ourselves why their confidence has changed what concretely has brought it about. Has it a direct relationship to the project involvement? In other words; what is the causal link.

At the beginning of the project, female teachers were so shy they lumped together in a corner of the seminar room and male teachers offered “to speak for them.” (It must be added that Ethiopian culture is very gender segregated and male chauvinist.) Today we observe that the most eloquent of the QEP teachers are females who stand up to tell or read about their action research project to a big audience who pose critical questions to them. This does not topple them at all. Not all the teachers have undergone this striking change to the same degree, but most report about an increase in confidence.

In the preparations for the interview with teachers, the teacher educators and project leader had brainstormed topics to cover in the interview. One such topic was what changes had occurred and what were the reasons fro these changes. – We learnt that their confidence had increased by observing other teachers and understanding that the problems they have are exactly the same problems as they themselves struggle with in their daily lives in class. One teacher said in the interview that “Today I am welcoming anybody to visit my classroom. I know I have no reason to be shy, as I have no other special problems than what other teachers encounter in their classrooms. On the contrary, I am aware of them and now play a leadership role in my school in trying to change my classroom behaviour!”

For teacher educators we have seen the same. They feel much more confident as they now know more about the professional problems and that they are shared by many. They claim to be the ones at staff meetings who speak out about professional issues with confidence. At a degree course for all teacher educators at the college, they claim to be the most reflective and those who can participate most actively because of the involvement in QEP. They also admit that they did not know the situation in schools, as the only ones they had observed were their own students during teaching practice. This has prompted them to reflect upon their role in the preparation of teachers. They have also started to question their own practice by carrying out action research projects. But teacher educators have been much slower in developing critical self reflection. Why this is so, we don’t know, but it might be attributed to status. It is something we see in all the other countries too.

In afterthought it can be summarized that if the project had started out by asking teachers and teacher educators to do action research directly on themselves without going the way via observing others, we might not have had the same results. Starting by self reflection is the model prescribed in much of the literature on action research in schools and the teacher as researcher tradition, especially the Australian and British traditions. Since I saw that reflecting about one’s own practice and underlying values is more demanding than reflecting about others, I designed the project in such a way that the participants could first be brought to reflect upon others and then gradually be led towards-self reflection.

As I had experience from Laos by introducing action research into teacher education in a development project there, I had seen and experienced how extremely difficult it was to bring about critical self reflection. It was therefore built into the project design not to start out by demanding self reflection in cultures where this phenomenon is totally alien. (Nagel and Stephens, 2001) . I would therefore like to suggest that the design of the project has been important for the outcome, by letting participants start their observation of other people’s practice and then only gradually turn their view towards themselves.

Evaluation results and their potential for illumination and change

Tripp, for example, believes that the evaluation research process should ideally bring about increased understanding both to the interviewer as well as the interviewee (Tripp, 1983, p.34) and that the two through a mutual interpretation process arrive at an agreed-upon outcome.

Participants all claim that they have come to see their job in a very different light. They also acknowledge that the problem often lies with themselves and their lack of awareness about the individual child. When interviewed, teachers say that they have changed from seeing the class as a whole to seeing the individual child. They say they used to punish children by pinching and beating, and that they now have stopped that, acknowledging that it doesn’t promote learning. A critical outsider might say that this is being said to please the interviewers. Being aware of this danger we also observed the teachers whom we interviewed. All 16 teachers in the first group were observed before they were interviewed by the 9 teacher educators and SCN QEP staff. Our observations and interviews were discussed and written up in a plenary session.

All teachers were all found to be behaving very empathetically towards their pupils. Only one exception was found. One elderly QEP teacher had relapsed back to old stereotyped teaching, displaying uninspired subject delivery of content and denied to correct childrens’ work when asked. An indicator of theachers’ interest in their students is the fact that they knew the names of all the children in class (often close to 60 or above). This is remarkable as schools had just resumed and classes were in most cases new to the teachers. This has not been a common observation we have done of classrooms in Ethiopia before. This finding was also corroborated by interviews of children, who said their teachers were nice and do not pinch or beat. “They give us a second chance to respond to questions, and they explain if we don’t understand.”

But it was also found that there are still weaknesses in teaching by some teachers. For example one male teacher asked his grade one to copy the tasks written on the blackboard, when children had just been to school for 2 weeks. The major change found in teachers’ behaviour was in the relationship with their students. Other findings also corroborated this, both interviews with children, but also parents, who are now said to demand that their children are put into the QEP teachers’ classes, because they learn well.

In one grade 2 all the 60 children could write some Amharic words on the blackboard and respond to simple questions about their name and their fathers’ name in English. They could also write simple English words. This is remarkable, as English has different letters from Amharic. A previous test carried out by the lecturers showed that even in grade 4 less than 50 % were able to read 15 common words correctly in the Amharic alphabet.

Further inquiry revealed that all teachers say that qualitative research and action research have been new to them, also the lecturers, but that it is reflection that has been most helpful, but also most difficult, as it is a totally new concept to them. As a result they claim that they have changed their attitude to many things, also privately. Some teachers are now being used as conflict mediators in their communities, “because of QEP we know how to investigate problems and to find solutions.” This finding was corroborated by other informants: For example the husbands of two of the teachers in Delgi said that their wives had become so different. They solve problems at home and are even asked for advice by the neighbours.

Teachers say they have changed their behaviour in class, something which can be corroborated by observation and information from their colleagues as well as from parents, who now want their children to be put in the classes of the QEP- teachers. They say that “before we did not use to discuss professional issues at school, but now we even stay after working hours to talk about our work.” This claim by the QEP teachers was found to be true when we organized visited to all the five schools where the QEP teachers work to invite all their colleagues to become part of the project. They had seen how the QEP teachers had changed and become so interested in their work and had, as a result, brought change to the whole school, also motivating their colleagues. There are two examples of this; in one case all teachers at one school had each contributed 10 birr from their meagre salaries to pay for accommodation close to school for girls and children with disabilities. Another example is that teachers beautify their school, as they have discovered that the traditionally messy environment is not “nice for children to learn in.”

In the evaluation interviews children said that the QEP teachers give more feed-back, explain better and do not pinch. They also “give clues when we are not able to give the answer”. Children compare with other teachers by naming them. They said that their teachers have repaired school furniture by themselves. All these examples testify to change, not only by individual teachers but also in changing the school environment.

Teachers have developed more self confidence and also less fear to be observed by others, since they have learnt from being given the opportunity to observe other teachers. “We now know that we all have more or less the same problems. There is nothing to hide!” At the first QEP intervention female teachers were silent and sat together in a corner. One male teacher offered “to talk for them.” At the evaluation there were mostly female teachers who presented their action research projects, confidently and with a good audible voice to an audience of about 300 people in a big lecturing hall at the University of Gonder premises. Other invited guests from the MoE at different levels were surprised and impressed by what they saw.

There is a lot of change and enthusiasm in the QEP group, but there are still many things that need to be strengthened. Not all teachers are fully aware of their shortcomings in teaching, but they do treat children in a more friendly way than before.

In terms of the evaluation we can again ask: How can we be sure to attribute these changes to the QEP project? It is of course difficult to claim that there is a causal link. But knowing the situation in schools and having worked with them for a number of years, especially the people from the Ministry of Education, say they are astonished at the changes they observe in the schools. They want the project to be replicated in other woredas (districts) as well.

When asking the teachers about their change, many of them have statements like the following: “Before we were blind, now we see.” “QEP has changed my life.” Many say that their work has become more interesting and they have become more motivated and discuss professional matters with colleagues, something that never happened before.

Also lecturers claim that QEP has been important to them, both professionally as well as personally. In the first place, they now know what the real situation is like in the schools. This has a positive impact upon their own teaching at college. They have also learnt much more about research, and can apply it both at college and when helping the QEP project teachers as their monitors. They have also started their own action projects at college, and are gradually questioning their own pedagogy at college. But more importantly, they say that they can utilize their knowledge from school observations in their own teaching. Some claim that QEP has also benefited their own life, especially when it comes to the relationship with their own children.

One somewhat surprising finding has been that teachers started to change much more quickly than the lecturers. They also displayed less resistance. This may be attributed to status. This question needs to be further investigated.

The current challenge in the project is to consolidate the developments and keep up motivation of the old teachers and also to be able for the lecturers to accommodate for the new teachers who will gradually be taken into the project.

Relationship between evaluation methodology and project design

QEP is a project which is typically dependent upon a good process to achieve results. It is dependent upon bringing about reflection by the participants. Reporting upon and understanding the process is therefore significant, and the method of evaluation ought to be chosen accordingly. There have been formative assessments carried out at frequent intervals during the project, which in turn are taken into consideration as the project develops. Careful planning of next steps is always carried out at the end of all interventions, which are normally held 3-4 times a year with local monitoring meetings with participants in between. In this way evaluation becomes a tool helping the project to develop, instead of a review of what went wrong and what did not.

The evaluation belongs to the family of “illuminative evaluations”. This implies that the findings will be used to enlighten the participants in their future work. It is therefore also participatory in the sense that participants are actively involved in carrying out the evaluation. It tries to take into account everybody’s perspectives and it is also of a formative nature so that results can be ploughed back into the activities to direct and improve the process.

The advantage of a qualitative, illuminative evaluation method is that it truly becomes “multi-perspective” and “multi-voice”; people can recognize their own views. A participatory evaluation is also less intimidating for participants and the have an ownership to it. It is something participants welcome and do not fear.

Also representatives from the Ministry of Education from other woredas (municipalities) than the two where this project takes place (Alefa Takusa and Chilga) have voiced their opinions about the project on the basis of the presentations of the QEP teachers that they witnessed. As a result, some QEP teachers were invited by the zone education office to present their action research projects to a wider audience of representatives from the Ministry of Education in a meeting involving all the woredas in the zone. This was a triumph both for the teachers invited as well as for the project. There is now much interest for the project in the whole zone (2,2 mill inhabitants)

Issues of validity

A common way of ascertaining validity in qualitative research is to validate the interpretation of data; i.e. the results of analysis with the informants. By inviting many groups of people to state their opinion of the project after having listened to QEP teachers’ presentation they have presented their views in writing in a non-censored way. This approach should safeguard member validation as one way of validating the evaluation. In qualitative scientific approaches to development and evaluation, validation should not be less important or avoided because of the fact that it is an evaluation one is doing, on the contrary. Besides, if the purpose of the project is to bring about change, the educative potential in an evaluation should not be underrated.

When the purpose of the evaluation includes “to enact rather than simply state” (Lather, 1991, p.123) then analysis will be part of the endeavour to “maximize the research process as a change-enhancing, reciprocally educative encounter” (Lather , 1991, p.72) .

Some evaluation topics that have transpired as a result of the nature of the project are:

Triangulation, both when it comes to methods and respondents have been observed, and there have not been any striking differences in findings between observers or observed or as a result of methods applied.

The issue of external versus internal evaluation has been addressed in searching for a combined perspective from insiders and outsiders. It has been attempted to give preference to insiders’ perspectives.

During the work with writing up the evaluation it has become even clearer to the project leader that the evaluation findings have a further development potential than what was originally envisaged. At the beginning it was seen as an ethical and necessary part of the evaluation to make it a participatory exercise. Individuals and groups were invited to reflect and contribute to the evaluation. Findings of the interviews and observations with teachers were indeed discussed with lecturers, as they were analyzed and written up together with them. But they were never taken back to the teachers in a systematized way.

It has gradually become clear to me that if results are “given back” to all the participants to discuss in amore organized way, even more enlightenment might be brought about. For the teachers, and especially for those who are new to the project, it may bring about new understanding of what the project “is all about”. In doing so, we may create another “spiral of reflection” in the true spirit of action research, a spiral that has the potential for never ending!

References

Carr, Wilfred and Kemmis, Stephens (1986) Becoming Critical. Education, Knowledge

And Action Research, The Falmer Press, London

Carson, Terrance and Sumara, Dennis (1997) Action Research as a Living Proctice

Peter Lang, New York

Campbell, D (1964) “Distinguishing differences of perception from failures of communication in cross-cultural studies” in Northrop, F. and Livingstone, H (eds) Cross-Cultural Understanding: Epistemology in Antropology. Harper and Row, New York

Campbell, D. (1996) “Can we overcome worldview incommensurability/relativity in trying to understand the other?” in Jessor, R. Colby, A and Schweder, R (eds) Ethnography and Human Development Chicago University Press, Chicago

Chronbach, L. et al (1980) Toward Reform of Program Evaluation Jossey-Bass, San

Francisco

Day, Christopher, Elliot, John, Bridget Somekh and Winter, Richard (2002) Theory and Practice in Action research Symposium Books, Oxford

Dewey, John (1916) Democracy and Education – An introduction to the philosophy of education The Free Press, New York

Dewey, John 1933, How we think Henry Regnery, Chicago

Fournier, D (ed) (1995) Reasoning in evaluation: inferential Links and Leaps,

San Francisco: American Evaluation Association /Jossey-Bass

Habermas, J, (1972) Knowledge and Human Interest Heineman, London

Habermas, J (1987) The Theory of Communicative Action, Vol. 2 Lifeworld and system Polity Press, Cambridge

Hammersley, M (1992) What’s Wrong with Routledge, London

Kemmis, S and McTaggart, R (1988) The Action research Planner Deakin University

Kulpoo, Durumbeer The quality of education: some policy suggestions based on a survey of schools SACMEQ policy report 1, Mauritius, 1998, IIEP, Paris

Liston, Daniel and Zeichner, Kenneth (1996) ,Reflective teaching. An Introduction ,

Lawrence Erlbaum, New Jersey

Machingaidze, Thomas, Pfukani, Patrick and Sibangani, Shumba The quality of education: some policy suggestions based on a survey of schools SACMEQ policy report 3, Zimbabwe, 1998, IIEP, Paris

McTaggart, Robin (ed) 1997 Participatory Action Research International Contexts and Consequences , State University of New York Press

Nassor, Sebtuu and Khadija Ali Mohammed The quality of education: some policy suggestions based on a survey of schools SACMEQ policy report 4, Zanzibar, Tanzania, 1998, IIEP, Paris

Nagel, T and Stephens, D (2001) Where East meets West. Introducing Participatory Action research into teacher training colleges in the Peopls’ Democartic Republic of Laos paper at the Oxford International Conference on Education and Development, Sep 19-21

Nagel, T, (2004) Adressing the quality of education through participants’ construction of

Knowledge. Save the Children Norway’s Quality Education Project in teacher education and primary schools in Ethiopia, Zambia and Mozambique

NFU conference, Bergen, October

Nagel, Tove (ed) (2005) A participatory, formative evaluation of the first phase of the Quality Education Project (QEP), in Gonder Zone, Amhara region, Ethiopia Sep 2002 – May 2005 Save the Children Norway, Oslo (In process) to be published at www. rb2.no/qep

Nkamba, Manasseh and Kanyika, Joe The quality of education: some policy suggestions based on a survey of schools SACMEQ policy report 5, Zambia, 1998, IIEP, Paris

Noffke, Susan and Stevenson, Robert (eds) Educational Action research. Becoming Practically Critical Teacher College Press, Columbia University, New York

Nzomo, Juliana, Kariuki, Mary and Guantai, Lilian The quality of education: some policy suggestions based on a survey of schools SACMEQ policy report 6, Kenya, 1998, IIEP, Paris

Ross, Kenneth (ed) (1998) SACMEQ Policy Research Reports no 1, International Institute of \Educational Planning, Paris

Schön , Donald 1983, The reflective practitioner: How professionals think in practice New York, Basic Books

Scriven, M, (1995) The logic of evaluation and evaluation practice in Fournier, D.(ed) Reasoning in evaluation: inferential Links and Leaps,

Shaw, Ian F. (1999) Qualitative evaluation. Introducing qualitative methods,

Sage, London

Stake, Robert (1995) The Art of Case Study Research Sage: Thousand Oaks

Tripp, D.H. (1983) “Co-authorship and negotiation: the interview as an act of creation”

Interchange, 14 (3) (pp 32-45 )

Voigts, Friedhelm The quality of education: some policy suggestions based on a survey of schools SACMEQ policy report 2, Namibia, 1998, IIEP, Paris

www.rb2.no/qep The Quality Education Project web-site

“Defining Quality in Education”, (2000) UNICEF, New York

The Leap to Equality EFA Global Monitoring Report 2002, UNESCO, Paris

The Quality Imperative EFA Global Monitoring Report 2004, UNESCO, Paris