ATTACHMENT_4_-_EGRA_-_2012.pdf

Technical Report on the Administration of the EGRA National test

Carried out October 6 to 8 2012 in Djibouti

Ministry for National Education and Professional Training of the Republic of Djibouti

USAID Project AIDE Djibouti fhi360, Washington, DC

Executive Summary

In 2001 - 2002, the Djibouti Minister of National Education began a reform of the education system to improve access and quality.

The reforms were based on:

- improvement of teaching quality,

- design of a national curriculum,

- publication of textbooks and teaching tools adapted to the socio-cultural context,

- increase in the role of parents and communities,

- introduction of competence-based approaches to pupil learning,

- strengthening of the informal, private system to reach young people who are not enrolled, particularly girls (Development plan 2001/2005).

Since then, the number of primary schools in the country has increased, going from 108 in 2003 to 159 in

2012, while the Gross Enrolment Ratio in the primary sector has gone from 49.5% in 2003 to 78.5% in 2011.

During the same period, the Gross Admissions Rate has increased from 50.7% to 76.6% (Annual statistics, 2012/2013).

Communities are increasingly involved in the education of their children with the active support of school management committees and parent associations. Despite this success with access indicators, the improvement in the quality of learning has not had the desired impact, especially in the ability of pupils to read fluently.

Methodology

In September-October 2012, in collaboration with MENFOP, FHI360 through the AIDE Project financed by

USAID, put in place an evaluation of basic reading competence (EGRA) in the 2nd, 3rd and 5th year classes in the 10 districts of Djibouti.

A multi-sectorial sampling method was used to select the pupils. The global sample was divided into two steps:

- first, the class was the primary unit of selection,

- then, random choice of the pupils in each class participating in EGRA.

A total of 1106 pupils were sampled in the 10 districts.

The following EGRA sub-tests were applied to each of the selected pupils for this study:

- orientation towards reading

- identification of initial sounds,

- knowledge of letters / sounds,

- reading of common words,

- reading of made-up words,

- verbal reading fluency, reading comprehension,

- listening comprehension, and writing.

The EGRA results are summarised below.

Results

The best result for the pupils was in recognising letters/sounds and awareness of phonemes. In the 2nd year, only 14% of the pupils in the sample had a null score (zero). In the 5th year, only 5% got zero. Similar results were observed for knowledge of sounds / letters with only 12% and 9% of pupils scoring zero for the 2nd and

5th years respectively. As pupils progress from the 2nd to 5th year, their ability to name letters and sounds continues to improve, as shown by the average scores. In the 2nd year, 49% of pupils were able to give the names of letters correctly and just 17% of pupils knew the sounds of the letters. In the 5th year, 63% of pupils named the letters correctly and 31% knew of the sounds of the letters.

The sections where pupils had the most difficulty were in general:

- reading of made-up words;

- fluency in reading a text aloud;

- reading comprehension.

The ability to read made-up words allows verification of pupils’ ability to decode words, allowing them to learn how to read. Fifty per cent of the pupils in the 2nd year in the sample were not able to read a single made-up word and the average number of words that the pupils were able to read among all pupils in the 2nd year was 6. In the 5th year the results were better - 75% of pupils at this level were able to decode at least 1 word and the average for the number of correct words per minute was 21.

In terms of verbal fluency, 45% of pupils in the 2nd year were incapable of reading a single word of text. In the 5th year, more pupils were above the zero score, but 16% of pupils at this level were still incapable of reading a single word. In terms of words read per minute, the 2nd year pupils were able to read an average of

11 words per minute. In the 5th year, the pupils had made significant progress in improving their fluency in reading a text aloud, the average in reading text being 39 words per minute.

However, this rate is still below the proficiency standard set by cognitive science, in relation to the level needed to reach understanding (60 words/minute).

The importance of developing a solid foundation competence in reading appears when we separate the results of the pupils who had succeeded in understanding a text at 80% or better from the pupils who were below

80% for reading comprehension. In terms of knowledge of letters / sounds, the pupils having more than 80% reading comprehension had a result twice as high in phonemic knowledge (43 against 23) and four times as high in fluency with spoken texts (81 words per minute against 20 words per minute) than their peers who were below 80% in comprehension. These differences demonstrate that children who understand what they read know phonetics and have a more developed phonemic awareness.

EGRA is a tool for evaluation of fundamental competence in reading used internationally, and in comparing the results obtained in Djibouti with those of other countries, Djibouti rates at an average level and ahead of several African and Middle Eastern countries (see figure 6 p54).

Recommendations

To support the development of improvement in reading competence, two areas of intervention are preferred:

improvements to the teaching of reading and putting in place remedial strategies for pupils with learning difficulties. Other types of activity which can optimize teaching of reading are detailed under each topic below.

Improvements to the teaching of reading

Give more attention to teaching of phonology in general and to decoding in particular;

Improve the direction and structure of teaching reading to favour greater fluency in reading text by the end of the 3rd year.

Remedial strategies

Use of differentiated teaching methods in class;

Individual support outside the class for pupils in difficulty;

Systematisation of reading support, especially in disadvantaged areas.

Provision of reading resources

Ensuring that schools have libraries and books that the pupils can read;

Ensuring that classrooms are an environment rich in reading materials and publications.

Training and support for teachers

Professional development for teachers based on the didactics and teaching of reading;

Supplying teachers with written lesson plans and teaching materials that will help them to create activities that help children with reading.

Language development

Teachers must promote development of oral skills in the pupils to help them master the language of teaching.

Develop pupils’ phonemic awareness, through nursery rhymes and word play.

Community / Parent participation

Engage the community in development of reading activities;

Make books available in the community to allow involvement in these activities;

Implement workshops for parents (design of reading material);

Promote the proliferation of reading competitions and festivals.

In consequence, by putting the accent on specific activities that reinforce:

- fundamental reading competences in children;

- the engagement of the community by setting reading milestones;

- ensuring that all children receive an education from early childhood. Djibouti will continue to improve its literacy levels during the coming years.

http://www.alliance.brown.edu/tdl/elemlit/readingk3.shtml# strat1 http://www.alliance.brown.edu/tdl/elemlit/readingk3.shtml# strat1 http://www.alliance.brown.edu/tdl/elemlit/readingk3.shtml# strat4

Chapter 1: Introduction and context

Djibouti is a small country of 23,200 square kilometres with a population of around 800,000 inhabitants, of which 87% live in urban areas. Administratively, the country is divided into six regions (Djibouti, Arta, Ali-

Sabieh, Dikhil, Obock, Tadjourah). The population of Djibouti is young, with around 40% under 15 years old

(World Bank, 2009). The education system is subdivided into 10 districts of which 5 are in Djibouti city.

Compulsory education (from age 6 to 16) is for nine years of which five are in primary school and four in middle school.

I. Educational context in Djibouti

A major event in the history of the education system in Djibouti was the reform undertaken in 2001-2002, following an education forum that took place in December 1999. This reform, based on both access and quality, sought to improve the quality of teaching through the introduction of a new pedagogical approach (a competence-based approach), the design of appropriate teaching materials (curricula and textbooks), the training of teachers based on their professionalization and the strengthening of the role of parents and communities. Further, the reform sought to strengthen the capacity of the informal private sector, to include young people who are not enrolled and to strengthen also the schooling of girls (Master plan 2000 – 2008).

Certainly enormous progress has been made, particularly in the field of access to education. According to the

MENFOP Statistical Yearbook 2012-2013, the number of primary schools in the country has increased, going from 108 in 2003 to 159 in 2012, while the Gross Enrolment Ratio in the primary sector has gone from

49.5% in 2003 to 78.5 % in 2012. During this same period, the gross admissions rate has gone from 50.7% to

76.6% while the net schooling rate in primary schools has gone from 30.9% in 2003 to 68.5% in 2012;

during the same period, schooling of girls has risen to reach an average rate of 46% in 2012 (47% in urban areas and 44% in rural areas).

Communities are increasingly implicated in the education of their children with the active support of school management committees and parent associations. In addition, important reforms have been achieved by

CRIPEN (Centre for Research, Information and Production in National Education) in the field of design of new curricula and the publication of teaching material (textbooks and teaching tools), corresponding to the

Djiboutian context and allowing each child to have one book per major subject (MENFOP, 2011).

Despite the progress made, there are still difficulties to be overcome such as: large class sizes; a high number of pupils repeating a year, especially the final year of primary or middle school; and the inadequate initial training of teachers in middle schools. Furthermore, the National Education Inspectors and Pedagogical

Advisers charged with monitoring teaching quality do not always find it easy to visit remote schools that fall under their responsibility.

It is clear that by putting the accent on learning quality, one can influence the success of pupils at the level of basic education where reading remains the means of access to the scholastic culture. Moreover, through its education system, Djibouti rises to the challenge of instituting “a school system, for the people, inclusive and of quality, centred on pupil success, in order to make responsible citizens prepared for the demands of society, but also for regional and international needs.” (master plan 2010-2019).

II. Socio-linguistic context

French and Arabic are the official languages in Djibouti. The vast majority of public schools use French as the teaching language right from the first year of school, though there are some Arabic-language schools. It should be noted that a peculiarity of Djibouti is that the national languages spoken at home are not taught at school. For this reason, learning in a foreign language leads to obvious difficulties for the young children who are starting to learn to read. Further, the lack of a cultural environment favourable to the use of the teaching languages outside of the school constitutes one of the major difficulties encountered by young children who are starting to learn to read. Also, the high level of illiteracy among the vast majority of parents forms a further obstacle to learning to read for children at the start of their schooling. It should be underlined that written material in mother tongues is not used at home or in the immediate environment (neighbourhood, village) of the young children.

III. Teaching reading in Djibouti

The current approach to teaching reading in schools in Djibouti is the competency-based approach highlighted in the education reforms. In the first year of primary school, 7 hours per week are devoted to oral expression (a pre-condition for reading), 9 hours to French and 5 hours to reading. The programme offers a mixed approach (starting from a sentence, word to introduce letters and sounds) though some teachers have a tendency to favour recognition of entire words. Twice-weekly study of letters and their corresponding sounds is introduced in the textbooks for pupils in the first two years, in a structured way. This gives an important place to the combinatorial approach to learning to read. Also the exercise books, which complement the main textbooks, provide for syllabic practice by putting the accent on coding and decoding of graphemes and phonemes (a phonetic and syllabic approach) to the teaching of reading.

A national evaluation carried out in 2010 allowed for a measure of the reading competence level of pupils after two years of school. In the report analysing the results of this evaluation, it is stated that “The weaknesses can be explained in part by the fact that the process of teaching and learning reading is done in

French or in a language other than the mother tongue of the pupils. In other words, for most pupils school is the only place where they speak French.” (Analytical report, IG – MENESUP, 2010). Also, the conclusions showed that the average results were poor, even if there was a large range of scores including some very strong pupils and some others who were very weak. The same evaluation also showed the importance of learning to read early, in accordance with work done in the field of reading indicating that learning to read early is essential to learning to read well.

IV. National evaluations

Djibouti has a well-developed system of national evaluations in basic education (at the end of the 1st and 2nd stages of primary education and at the end of middle school). It should be remembered that the teaching programmes developed as part of the reform describe the basic competences that pupils must acquire in each subject and clearly determine the knowledge, know-how and social skills on which learning is based. Thus at the end of each stage of school, the curricula define Terminal Integration Objectives on which the evaluations are based.

Each year, tests based on the specific competences that the pupils should have acquired by the end of the year, as specified in the national programme, are administered by teachers throughout the country. They include an element of individual oral evaluation and sections on phonemic awareness, the correspondence of graphemes and sounds, reading of known and unknown words and reading comprehension, as well as production of a written text. The analysis of the evaluations at the end of the 2nd and 5th years published by

MENESUP indicate that, by reason of their focus more on administration (certification) than pedagogy, these evaluations show average performance in one subject or another which do not represent the basic levels of competence.

However, it is difficult to measure the level of acquisition of competences, because the tests rely on written tests which require a minimum knowledge of reading. For this reason, in case of an incorrect answer by a pupil, it is not possible to determine if this is due or not due to reading proficiency or simply to the answer itself. In consequence, with a view to better measuring pupils’ achievements, the minister ordered an evaluation of reading competence, to diagnose the level of pupils in primary education. The choice made was to put in place the EGRA system in Djibouti.

V. Reasoning behind EGRA

The reasoning behind the choice to EGRA flows in part from the international recognition that the quality of primary education has not kept pace with access to education (Fast, 2011). Reading has been described as the gateway to understanding (McPile, 1995). Since acquisition of learning takes place in writing, the learning path for children is blocked if they cannot read and understand fluently.

Some of the existing problems are as follows: traditional methods of teaching reading do not facilitate acquisition of reading competences; children are not encouraged or motivated to read at school or at home;

reading material is not easy to find; parents and communities are not aware of the importance of reading or the scale of the problem and therefore don’t help to find solutions (Fast, 2011).

Already, in several countries where the children have received schooling for 4 or at most 6 years, they are not able to read or understand simple texts (RTI, 2009). Data indicates that learning to read early, at a sufficient rate, with comprehension, is essential for learning to read well. Reading is essential for academic success in general. Children who do not learn to read in the first years of school are most often those who have to repeat a year, eventually leaving school, or who often encounter great difficulties compared with others for the rest of their lives (RTI, 2009).

In recognition of this, the international educational community has put a strong emphasis on the importance of pupils learning to read early. A good number of international organisations, notably USAID, ask countries to monitor the proportion of pupils who develop the reading competency adequate for leaning from the documents they read (USAID, 2011). For all these reasons, and to help measure reading competency, a certain number of international donors, and in particular the US Agency for International Development and the World Bank, have collaborated to develop and finance EGRA. The first evaluations of EGRA took place in 2006 and 2007 in requesting countries. The development of this tool was based on existing research into reading learning at the time. Since then, the EGRA tool has continued to be refined and improved. EGRA has now been widely adopted as a standardized evaluation tool in a certain number of countries in Sub-

Saharan Africa.

Chapter 2: Development of the EGRA pre-test

The design of the EGRA test took place under the supervision of the Inspectorate General of the Ministry of

National Education and Professional training (MENFOP) of Djibouti. Ministry personnel took part in all stages of the process. A certain number of preliminary meetings had been organised to discuss the global reach and goals of the evaluation. In January 2012, a one-week workshop took place in the presence of inspectors, representatives of the MENFOP evaluation unit, representatives of the exam bureau, representatives of the evaluation services of CRIPEN and CFPEN as well as pedagogical consultants.

Discussions took place to see if it would be necessary to prepare the EGRA test simultaneously in French and in Arabic. It was decided that the organisation of the test would take place first in French and that an Arabic

EGRA would follow, possibly in 2013.

The EGRA tool must also be specifically used in Djibouti for:

Providing information on basic reading competence among primary school pupils;

Providing information on the characteristics which would lead to good results in reading;

Allowing targeted measures to be taken to improve reading levels. Such measures could be targeted to specific early reading competence in the field of learning and to teaching tools (image boards, textbooks, exercise books);

Providing a reference point against which future measures could be evaluated, while reinforcing, at the same time, the capacities of key MENFOP personnel in the field of evaluation of reading measures over the course of the first years.

Consequently, EGRA planning and preparation began in January 2012. During the January workshop, the participants were thus able to familiarize themselves with the merits of the EGRA content. A decision was taken to carry out EGRA at the 2nd year and 3rd year level as well as at the end of primary education.A provisional calendar was put in place, envisaging the organisation of the test in May 2012, towards the end of the school year. The roles and responsibilities of the different parties involved in the planning of the test have also been discussed an organisation committee has been set up to supervise all aspects of the preparation of the pre-test as well as the corresponding logistics. Two other workshops were organised to prepare the tools:

one at the end of January 2012 and the second at the beginning of March 2012 to support the team charged with developing the test. The pedagogical consultants trained the core members of the team charged with adapting the tools to the Djiboutian context. They have developed texts adapted to the Djiboutian context and to the level of the classes.

The following steps have been respected in adapting the tools:

1. Analysis of the EGRA tools in other French-speaking countries such as the Democratic Republic of

Congo, Haiti, Mali and Senegal;

2. Choice of activities appropriate to the context and interests of Djibouti;

3. Modification of the content of certain activities to take account of the local context, for example to identify words not normally used in Djibouti and to adapt them;

4. For the written comprehension, preparation of a new text reflecting the background of the pupils:

5. Choice of new sentences for the dictation;

6. Thorough examination of all the contextual questions on the basis of the pupil survey, so as to make any required changes;

7. Modification and simplification of the instructions so that Djiboutian children can better understand them;

8. Deciding on a format that will make the test more efficient and easier to use.

Other documents, like the Evaluation Tool for reading in the first stage of primary school (RTI, 2009), have also been consulted and the decision to include them in the Djibouti EGRA tool has been taken.

I. Components of the EGRA tool

The activities in Figure 1, represented in function of reading level, have been chosen for inclusion in the

Djibouti EGRA tool.

Figure 1: Djibouti EGRA tool components according to reading level

1. Orientation and preparing to read: This activity is made up of three questions concentrating on understanding of the basic concept of reading, to know where to begin reading a text, in which direction to read and where to go once the pupil has reached the end of a line.

2. Identification of initial sounds: This sub-test is based on phonemic recognition, that is the breakdown of words into sounds. Pupils who find this difficult tend to have difficulties in acquiring alphabetic recognition competence and therefore have difficulties in recognising words (Stanovich, 2000).

3. Identification of graphemes: This basic competence implies recognition of isolated letters and groups of letters, as well as the ability to pronounce the latter.

4. Reading a well-known word: This evaluates pupil competence in decoding with the aid of lists of non-related words. This allows for a clearer measure of competence in recognition and decoding of words, as with paragraphs in a written comprehension, especially as the children are unable to guess the following word from the context. Well-known, frequently used words have been selected for this evaluation.

5. Reading of made-up words: Reading meaningless words is a measure of recognition capacity which avoids the problem of visual recognition of the words. Taking account of the global approach used in 1st year in Djibouti, children in the youngest classes can memorize and visually recognize a series of words.

To be good readers, children must combine competence in decoding and visual recognition. Tests that do not include a decoding exercise can overestimate the capacity of children to read unknown words (given that the words in the test can be taken from the visual recognition vocabulary).

6. Reading a text: Oral reading fluency is a measure of general reading competence: the ability to transpose letters and sounds, to unify sounds to make words, process connections, relate text to meaning, and make inferences to fill in missing information (Hasbrouck and Tindal, 2006, RTI 2009). Tests in oral reading fluency, as measured in timed evaluations of correct words per minute, have been shown to have a strong correlation with measures of reading comprehension (RTI 2009).

6a. Written comprehension: Children are asked two factual and inference questions linked to the part of the text they have read. Two associated tests have also been included:

6b. Oral comprehension: Evaluation of oral comprehension is an exercise during which the children listen to a short passage then respond to oral questions. The poor results in the oral comprehension test are due to the fact that the children do not know the basic vocabulary included in the texts, or they have difficulty in translating the sounds they hear into the correct letters and words. The team decided that it was therefore important to include this test, given the socio-linguistic situation in Djibouti.

8. Writing: This tests the ability of the pupils to listen to sounds and correctly write the letters and words corresponding to the sounds they hear. This serves to show their success with the alphabetic process.

II. Link between the sub-tests and competence

Table 1: Components of the test and reading competence

Component

Early reading competence

Competence shown by the pupils through their ability to:

1. Orientation towards reading

Recognition and organisation of a text, direction of reading

Show where reading begins in French (top left-hand corner).

Show in which direction the text proceeds (from left to right).

Show where to find the next word after the end of a line

Show the last word of a text (just before the full stop).

Show the last word of the first sentence (last word of the first sentence before the full stop).

2. Identification of initial sounds

Identification of the initial sound of a word

Segmentation of a sequence of sounds making a word

3. Knowledge of sounds

Recognition of letter names and sounds

Give the name or the sound of capital and lower-case letters given in random order.

4. Reading of common words

Reading words Read simple, common words of one or two syllables (recognition of groups of graphemes combining to form sounds groups and production of words making sense).

5. Reading of made-up words

Reading “logatomes” or

“nonwords”

Make the grapheme-phoneme connection through reading made-up words

6. Reading text and comprehension of a read text

a. Reading text

Read a text continuously and with appropriate segmentation of words and sentences.

Component

Early reading competence

Competence shown by the pupils through their ability to:

b. Comprehension of a read text

Reply correctly to different types of questions, including literal and inferential questions on the text read aloud in the previous activity.

Show the ability to understand the main ideas in a text and achieve a level of automaticity

(fluency) allowing immediate comprehension.

7. Listening comprehension, Comprehension of a text read by the assessor

Reply correctly to different types of questions, including literal and inferential questions on the text read to them by the assessor.

8. Writing a sentence Correct use of writing conventions through a dictation exercise

Write a simple sentence in French.

Based on World Bank (2011), MENFOP, FHI 360 (2012) Training manual for Supervisors and administrators of the reference line (EGRA Test)

III. Tools and material to accompany the EGRA TEST

A. An EGRA test kit has been developed for pupils. It contains:

A test-book to record the individual responses of pupils being tested and also containing the protocol and instructions to give to the pupils as well as a questionnaire on the pupil’s learning environment.

A laminated sheet with a reading text on one side, to use for the “Orientation towards reading” task (Doc.

A) and on the other side a list of 100 graphemes to use for the “Knowledge of sounds” task (Doc. B)

A laminated sheet with 50 common words on one side (Doc. C) and 50 made-up words on the other side

(Doc. D)

A laminated sheet with a reading text on one side, to use for the “Reading text and comprehension of a read text” task (Doc. E) and on the other side the sentence for dictation for the task “Writing a sentence”

(Doc. F)

(See annex 1 for the criteria used in the pre-test and annex 2 for the final version of these tools.)

B. The training manuals

Training manual for Supervisors and administrators of the reference line (EGRA Test); working document used during the training workshop for the 80 administrators, from September 2012;

Training manual for administrators that was distributed prior to the building of their skills;

Training manual for Supervisors which was distributed with the same objectives as for the administrators;

Questionnaires for teachers and school directors have also been adapted to the Djiboutian context.

IV. Objectives of the EGRA pre-test

The main objectives of the EGRA pre-test can be summed up in three broad categories:

1. An evaluation of the EGRA tool to allow us to revise it if necessary, so as to ensure that the reference line can provide valid and reliable information on the reading situation in primary classes in Djibouti.

2. An evaluation of the package of procedures involved in the evaluation of this type of evaluation so that it could be improved and modified if necessary.

3. A first analysis of the results to distinguish the sub-groups and sub-tests that can provide the most interesting information. This can provide detail on what information the EGRA tool can give us, with a view to determining if supplementary information might be necessary.

The objective being precisely to pre-test and validate the following factors:

Table 2: Aspects of the tools and material to validate in the pre-test. Fast (2011)

Tools and accompanying material Procedures Analyses

Book used to record the responses

Tools to stimulate the pupils.

Instructions for test administrators.

Questionnaires for pupils, teachers and school directors.

Training material for administrators and supervisors.

Use of a watch to time the activities.

Role of the sub-groups involved in test administration.

Printing of the test and accompanying material.

Training for trainers and supervisors.

Logistics for test planning.

Organisation of the test at school level.

Marking, anonymity and ranking of pupil results.

Data processing, including procedures for entering and analysing the data.

Groups and sub-groups of interest in the analysis

(class, region, type, etc.)

Information from the individual tests.

Correlations which may be of interest (between different parts of the test and between the test and the context).

V. Training of administrators and supervisors for the EGRA pre-test

In conformity with the recommendations made by MENFOP, it has been decided to use student teachers as administrators and therefore 34 of the best future teachers were chosen among those being trained at the

CFPEN training centre. A training workshop took place at the beginning of March 2012, immediately before the test took place. Detailed descriptions of each of the sub-tests were given, notably an analysis of the different possible responses that pupils may give. In addition to familiarisation of participants with the

EGRA test, the training also covered the administration of the test and safety questions. It included practical sessions with pupils of the school level to be tested. Thirty administrators were selected (10 to manage each level evaluated) and the remaining four were to be used as substitutes in case of absence. Twenty members of

MENFOP personnel, whose role was to supervise the administration of the test, also received training.

VI. Administration of the EGRA pre-test

The pre-test involved 150 pupils of the 2nd, 3rd and 6th year classes in four schools in Djibouti city on 6

March 2011. Two were primary schools with 2nd and 3rd year classes, the other two were schools including

6th year pupils. The pupils were chosen among the schools in the following way:

Table 3: Class numbers, EGRA pre-test pupils

Schools Class level Number of pupils

Primary School 1

2nd year

3rd year

Primary School 2

2nd year

3rd year

Middle school 1 6th year 25

Middle school 2 6th year 25

Around 5 MENFOP personnel were present in each school to observe and supervise the administration of the pre-test. Even if this number might appear excessive, it allowed competence building within MENFOP.

VII. Analysis of the pre-test and development of a report

In the weeks following the test, the MENFOP assessors undertook the data entry. Analysis was performed using the pre-determined list of descriptive statistics and correlations to be made. A report was produced with the detailed changes that would need to be made to the pre-test (Fast, 2011).

Table 4: List of suggested changes after the pre-test

N° Observations:

Test book

1 General comments. The font size of the letters could be increased.

Use of a grey background, and use of bold and italic, proved useful in distinguishing instructions from the texts to read to the pupils.

The small A5 test book proved to be easier, as was the reduction of the copies.

The instructions should be shorter and simpler.

Use of a code instead of the name of the pupil.

Increase in the number of questions from 3 to 6 to improve the validity of part 1.

An indication once a sub-test is finished would be useful.

Acceptable responses (name of a letter rather than the sound of the letter) should be determined.

Part 9 requires modification.

Training for test administrators.

1 Choice of administrators Change of administrators: the student-teachers were not sufficiently at ease with the children and could not take the necessary linguistic decisions.

2 Pronunciation questions Give more detailed information regarding acceptable pronunciation for the responses in the following sub-tests: recognition of graphemes, reading common words and made-up words.

3 Classification of pupil responses

Many errors were made in part 2 and in the small parts requiring timing. More training is required.

4 Use of a watch Further training required.

A stop-watch would be useful.

5 Scoring and evaluation Despite the training, there were problems with the scoring and the quantity of missing data. Allow pupils to write the dictation in the test book, so that the correction of this aspect of the test can be checked.

Concerning the procedures to be followed for the test, it has been decided that administrator training should be extended from 3 to 5 days. It was also highlighted that these should be experienced teachers rather than student-teachers.

VIII. Conclusions regarding the pre-test

The EGRA pre-test achieved its objectives with regard to suggesting improvements to the tool as well as to the test procedures. The descriptive analysis clearly shows the usefulness of the potential results from a test like EGRA given that it provides information that can lead to action that can improve reading.

Chapter 3: Preparation and implementation of the Test.

After experimentation with the tool, preparations began with the development of the reference line. A new committee was appointed by the inspectorate general to supervise all aspects of planning and preparation of the tools. A detailed schedule with clearly defined responsibilities and time-lines has been prepared. This planning tool has been regularly updated during the test preparation period (see annex 3).

After the pre-test, changes have been made both to the preparation process and administration of EGRA and even to the corrections made to the tools themselves. The main changes made to the tools are as follows:

1. Instead of the pupils writing the dictation on a piece of paper so that the administrator can correct it, a space has been created on the back of the test book so that the sentence can be written there. This gives the option of being able to correct separately the dictation that posed problems during the pre-test.

2. The number of questions in section 1 has been increased, from 3 to 6.

3. In section 3, only the sound of the letter must be accepted, and not the name of the letter.

4. The test book instructions have been simplified.

5. Instructions have been added to the test book so that the administrators can alert the children when the sub-test is finished, so that they can pass to the next stage.

The final version of EGRA contains the following/

Orientation towards reading: (6 questions)

Identification of initial sounds: This test measures the ability of the pupils to hear and isolate the first sound of a word. This is generally considered as an essential pre-requisite for reading competence. One of the test words which gave difficulties has been replaced (10 sounds).

Knowledge of letters and sounds: This test measures the ability of pupils to recognize letters and groups of letters. It tests the ability of the pupils to pronounce only the sounds represented by the letter and not the names. This is a timed test of one minute (100 graphemes).

Reading of common words: This test measures the ability of pupils to read, with prevision and fluency, commonly used words. It lasts for one minute (50 words).

Reading of made-up words: This test measures the ability of pupils to recognize words which respect

French linguistic rules, but which do not exist in the French language (50 words).

Reading and comprehension of a read text: This test measures the ability of pupils to read a short passage of about 63 words with fluency.

Listening comprehension: This test measures the ability of the pupils to reply to six questions on the part of the text that they have read.

Oral comprehension: This test measures the ability of the pupils to understand a simple oral story of 7 sentences and to respond to six questions including literal and inferential questions.

Writing: This test measures the ability of pupils to write, from dictation, a simple sentence and evaluate the content and spelling as well as knowledge of the rules of writing. Use of capitals and spacing of words are also evaluated. The pupil questionnaire also deals with demographic data, socio-economic status, and other possible home and family factors that could influence reading performance. To facilitate use of the test book, the test was photocopied double-sided; the layout showing how the book should be assembled (see annex 2).

I. Schedule for the initial survey

The initial EGRA test was originally planned for May 2012, so as to measure French learning at the end of

2nd, 3rd and 5th year classes. However this was later delayed and programmed for the start of the following school year, in September-October 2012.

For this reason, children starting 3rd, 4th and 6th year classes were selected for the sample, as they had completed 2nd, 3rd and 5th year in May 2012.

II. Sampling plan

Administratively, Djibouti is divided into six regions which are split into 10 school districts of which 5 are found in the capital Djibouti city.

Table 5: Regions of Djibouti

Districts in the capital Districts outside of Djibouti city

Djibouti 1 Ali Sabieh

Djibouti 2 Arta

Djibouti 3 Dikhil

Djibouti 4 Obock

CFPEN Tadjourah

The target population for the EGRA is made up of pupils of 2nd, 3rd and 5th year classes during the 2011-

2012 school year.

Table 6: Djibouti pupil population

Region 1st year 2nd year 5th year Total

Classes Pupils Classes Pupils Classes Pupils Classes Pupils

Djibouti 1 32 1532 31 1497 37 1736 100 4765

Djibouti 2 27 1284 26 1098 34 1580 87 3962

Djibouti 3 28 1801 31 1844 57 3270 116 6915

Djibouti 4 30 1788 28 1583 38 2166 96 5537

CFPEN 20 1160 20 1130 22 1147 62 3437

Arta 16 546 16 545 13 554 45 1645

Ali Sabieh 22 883 22 819 21 788 65 2490

Dikhil 27 1102 28 1184 26 1051 81 3337

Obock 14 333 13 292 13 272 40 897

Tadjourah 26 724 26 703 23 787 75 2214

Total 242 11153 241 10695 284 13351 767 35199

Given that the organisation of the test was delayed until the beginning of the school year 2012-2013, the sampling became rather complex as it had to take into account the following factors:

For the pupils in 2nd year in 2011-12:

o Those repeating a year would be in 2nd year in 2012-13 o Pupils in 3rd year in 2012-13

For the pupils in 3rd year in 2011-12:

o Those repeating a year would be in 3rd year in 2012-13 o Pupils in 4th year in 2012-2013

For the pupils in 5th year in 2011-12:

o Those repeating a year would be in 5th year in 2012-13 o Middle school pupils in 6th year in 2012-13

In addition, the target population was spread over the different levels, so sampling was performed according to a practical method. A sampling method in several groups was used to select the pupils. Overall sampling was split into two stages: first the class was selected, then the pupils. For selection of the sample, the strata defined by MENFOP for learning in 2nd year were used. The size of the sample was fixed at 600 pupils per level, as recommended by the EGRA tool. “It seems prudent to conclude that the sampling of a number of pupils between 410 and 600 per primary year concerned (in whole numbers) should be sufficient for all cases.”

III. Selection and training of test administrators

To avoid certain difficulties encountered by the administration during the pre-test, it was decided to use pedagogical consultants and regional directors rather than future teachers. The justification of the choice of these experienced educators is that they would be better able to interact more easily with the pupils and are more experienced in questions of reading and literacy. For the training, 75 trainers were chosen by

MENFOP. Three reading specialists from FHI360, in collaboration with MENFOP managers and personnel from Projet Aide, organised, for the data collectors, a week of training and familiarisation with the EGRA tool and the range of responses that the children might give.

Table 7: Training programme

Sunday

30 September

Monday

1 October

Tuesday

2 October

Wednesday

3 October

Thursday

4 October

Introduction

Workshop goal

Programme

Organisation (into administrators / supervisors)

Historical summary of the EGRA technique

Summary of EGRA tools

Explanation of workshop rules and procedures

Detailed explanation of each

EGRA sub-test

Demonstration of

EGRA

administration (by the trainers)

EGRA simulation in pairs

Clarification of administrator and supervisor responsibilities (by group)

Presentation of responsibilities by group

EGRA Simulation

N°1 (in groups with children of the target age)

Discussion of the simulation results

EGRA Simulation

N°2 (in groups with children of the target age)

Discussion of the simulation results

EGRA Simulation

N°3 (in groups with children of the target age)

Discussion of the simulation results

EGRA Simulation

N°4 (in groups with children of the target age)

Discussion of the simulation results

Final comments and questions on test administration

Organisation of material for the test

Overview of logistic aspects of administration

The practical part of the training consisted in placing each administrator in a real situation to practise their position relative to the pupil, how to use the chronometer and manipulate the answer book. Thus, all remarks and observations concerning performance during the test period were note on an evaluation form. (See

Annex N)

On the basis of these results, a total of 40 out of 75 training participants were selected. The administrators were assigned by region, but care was taken not to send a consultant to a region where he had teachers under his supervision, so as to avoid any bias. The supervisors were equally charged with checking the conduct of the test.

IV. Test period

The field work took place during the week of 4 October 2012. The schools had been alerted beforehand, to provide a room dedicated to the national test. 16 observations by the supervisors were carried out in Djibouti city using forms provided and 27 observations were carried out in the regions (see annex 4).

V. Data entry

Data entry, carried out by experienced Ministry personnel, followed the collection schedule and was carefully quality controlled, notably by using software to limit the possibility of error. A daily quality control of 10% of the data was undertaken, thanks to a double verification of the data entry.

Chapter 4: Results of the EGRA test

This chapter presents the results analysis of the pupil questionnaire. In a first step, the Cronbach alpha coefficient for the tool was calculated to measure the reliability of the test at all three levels. As a general rule, an alpha coefficient of 0.80 is considered as being good and a coefficient of 0.7 is generally taken as the minimum acceptable. The results show strong internal coherence with a coefficient of 0.88 in 2nd year, 0.92 in 3rd year and 0.90 in 5th year. As shown in the table below, the tool is sufficient reliable for analysis to proceed.

Table 8: Reliability of the EGRA tool

N2 N3 N5

Test subjects p value Item-rest correlation

If alpha removed p value

Item-rest correlation

If alpha removed p vp value

Item-rest correlation

If alpha removed

Section 1: Orientation towards reading

0.49 0.48 0.89

0.53 0.52 0.92

0.63 0.35 0.91

Section 2: Identification of initial sounds

0.38 0.53 0.89

0.39 0.49 0.93

0.40 0.35 0.91

Section 3: Recognition of sounds

0.17 0.70 0.87

0.23 0.67 0.91

0.31 0.59 0.89

Section 4: Reading of common words

0.16 0.80 0.86

0.28 0.89 0.89

0.42 0.89 0.87

Section 5: Reading of made-up words

0.11 0.74 0.86

0.21 0.84 0.90

0.36 0.86 0.87

Section 6a: Reading text 0.18 0.76 0.86

0.30 0.86 0.90

0.47 0.85 0.87

Section 6b:

Comprehension of a read text

0.10 0.72 0.86

0.15 0.77 0.90

0.24 0.76 0.88

Section 7: Listening comprehension

0.17 0.59 0.87

0.24 0.71 0.91

0.39 0.57 0.89

Section 8: Writing a sentence

0.18 0.73 0.86 0.25 0.75 0.91 0.40 0.80 0.88

Mean 0.21 0.67

0.29 0.72

0.40 0.67

Reliability coefficient (Cronbach alpha coefficient)

0.88 0.92 0.90

Demographic information

The EGRA results were obtained from 498 girls, 598 boys and 13 pupils whose gender could not be identified, a total of 1109 people tested, distributed as follows:

Table 9: Number of pupils tested

Girls Boys Total

2nd year 161 212 373

3rd year 185 193 378

5th year 152 193 345

This constitutes an excellent rate for the sample of pupils participating in the test.

All six regions of the country were represented according to the following proportions:

Table 10: Pupils tested by region

2nd year 3rd year 5th year Total

Djibouti city 261 251 240 752

Arta 19 20 17 56

Ali-Sabieh 28 30 30 88

Dikhil 40 39 24 103

Obock 13 20 17 50

Tadjourah 20 21 19 60

TOTALS 381 381 347 1109

I. Descriptive statistics

The section above gives the results by sub-test. An early stop rule was applied so that when a child failed to give any correct response in the first line, the assessor did not pass to the following sub-test. Two reasons justify the interruption of the sub-tests. First the early stop rule avoids any frustration among the pupils for whom the lack of motivation could affect their work on the later sub-tests. Second, the early stop rule is also an approximate evaluation of a zero result for the entire test.

Sub-tests 7 and 8 did not follow the early stop rule. In consequence, the results from these sub-tests apply to the total pupil sample and report the percentage of correct responses to the total number of exercises in the sub-test. For each sub-test, the results are presented both in terms of average results and in terms of percentage zero results for the pupils. The reason is that the zero results for pupils represent the number of pupils showing no proof of the competence tested while the average result shows the mean rate of acquisition of a given competence.

In the Djibouti EGRA test, if one takes into account the sub-tests and the levels of class together, the highest proportion of early stops appeared at the level of sub-test 1 (Orientation to print), sub-test 5

(reading of made-up words), and sub-test 6a (reading a passage aloud). The standard deviation (SD) is also shown. This is a measure of the way in which the results are scattered. A low standard deviation signifies that all the results are close to the mean, a high standard deviation signifies that the results are scattered. So as to continue to give an indication of how the results are scattered, the distribution of the results is also given relative to pupils having obtained between 0 and 25%, between 26 and 50%, between

51 and 76% and 76% plus. The results are reported by class and by gender. The results by region are found at the end of this chapter.

Sub-test 1 - Orientation towards reading

This sub-test represents the first step in learning to read. It consists of questions designed to make them aware of writing conventions:

if the children know where to start and where to finish reading on the page;

if they know what to do once they arrive at the end of the line, etc

After the results of the EGRA pre-test, the number of questions in sub-test 1 was increased from three to six.

Table 11: Percentage of correct answers to 6 questions

Table 12: Distribution of results as % for sub-test 1

G B Total SD

2nd year 49.6 48.0 49.0 29.2

3rd year 55.5 50.8 53.0 28.4

5th year 63.2 63.6 63.4 26.7

% 0-25 26-50 51-75 76-100

2nd year 23 32 22 23

3rd year 19 31 24 25

5th year 10 29 22 40

Table 13: % of zero results for sub-test 1

As expected, the overall results are improved in all school levels, while the percentage of children scoring zero is lower. Overall, the percentage of zero scores was low in this sub-test, with no 5th year pupil scoring a zero.

Sub-test 2 - Identification of initial sounds

Pupils were asked to identify the initial sound for ten common words of one syllable. For each question of the sub-test, the assessor read each word aloud twice before asking the pupil to identify the sound, and then noted if the response was correct, incorrect or if there was no response.