Skip to main content

Full text of "ERIC ED133856: Possibilities and Limits of Evaluation Research in Educational Planning. IIEP Seminar Paper: 11."

See other formats


ED 133 856 

BA 009 110 





Holler, W, 

Possibilities and Limits of Evaluation Research in 
Educational Planning, HEP Seminar Paper: 11, 
United Nations Educational, Scientific, and Cultural 
Organization, Paris (Prance). International Inst, for 
Educational Planning* 

lip,; A contribution to the HEP Seminar on "The 
Evaluation of the Qualitative Aspects of Education" 
(September 30-October 4, 1974) ; For related 
documents, see EA 009 106-111 and EA 009 113-116 

MF-$0,83 Plus Postage, HC Not Available from EDRS, 
♦Developing Nations; Educational Change; Educational 
Innovation; *Educational Planning; Educational 
Policy; *Educational Quality; Educational Research; 
♦Evaluation Needs 


The evaluation of the qualitative aspects of 
education is not as simple a task as the evaluation of quantitative 
aspects. The research frequently fails to offer the educational 
planner concrete guidance in the area of educational quality. The 
planner must nork in a framework of conflicting criteria requiring 
political decisions, with limited resources, and within a set pattern 
of rules and policies that cannot be changed rapidly. Many factors 
are simply out of the scope of the pleoiner^s influence. The problems 
with educational planning are exacerbated in developing countries, 
and the difficulty with implementing educational innovations 
increases in these countries, (Author/DS) 

♦ Documents acquired by ERIC include many inf 

♦ materials not available from other sources, ERI 

♦ to obtain the best copy available. Nevertheless 

♦ reproducibility are often encountered and this 

♦ of the microfiche and hardcopy reproductions ER 

♦ via the ERIC Document Reproduction Service (EDR 

♦ responsible for the quality of the original doc 

♦ supplied by EDRS are the best that can be made 
^^^^(♦♦♦♦^ ♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

ormal unpublished 
C makes every effort 
, items of marginal 
affects the quality 
IC makes available 
S) , EDRS is not 
ument. Reproductions 
from the original, 















HEP seminar paper: 



W. Moller ' 

A contribution to the HEP Seminar 
on "The evaluation of the qualitative 
aspects of education" 
30 September - 4 October 1974 

(established by Unesco) 

0 7-9, rue Eug&ne-Delacroix, 75016 Paris 
■ H 

(c) Unesco 1975 

1 2 



The opinions expressed in these papers are 
those of the authors and do not necessarily 
represent the views of the Institute or of 



Quality of education Is an elusive , concept not p^K-na-io 
to clear comprehensive description. In a previous HEP sorrinar ' 
on the aualltativo aspects of educational planning? man^" 
distinctions for determining quality of an education system were 
considered such as quality of the process of education taking • 
piece In a given Institution and quality of the product of su<:h. 
education; education considered primarily as an Investment with 
lesii emphasis on Its humanistic aspect as a good In Itself or 
vice versa depending on the ntate of economic development, or a»;aln, 
the democratic Ideal of equality of opportunity contrasted with 
the needs of manpower development. Of late, education In regard 
to employment In developing countries has received attention 
leading to quite revolutionary proposals, - 

These are Issues of continuing debate In which values • 
and aims are determined xore often by political decision than 
a process of experimentation, and measurement followed by adjustme / 
and change. Change in political leadership often means change in 
eduoational policy with the result that the planner must resort 
to strategies wHlch will allow hin; to maintain the fundamental 
objectives of his plan and to pursue reform • Such strategies are 
well described by McKinnon (Realistic Educational Planning Unesco 
IlEP 1973)' In this context of policy formulation, research and 
evaluation have so far contributed little to practice in particular 
in developing countries, and it is not difficult to see why. 

In part, it is due to the lack of people doing research; 
in a much larger measure however. It is due to the frecuent 
inconcluslveness and contradictory Interpretations of research 
results. Thus, Professor Thorndlke, author of the lEA report on 
Reading? Comprehension Education, in Fifteen Countries says flatly, 
''It must be confessed that the. results of the study provlc'e little 
guidance for the Improvement of the educational enterprise." In 
the J^^lence Education Study the conclusions begin, "Although the 
fi: .. j;3 of the multivariate analysis will be disappointing to 
teaciu;rs and others concerned with the learning process more 
efficient, • the explanation lies in the very nature pf learning 
and schooling.'- In the study on literature education one reads, 
'in sum, although the lEA study leaved' uncertain the effects of 
school on the ability of students literature and on. the 
students' interest In reading ' literature in their spare time, it 
points to the potentially profound effect of schools on the 
patterns of questioning and response of students 

These .cuotations have not been selected with an insidious 
intent of discrediting 'the lEA stuc'y. They are meant to show why 
such research and evaluation has so far had little influence on 
the format ion of policy. 


~ 2 - 

In order to bo of use as a tori for the planner an 
evaluation should deal with those aspects of the education system 
over which he has some measure of control or direct Influence. 
Thus, for example, under a policy of compulsory education and 
equality of opportunity he has no influence on the home background 
of the pupils In the system. And yet this Is recognized as. a 
most Important factor In many studies on achievement He may of 
course stretch the notion of eouallty of opportunity to the extent 
of proposing special redemptive schemes, but It Is evident that 
In the poorer countries such a policy would hardly be feasible 
when larse proportions of the school age population do not even 
have a school to go to. 

While student performance on standardized tests l3 the 
most reliable direct measure of quality of learning In the 
classroom It Is not an aspect on which the planner or the 
external donor agency ^has such direct Influence This Is another 
reason why researc^h an^^ evaluation on student performance has so. 
far had little share in the planning process Even If Improved 
performance could be imequlvocally attributed to a particular 
combination of measures adopted and methof^s used It does not 
follow that some other (and perhaps Ic'SS costly) combination 
could not have been equally effective 

The answer that further research Is needed Is not much 
help In such a situation since the planner's main task Is the 
allocation of scarce resources over a period of a few years, and 
such period usually begins with the next school year since annual 
adjustments to any long term development plan are Inevitable. 
Not only are resources limited, but the pattern of their 
distribution tends, to be fairly rigid - teacher salaries usually 
account for the largest part of the recurrent budget and there 
Is little scope for costly experiments long term plan will 
of course provide for changes and Innovations, but If during Its 
Implementation a budget squeeze becomes necessary allocations for 
equipment, teaching materials and maintenance are the first to . 
be cancelled. Moreover, since politicians will rarely sacrifice 
quantitative targets, resources will have to be spread even more 
thinly. In such a context evaluation assumes a different 
connotation best Illustrated by Arthur Lewis' apodlctic statement 
at the previous IIEP se. .Inar, "The problem Is. not to measure but 
to evaluate.*' For ,the planner having to <^eclde the best possible 
distribution of resources, reliable methods, of appraisal and 
cost effectiveness analysis are of greater Importance than 
psychometric research. 

.He may of course resort, to the well kiiown .flndln/-, of the 
TEA study that within reasonable ranges there Is no slt^rilf leant 
correlation between class 5>lze and student performance In certain 
subjects and therefore propose Increases In class size In fact, 
however, he has always done so unc'er pressure of pulllc demand, 
and In many developln{J).countrles stark reality Is beyond such 
reasonable range when classes of cO to 100 pupils are -jrowde-^ 
into rooms built at most for 40 or when 4 classes of 40 are 
accommodated together in the school assembly hall. 

5 . 

In such cases administrative measures such as double shift 
operation of schools or arbitrary reduction of the number of 
hours of attendance per week are palliatives whose effects 
would warrant invest if;at ion. 

In general, it may be said that in conditions of 
scarcity of resources - buildings, equipment, uooks and other 
educational materials, refined statistical , analysis of student 
performance in standardized tests will be of little use to a 
planner intent upon improving the quantitative basis for oetter 
quality in performance. In developing countries where education 
systems have been created almost from scratch within the 
decade evaluation of a more simple kind would however, be of great 
use to the administrator (and eventually the planner) in 
determining- whether the system functions reasonably well within its 
presently established rules and procedures- ' . 

This of course is something entirely different from the 
subject of the contributions to the seminar, but in practice is 
a most important aspect. In his c >ntribution to the seminar, 
.Professor Levy says: "Each nation must^ find ways of bringing 
its teachet's up to date on the new subject matter, methods, and 
ideas used in the new curricula. Secondly, there Is a need to 
secure adequate and timely^ supply of all equipment and curriculum 
'ingredients' which are needed for the implementation of the 
programme. Thirdly, there is a need to introduce apipropriate 
changes in different branches of the educational mechanism • 
which may effect the implementation of the programme; the national 
examination system, university entrance requirements, supervisory • 
staff activities, and * enrichment' education programmes should be 
adopted to the needs of the new programme. Finally, care should 
be taken to update the new programme whenever the need emerges." 

One can only agree and add the remark that to meet all 
these conditions is in itself a difficult and. long range task 
•"of planning 

Training of teachers is an important aspect of planning 
and more effective, less costly methods will always be welcom„e. 
For the planner in a ministi?y, however, the very existence of a 
teaching force with considerable acouired rights and privileges 
is a problem when it. comes to introducing innovations- Not only 
is it notoriously difficult to make research results percolate 
.to the level of the practising teacher, but there are many points 
on which research cannot be of much help even on a fairly large 
run. Such are problems of service conditions, in particular 
where teachers well organized in unions and/or have civil servant 
status - what is needed here are political decisions. 

. 4 - 

Improvemotit of or.'.^anlzatlon and adnilr^lsti^atlon on the 
central or pro\lnclal ..overnment lovel or f»ven In the Individual 
schools is a planner's task to the ertent that it concerns long 
range issues such as decentralisation and delegation of authority, 
simplification of existing procedures and administrative rules. 
In many of the poorer countries there are excessive controls on 
expenditures, well intentioned, but with dam.aging results. Teachers 
who have not been paid for weeks or months are unlikely to be 
enthusiastic for innovations usually roquirint greater effort on 
their part. Many schemes' of fundamental . education failed in the 
fifties and early sixties because too much was 'being asked of the 
village teacher Equipment often gathers dust on shelves or is 
neatly locked up. This is often because teachers may not be 
sufficiently trained to use it,. but if there is a nale by which 
whoever is in charge of the equipment' is also financially rcdponsiole 
for it, he will hesitate to let it be used by inexpert hands. Often 
too equipment has been bought with foreign help for which there is 
no repair facility or spare parts may be unobtainable because of 
lack of funds in the recurrent budget. 

In many developing countries building maintenance is poor 
Ministers intent on expansion or with meeting social demand tend noi 
to pay much att^ention Lo this aspect of maintaining a sound- 
material basis for the process of education that is supposed to go on. 
The allocations mentioned above for maintenance are usually the first 
to be ^t. 

> The availability and actual presence of textbooks In the 
classroom is another indicator of the f,ood or bad functioning of 
the system. Such a problem precedes that of the actual use made 
of those materials in the classroom* ■ Audio visual^aids may also 
be used unimaginatively as merely an adjunct to purely expository 
tall, by the teacher. 

• •• .. ► 

The other extreme is reached when In an attempt to introduce 
new methods of assessing pupils* pro^jress teachers are asked to fill 
in such masses of questionnaires that the whole exercise becomes 
perfunctory. • Enthusiasm may. also prevail over good sense in giving 
a one week course on programmed instruction and then lotting the 
teachers loose to devise their own- programmes -New curricula in 
science have been written, formulating objectives (though perhaps 
not with thfe detailed clarity required) emphasizing learning by 
doing but have. ended up in an attempt at encyclopedic coverage 
that 1^ bound to defeat the purpose of the new curriculum 

These examples of what can and does happ-en are given to 
show the Importance of improving the existing process by means more 
readily accessible ^to the planner To evaluate the functioning of 
the process with regard to these practical and administrative 
aspects the planner nef*ds to have other measures than pupM 
performance. He needs reliable first hand reports on what is going 
on in the schools and ^district administration 


Data collection and analysis usually noed iinprovr^ment . 
Improvement here moaris that oftVctive and reliable fi^cures on 
enrolments, teaching force etc woul^^ be available at least by 
the end of the' current school 'year, but there are many countries 
in which this is not the caae. In some the delay between 
collection and availability is even increasing. Publication of an 
analysis of enrolments by gra^ie an^^ age for a period terminatine 
five years before may be an interesting contribution to the stRtistioal 
literature^ but is not of interest to anyone having to plan a 
project for the construction of schools during the next five years 
Reliability of figures and estimation of error need much greater 
attention. This has often been said, but recent experience provides 
examples such as that of four different foreign expert missions 
having to work with four different estimates of gross population 
growth ranging from 1.8?^ to 2.8* in a period of five years. 

If evaluatioii research on the above aspects can help the 
planner to obtain a n-iOre realistic picture <^X the functioning of 
the process of education in the system he is to develop he will have 
a soun^-er basis for appraising the feasibility of new projects and 
the introduction of Innovation and reform. 

On a long term. view he will have to be concerned with 
arrangements which will rive the opportunity for permanent 
education. In this respect he w uld bp interested in findings aoout 
the intensity and duration of eriucation It would be an innovation 
indeec'. if contrary to the general trend one might be able to reduce 
the period of compulsory general education thus freeing greater 
resources for those who might wish to return to education after a 
period of some practical wprk. In technical and vocational 
education such arrangements have already, been made, though not 
with'as radical a measure as suggested here. Moreover there are 
count'-ies where there is ^ in theory at least - compulsory education 
of nine years, but where available facilities do not even permit 
two years of educatl'.'>n for many of the rural population. 

, j • 

In many rleveloping countrie.s hi^-.hor 'education absorbs an 

undue share of natioi^al resources and the trend now is to favour 

investment in basic education. But i^iven the existing 

arrangements - being very favourable often onre a student has 

reached university - what can the planner conc'emed with the 

longer- term do if he has to work in the framework of democratic. 

liberties and equality of opportunity In order to obtain some 

self regulation of the system he mij^ht recommend a policy that 

maximum educational ambitlnjn-^shaal rl be acco mpanied by the maximum 

risk. Such a policy would of course be very unpopular in the 

present climate of opinion and evaluation of its possible 

effects would be welcome. 

■To sum up: in evaluating the qualitative aspects of 
education the planner has to work in a framework of conflicting 
criteria requiring political decision.s, limited resources recuiring 
cost effectiveness araly.sis, and within a set pattern of rules and 
procedures- which must be rendered eff eotive and cannot^ be chan£:;ed 
rapidly. He has to deal with *a teaching force whose opinion he 
dlorer-ards at the peril of -.never' seeing beautiful new schemes applied 
in practice (examples of this abound even in developed countries). 
Ho will be even tnore rnod(»st when he realises how strong are the 
factors influencing: quality which are outside his direct or 
indirect Influence. 


The following extracts from a report on a short visit 
to three schools of a completed World Bank Project illustrates 
what type and amount of infomation, and useful to a donor 
agency, can be collected in a few days; 

Operation of th^ Schools (January 1972) ^' ' ' . 

Since August 1971, date of the last supervision, there 
have been few quantitative changes Enrollments have increased 
to a total of 19,206 of whom 5,92^ were girls .Problems with 
overage pupils persist and have led the project unit to limit 
intake in the forthcoming school y? ar to the first grade and 
to pupils un^ er l4 years Five more schools are to open by 
March with enrollments Bogota 960, Villavicencia 520, 
Neiva 52Q, Tunja 400 and Pereiva 400 The school in Popayan is 
to open in September with 400 pupils 

Avera£,e teacher-stud^^nt ratios, . dropout and retention rates 
have not changed significantly As a measure of efficiency, the 
latter are in any' case open to doubt since thoy presuppose 
locketep progression Since the schools use a system of "setting" 
students by ^performance in particular subj-ficts and give remedial 
courses during holidays ordinary repetition rates are not very 
meaningful As for dropout, the rate of 5% last reported Is > 
comparatively low However, one school reported 8^ from grade 
1 to »2 and a higher rate at higher grades owing to false pretensions 
at intake. This will not happen again as all applican^ts, almost 
twice as many as available places, have to take a multiple- choice 
entrance test 

Juog^d by the criteria t|iat secondary education must be 
diversified, that sorpe more practically oriented instruction be 
given and that rievi methods of Reaching and employed 
the schools are a success Th^y are, however, s >mewhat isolated 
in the Colombian -system of education and final evaluation of 
success Judged by employment or. access to^ another education will 
only be possible when the first cohorts have passed through the 
system. Meanwhile,' there is na doubt that the schools have support* 
' from their c-^mmunity their teachers And particularly the 
menGging staff are working enthusiastically in preparing programs, . 
teachins materials, introducing b.etter evaluation and testing 
methods The supply of teachers is quantitatively adequate, but 
oualifications vary consi-derably According to a recent statistic, 
° out of 971 teachers employed, C60 were "licencJiados" mainly^ in 
languages, matherriatics, sciences and social sciences Vocational 
and technical teachers continue to be scarce and few have adequate 
qualification: most of them lack industrial experience There is, 
for example, only one fully rualif led instructor for technical 
drawing and 27 have only a bachillerai^ tecnico " 'This familiar 
oituation is not likely to change quickly^ evert- though there were 
more than 2,000 applicants for the 65O posts to be filled in .the , 
coming, school year 


Inspection of some applicants' files showed that thrre are now some 
candidates with reouisite industrial experience, but their number 
was not sufficient ^to meet actual needs Moreover, some 800 of 
these applicants were univer-*^Vy graduates without teaching or 
other practi^l ex^rionce* 

'Training for IMEN teachers was severely h^mportd during the 
past acac'emic year owing t-^ disturbances at the universities 
Various^forms are used such as introductory eemjlnarsiof 15 days 
duration, specialised upgrpdin^ j^ourses, meeting of subject 
specialists or department diijectors ^d. post graduate courses at 
universities. The main load^ is carried by the pedagogic proup 
During"*- 1971 some 52 Such courses were given for 'this traininc, 
involving JSome' 1,250 teachers Even so, of the 971 teaohers in 
service in December 2^6 hi(\ n^t received t^ie introductory seminar 
The training prorjra'm for 1972 was aid to be of a similar-' scale 
thouch no rletailed fin^ares were provided ' ' a ^ * 

The distr^ibytl^on of textDook^ is uneven - some sc>iools have 
sufficient for some subjects to supply eaeh student, in others, 
texts are shared by groups or again are neatly stacked in. the central 
library An enormous effort is made in schools to p'rovide 
their own course material, mainly in^'the form of worksheets 
supplemented by bibliographical references intended to stimulate 
individual 01? grblipl effort and learning .It would at some stage 
be worthwhile to investigate ^to wfiar extent first or second year 
students actually pursue 'such references VJ^^Al^aids are used, 
extensively, but do not necessarily preclude ^^^fgfily expo9>tbfy 
talk by- the teachpr ^ 

Since* July und6r guidance? from a professor" of the 
National UfTiversity, programmed instruction tests have been. ^ 
prepared at a number of schools They are said to have ,beep 
enthusiastically recei^^ed by , both teachers and students A 
cursory Review of some texts was made: the gospels, colonization . 
and liberation,' elementary theory of sets, photosynthesis., 
commercial organization, . electrical circuits, concordance between 
adjective and. substantive, ■ description and its formsr*etc Here^ 
again enthusiasm seems to Kave prevailed over careful selection' 
of topics amenable ,to such progi^mming" techniques As a repult, 
' many t.exts contain lunbiguities and som^ downright, howlers, others 
mainly present a modem form of r^te learning This^ission 

'nted out the 'dangers inherent in such an indiscriminate, approach 

The pf^dagogic unit has prepared ji^i^ps f or^ teachers to 
ensure some standarization of method^ and content' of study programs 
for each subject area These guides should on the whole, be quite 
useful to teachers They use tj\e modern approach of defining 
objectives for each unit of I'oarnins' , 



In the guide for science one flndc the foliov/lripr: that the 
student should become able to acqulrt* knowledf^e and ability to 
understand or explain certain aspoctri ol a subject and to apply 
such knowledge; to acaulre ^mowledi^e and evelop attitudes 
enabling him to Interpret the Intellectual process In formulating 
hypotheses; to become able to Judge the result of his Investigations 
In a logical manner as a means to enfjufo certainty of his conclusions. 
There fo'ilows In most cases a list of Itoms which closely resembles 
that of tn old fashioned examination syllaburs (as witness) the 
ccnteiit of the compulsory 8 semester study piogram55 for science for 
all students 

Whereas the three- remesters cover subjects such as 
fonns of life, soils, General properties of matter and the handling, 
of simple laboratoi'y rqulpnlent, semester 4 raes straight to 
radioactivity, atomic theory of matter and measurements to prove 
the theory, molecular novements, heat un^'er which electric static 
(Hfid mafgnetlc fields appear somewhat Incongruous, to wind up with 
I'adlant energy-' both soiond and the whole electromagnetic spectmom 
All- this In an l8 week semester of ? periods a week Semester 5 
deals with the earth, semester 6 goes on to the origin and 
evolution of life, the moon and problems of getting there, the 
solar system and the universe Semefjters 7 and (3 f^lve biology In 
equally comprehensive manner This attempt at encyt lopedlc 
coverage Is liable to defeat the declared purpose of learning by 
doing and to Infiuce a return to purely expository methods The 
mission pointed this out and suggested a careful revision and 
pruning of the course 

j Evaluation of stur^ents* performan.e Is.^lven much attention 
The gulr^es provide suggestions how to c -instruct multiple choice 
answer tests Students are classified in three levels of 
performance Judged for 75^ on the basis of individual or group 
work and for 25% on the result of an end of semester test 
Tn one school, every teacher is given a 20 question questionnaire 
for each student in '^ips charge to be filled in each weel^ The 
questions cover work done, assiduity and also attitudes (per week) 
Tho result is, of course, that the questionnaires are filled in 
Ir; a very perfunctory manner % 

11 '