THE ASSESSING OF LANGUAGE TEACHING MATERIALS

Published By Melissa (1507042035)

Evaluation in Language Teaching

Over the last two decades, there has been a growing interest in the purpose and methods of evaluation in language teaching (e.g. Sheldon, 1988; Alderson & Beretta, 1992; Weir & Roberts, 1994; Ellis, 1997,1998). In the literature, the term evaluation is used in a number of different ways and based on the writers’ purpose, various definitions have been proposed. One of the most ‘workable’ definitions of evaluation was provided by Richards et al. (1985:98), who described evaluation as ‘the systematic gathering of information for purposes of making decisions’. Although this definition may seem broad, it is practical as it can be applied to any component of the language curriculum (needs analysis, objectives, testing, materials, teaching, or the evaluation process itself). Evaluation plays a crucial role in curriculum development as it allows instructors, material designers and administrators to assess the effectiveness and efficiency of a particular language program or any of its components and make informed decisions about how to proceed. Evaluations can be macro or micro in scale and can be carried out for either accountability or developmental purposes or both of these. In macro evaluation, various administrative and curricular aspects are examined (e.g. materials evaluation, teacher evaluation, learner evaluation), while micro evaluation focuses on the specific aspect of the curriculum or the administration of the program such as evaluation of learning tasks, questioning practices, learners’ participation etc. (Ellis, 1998). The evaluation in language teaching has been primarily concerned with the macro evaluation of programs and projects (Ellis, 1998), and most evaluation studies have been conducted in order to measure the extent to which the objectives of a program have been met, and to identify those aspects that can be improved. As Ellis (1998) observes, this kind of analysis is obviously of interest to teachers as they learn whether or not the goals have been accomplished and whether any changes should be made to the program. However, most teachers are less likely to be concerned with the evaluation of the program as a whole, and more concerned with the extent to which a particular textbook, or a teaching activity is effective in their teaching context. The evaluation of teaching materials may be done before they are used in the classroom in order to determine whether they suit the needs of the particular group of learners (predictive evaluation), or after the materials have been used in the classroom in order to evaluate their effectiveness and efficiency, and teachers’ and learners’ attitudes towards them (retrospective evaluation). This paper will introduce a systematic procedure for conducting the predictive evaluation of language teaching tasks

Language Learning Task: Definition and Componential Framework

Put simply, learning tasks are a means for creating the conditions necessary for the acquisition of language. Many definitions of language-learning tasks are found in the literature, but perhaps the most helpful is that provided by Richards, Platt and Weber (1985). They define a language-learning task as: “…an activity or action which is carried out as a result of processing or understanding language (i.e. as a response). For example, drawing a map while listening to a tape, listening to an instruction and performing a command, may be referred to as tasks. Tasks may or may not involve the production of language. A task usually requires the teacher to specify what will be regarded as successful completion of the task.” (Richards et al., 1985: 289) The definition above clearly highlights the key components of a task: (1) language input, (2) goals (a clearly specified outcome, which determines when the task has been completed) and (3) activities (what learners need to do in order to complete the task successfully). The interest in task-based language learning has been stimulated by psycholinguistic research, which suggests that learners have their own built-in syllabus, which is often different from the syllabus proposed by instructors (Ellis, 1998). Thus, the sequencing of the linguistic input designed by the instructor may not follow the order of the learner’s linguistic intake. Task-based instruction specifies in broad terms what language learners will communicate about and the procedures they will follow, but it gives learners more freedom in terms of the choice of language they use, allowing them to develop their knowledge and skills in accordance with their own interlanguage and order of acquisition. Since the mid-1990s, there has been a growing interest in the effects that the cognitive demands of different tasks may have on students’ performance and the restructuring of their interlanguage. One of the most comprehensive frameworks for the analysis of the cognitive characteristics of language-learning tasks was proposed by Robinson (2001, 2003). Robinson argues that successful task performance depends on the interaction of multiple factors that operate in three different dimensions: task complexity, task difficulty and task conditions. Task complexity refers to the cognitive demands (i.e. the attentional, memory, reasoning and other processing demands) that the structure of the task imposes on the language learner. Task difficulty refers to learner factors that may make a task more or less difficult. This includes affective variables such as motivation, anxiety and confidence and ability variables such as aptitude, proficiency and intelligence. Finally, the successful completion of a task also depends on task conditions or the interactive demands of tasks. Interactional factors include participation variables (e.g. one-way or two-way task) and participant variables (e.g. gender, familiarity, power and solidarity). In short, students’ task performance is likely to be influenced by the interaction of multiple factors across the three componential dimensions. Teachers and material writers can manipulate these variables either to allow learners access to an existing L2 knowledge base (a focus on fluency) or to promote form control in learners’ interlanguage (a focus on accuracy).

Predictive Evaluation

A brief review of the literature relating to materials evaluation reveals that, to date, the focus of attention has been more or less exclusively on predictive evaluation. There are two principal ways in which teachers can carry out this kind of evaluation. One is to rely on evaluations carried out by 'expert' reviewers. Journals like ELT Journal assist teachers in this respect by providing reviews of published coursebooks. In some cases (such as the Survey Reviews this journal provides from time to time), the reviewers identify specific criteria for evaluating materials. However, in reviews of individual coursebooks, the criteria often remain inexact and implicit. Alternatively, teachers can carry out their own predictive evaluations. There are numerous checklists and guidelines available to help them do so (e.g. Cunningsworth 1984, Breen and Candlin 1987, Skierso 1991, McDonough and Shaw 1993). These instruments are generally organized in a manner that reflects the decision-making process which it is hypothesized teachers go through. Breen and Candlin (1987), for example, organize the questions in their checklist into two phases, the first of which enables teachers to address the overall 'usefulness' of the materials, while the second caters for 'a more searching analysis' based on the teacher's actual teaching situation. The idea behind these guides is to help teachers carry out a predictive evaluation systematically. Evaluating course materials retrospectively However, there are limits to how 'scientific' such an evaluation can be. As Sheldon (1988: 245) observes, 'it is clear that coursebook assessment is fundamentally a subjective, rule-of-thumb activity, and that no neat formula, grid or system will ever provide a definite yardstick'.

Retrospective Evaluation

            This being so, the need to evaluate materials retrospectively takes on special importance. Such an evaluation provides the teacher with information which can be used to determine whether it is worthwhile using the materials again, which activities 'work' and which do not, and how to modify the materials to make them more effective for future use. A retrospective evaluation also serves as a means of 'testing' the validity of a predictive evaluation, and may point to ways in which the predictive instruments can be improved for future use. Somewhat surprisingly, however, there are very few published accounts of retrospective evaluations of course materials, and very little information about how to conduct them. The bulk of the published literature on evaluation deals with programme or project evaluation (e.g. Alderson 1992, Weir and Roberts 1994, Lynch 1996). Such evaluations may incorporate materials evaluation but they are necessarily much broader in scope. Otherwise, the only other published work on the empirical evaluation of teaching materials is to be found in accounts of the trialing of new materials (e.g. Barnard and Randall 1995). The purpose of this article is to begin to address the question of how retrospective evaluations of materials can be carried out.

2.2.1       Evaluating Course Materials Retrospectively

Teachers can perform a retrospective evaluation impressionistically or they can attempt to collect information in a more systematic manner (i.e. conduct an empirical evaluation). It is probably true to say that most teachers do carry out impressionistic evaluations of their teaching materials. That is, during the course they assess whether particular activities 'work' (usually with reference to the enthusiasm and degree of involvement manifested by the students), while at the end of the course they make summative judgement of the materials. Empirical evaluations are perhaps less common, if only because they are time-consuming. However, teachers report using students' journals and end-of-course questionnaires to judge the effectiveness of their teaching, including the materials they used. One way in which an empirical evaluation can be made more manageable is through micro-evaluation. A macro-evaluation calls for an overall assessment of whether an entire set of materials has worked. To plan and collect the necessary information for such as empirical evaluation is a daunting prospect. In a micro-evaluation, however, the teacher selects one particular teaching task in which he or she has a special interest, and submits this to a detailed empirical evaluation. A series of micro-evaluations can provide the basis for a subsequent macro-evaluation. However, a micro-evaluation can also stand by itself and can serve as a practical and legitimate way of conducting an empirical evaluation of teaching materials. A micro-evaluation of teaching materials is perhaps best carried out in relation to 'task'. This term is now widely used in language teaching methodology (e.g. Prabhu 1987; Nunan 1989), often with very different meanings. Following Skehan (1996), a task is here viewed as 'an activity in which: meaning is primary; there is some sort of relationship to the real world; task completion has some priority; and the assessment of task performance is in terms of task outcome'. Thus, the information and opinion-gap activities common in communicative language teaching are 'tasks'.

Evaluating a Task Involves a Series of Steps:

References

Alderson, J. 1992. 'Guidelines for the evaluation of language education' in J. Alderson and A.

Beretta (eds.). Evaluating Second Language Education. Cambridge: Cambridge University Press.

Barnard, R. and M. Randall. 1995. 'Evaluating course materials: a contrastive study in textbook trialling'. System 23/3: 337-46.

Cunningsworth, A. 1984. Evaluating and Selecting ELT Materials. London: Heinemann.

Lynch, B. 1996. Language Program Evaluation: Theory and Practice. Cambridge: Cambridge University Press.

McDonough, J. and C. Shaw. 1993. Materials and Methods in ELT. Oxford: Blackwell.

Nunan, D. 1988. The Learner-centred Curriculum. Cambridge: Cambridge University Press.

Sheldon, L. 1988. 'Evaluating ELT textbooks and materials'. ELT Journal 42: 237^46.

Skierso, A. 1991. 'Textbook selection and evaluation' in M. Celce-Murcia (ed.). Teaching English as a Second or Foreign Language. Boston: Heinle and Heinle.

Weir, C. and J. Roberts. 1994. Evaluation in ELT. Oxford: Blackwell