May 2021 – Volume 25, Number 1
Beatriz López-Medina
Universidad Complutense de Madrid, Spain
<beatlo07ucm.es>
Abstract
Despite the prolific research on textbook evaluation in language teaching (Garinger, 2002; Miekley, 2005; Mukundan & Ahour, 2010; Nimehchisalem & Mukundan, 2015, among others), Content and Language Integrated Learning (CLIL) textbook evaluation is still a field to be developed. The lack of specific tools prevents CLIL teachers and other stakeholders from easily identifying strengths and weaknesses of the textbooks that contribute to conveying content through a foreign language (usually English). This study provides educators with a checklist to analyse and evaluate CLIL textbooks. It is the second part of a three-stage process which replicates in CLIL contexts the study by Mukundan, Hajimohammadi, & Nimehchisalem (2011), who created a valid and reliable checklist for that purpose in the English language teaching field. The checklist for CLIL textbooks is the final version based on a previous tentative checklist (López-Medina, 2016), which is the starting point for this part of the study. The final version of the tool was created incorporating the input provided by a focus group of CLIL teachers. Further research is in progress to provide each item of the list with specific weight through factor analysis, and to test the checklist for its validity and reliability.
Keywords: CLIL; CLIL materials: CLIL textbooks; CLIL textbook evaluation checklists; coursebook evaluation
Rationale
Finding the most suitable textbook for a specific educational context is an important decision since textbooks affect classroom dynamics and the teaching/learning experience, mainly when teachers rely significantly on textbook use. However, the variety of contexts, syllabi, teachers, learners, learners’ needs, and the roles that each stakeholder plays in the process make the choice of an appropriate textbook even more challenging, and textbook evaluation might provide the support needed in that respect (Montijano Cabrera, 2014). Textbook evaluation can help not only teachers, but also curriculum designers and material developers to identify and reflect upon the core features of textbooks, such as approach, content, activities and tasks, or supplementary materials, among others.
In the field of ESL (English as a second language)/EFL (English as a foreign language), and in order to facilitate the process of choosing and analysing textbooks, numerous researchers have turned textbook evaluation into a relevant area (Cortazzi & Jin, 1999; Miekley, 2005; Shatery & Azargoon, 2012). In this respect, in the last decades, different studies have been carried out to analyse and compare English language teaching (ELT) textbooks through a reliable instrument (see Cunningsworth, 1995; Hedge, 2000; Skierso, 1991, or Tucker, 1975, among others). In most cases, the checklist has been the most commonly used tool to identify the key features of the texts and their relevance, and, as Shatery and Azargoon (2012) observe, “there are a plethora of checklists in the literature in order to evaluate esl/efl textbooks” (p. 2). Some of these checklists are already well-established and are used as aids, either to select textbooks for specific contexts or to analyse to what extent a textbook has helped to achieve the learning objectives of a course. Extensive research on this topic has generated “theory-neutral, universal, and broad characteristics of EFL/ESL textbooks” (Ansary & Babaii, 2012, para.1).
CLIL has been defined by Marsh (2003) in the following manner:
Content and language integrated learning (CLIL) refers to any dual-focused educational context in which an additional language, thus not usually the first language of the learners involved, is used as a medium in the teaching and learning of non-language content. It is dual-focused because, whereas attention may be predominantly on either subject-specific content or language, both are always accommodated. (para 3)
Within the area of CLIL, however, textbook evaluation is an area still to be developed, with no trajectory within research or available well-established checklists. Nevertheless, numerous bilingual academic environments which follow the CLIL approach would benefit from certain guidelines, mainly considering that most content teachers face teaching through an additional language in the first years of their professional experience (usually inexperienced teachers rely more on the use of the textbook); in this respect, Williams (1983) observes: “In situations where there is a shortage of trained teachers, language teaching is very closely tied to the textbook” (p. 251); other teachers start teaching through a foreign language when their schools turn bilingual, after years of teaching through the first language (L1) shared by teachers and students alike. In this latter case, the lack of confidence in the use of a foreign language might make teachers more text-dependent.
Other circumstances (lack of training in CLIL, not enough collaboration between content and language teachers, etc.), unfortunately, common in bilingual educational contexts (Banegas, 2012; Pavón & Ellison, 2001; Şimşek, & Dündar, 2018), also determine the need of valuable tools which could help teachers and stakeholders by facilitating choice, adaptation, supplementation, and evaluation of the coursebooks they will be using in their lessons. In addition, a CLIL textbook, at the crossroads between content and language textbooks, encompasses unique features which are not reflected in checklists devised to analyse textbooks used within other approaches.
In this study, a focus group of experienced CLIL primary teachers provide their insights on the features that any CLIL textbook should present, based on their training on CLIL and on their teaching experience. Their suggestions and recommendations are used to refine an already made tentative CLIL textbook evaluation checklist (see the appendix of López-Medina, 2016) and to elaborate a new refined and updated checklist (see the Appendix). The updated checklist, made especially for CLIL contexts, can replace the use of evaluation tools designed for the area of ELT, which cannot cater to the CLIL stakeholders’ needs.
Review of Related Literature
Throughout decades and worldwide, the textbook has been a key component of language teaching, right in the centre of learning: “A textbook is the visible heart of any language teaching program” (Jebahi, 2009, p.75). Advocates of textbooks highlight their importance since they “play a very crucial role in the realm of language teaching and learning and are considered the next important factor (element) in the second/foreign language classroom after the teacher” (Riazi, 2003, p.52).” This vision has been taken even further, being moved to a position ahead of the teachers and considered “the primary agents of conveying knowledge to learners” (AbdelWahab 2013, p. 55). Although these views summarize extreme positions, a great deal of the existing literature on this topic emphasizes the positive aspects of the use of textbooks and their impact on language learning. The reasons in favour of using textbooks are varied, from attractive layout and design to other utilitarian attributes such as cost-effectiveness or reliability (Montijano Cabrera, 2014). However, it is other facts such as organization of content (Graves, 2000), the useful support offered to teachers and learners alike (Zohrabi, Sabouri, & Kheradmand, 2014), and the safety felt by the students when using them (Razmjoo, 2007) which are the most common when considering textbooks as a valuable learning tool.
However, there are also some views against the use of textbooks: Opponents to their use point out that the textbook can be a tyrant to the teacher who, when trying to cover the syllabus, “feels constrained to teach every item in exactly the same sequence and context that is presented by the textbook writer” (Williams, 1983, p. 251). This view, textbook as a tyrant and teacher as a slave, is one of the main criticisms from those contrary to its use in the classroom, together with the fact that a textbook cannot cater for all needs in any educational setting (Cortazzi, & Jin, 1999; Gholami, Noordin, & Rafik-Galea, 2017; Simsek, & Dündar, 2018). These considerations regarding textbooks are common to foreign languages and other areas of the curriculum alike. The two different approaches to their use (for and against) are so opposing that they do not share any common ground.
As for the use of textbooks in a foreign language (FL) class, the decision on using a coursebook or not is a common practice in any planning. In this respect, Tomlinson et al. (2001) point out the need to consider the roles that textbooks, teachers, and learners have in any program. For example, the context might require a teacher-centred approach, there might be a lack of updated materials in a specific setting, learners may be quite independent and therefore might not need the teachers’ guidance as much as other groups, experienced teachers might prefer the freedom of teaching without a textbook, etc. These features, among others, determine both the need and the potential use given to a textbook.
Despite the variety of educational contexts worldwide, “the reality of most ESL/EFL classroom settings provides clear evidence of a preference for teaching with textbooks” (Byrd & Schuemann, 2014, p. 380). It seems that even though active methodologies are gaining ground in the field of learning languages, reconsidering the role of all the stakeholders involved in teaching, the textbook still lies at the core of the teaching-learning process.
CLIL textbooks
With the advent of numerous plurilingual schools in Europe, the demand for content books written in additional languages has increased dramatically. Since the turn of the century, the gradual introduction of bilingual or plurilingual education in schools throughout Europe has led to plurilingual programs which keep increasing in number and specialization. Regarding materials, mainly textbooks, the initial demand was higher than the offer since it was difficult to provide textbooks to the different contexts which were adopting bilingual educational policies. This difficulty is still present nowadays: CLIL textbooks need to consider the community for which they are written, and in countries like Spain, where each autonomous region has different curricula, the cost to produce so many different textbooks increases. In addition, content teachers are used to having an extensive offer of L1 textbooks from different publishers, and therefore expect to have a similar range of content books to choose from written in the vehicular language (usually L2).
Initially (at the beginning of the “boom” of bilingual/plurilingual schools), some of the CLIL textbooks were translated from their L1 versions, which offered disadvantages for CLIL learners, mainly regarding linguistic difficulties or appropriate methodologies (Kelly, 2014). The option of using textbooks addressed to native speakers was also considered at that point but did not work either (Morton, 2013), since they offer a significantly harder linguistic demand and culture-bound information and also require a pedagogic adaptation which is time-consuming for teachers; as a consequence, these textbooks were not considered suitable for most CLIL contexts (Martín del Pozo & Rascón, 2013). Nowadays, the commercial offer has increased, though, in the current literature, there is still a common view on the lack of CLIL textbooks, mainly when compared with the number existing in their monolingual versions (Morton, 2013, Banegas, 2016).
Even though the availability of textbooks is one of the main concerns for CLIL teachers (Navés, 2009), the linguistic difficulty of the texts is also paramount (Kelly, 2014), since the foreign language, the main vehicular language in the class, must be integrated simultaneously with the content. In addition, the exposure to the additional language in CLIL contexts is much higher than in mainstream monolingual educational centres (Merikivi & Pietlä, 2014), as it is the linguistic demand of this approach. In this respect, content teachers are mainly concerned with the lexis of their fields, which is related to the specificity of the vocabulary of the content area. This lexis should also be contextualized, appropriate to the students’ L2 level, and sequenced in order to ensure the understanding of the content. The lexical load should also be achievable, and each entry supported by re-entries so as to recycle new terms and facilitate learning (scaffolding). Finally, the presentation and the treatment of vocabulary throughout all the modules of a content textbook through L2 is also important in order to facilitate understanding, learning and remembering of key terms (Ball, Kelly & Clegg, 2015).
Finally, another feature that should be noted in the area of textbooks is the use that teachers make of them, depending mainly on the teaching style and their experience (Cortazzi & Jin, 1999). As for CLIL teachers, according to Pavón and Ellison (2013), their profile has changed significantly in the last years, depending significantly on the contextual features of each country/region, which determine key factors such as L2 level required, specialization, etc. In this respect, with the turn of the century and in the European post-Common European Framework educational contexts, the increase of bilingual schools demanded a significant number of teachers who had to teach through L2 for the first time, or teachers with an L2 level high enough to teach through an additional language, but who were recent graduates with none or scarce teaching experience. In this context, CLIL textbooks helped novice teachers, often overwhelmed with all the new challenges they needed to face at the beginning of their professional experience. Nowadays, although the teachers’ linguistic competence in L2 has increased in most contexts, the textbook still plays the role of support in CLIL settings, facilitating the three dimensions of CLIL: the development of sequenced conceptual/declarative, linguistic, and procedural content (see Ball, Kelly & Clegg, 2015).
There might not be a single aspect of the CLIL approach that makes the use of textbooks essential, but the effects of their use in the students’ general learning both currently and in the past are undeniable. Regarding the CLIL approach, the aid provided to scaffold language and content, the support given to novice teachers, or the presentation of a guided curriculum can contribute to the growing reputation of textbooks. In this vein, Martín del Pozo and Rascón (2013) observe “in the particular context of CLIL the relevance of materials and textbooks has been repeatedly advocated in literature as a factor for successful CLIL implementation” (p. 127).
Textbook Evaluation
Textbook evaluation involves processes that can take place before, during, or after its use. The most common classification distinguishes predictive and retrospective evaluation, corresponding to the considerations taken before and after using the textbook (Ellis, 1997). Cunningsworth (1995) includes one more stage and divides textbook evaluation into pre-use, in-use, and post-use.
Pre-use involves selecting the most appropriate textbook for a specific group of learners, a complex process, since the textbook marketplace is very competitive (Cox, 2008) and makes efforts to offer a wider choice of books year after year, making selection harder. In most European educational contexts, this choice is not a centralized decision made by schools/heads, but by the team of teachers of the same area/subject: This fact is relevant since the choice partially determines the teaching, especially in those cases where the design of the syllabus is made around the book (Garinger, 2002). The second part of the process, in-use, involves identifying the strengths and weaknesses of the book so that the teacher can adapt certain sections and/or use different materials to make up for those weaknesses. Finally, in the post-use stage, teachers reflect upon the textbook’s suitability that has been used in a specific situation or for the study of a subject.
Textbook evaluation requires teachers to be aware of their class realities, but also of the educational system where they are involved, and of the previous and future stages of the students they are teaching (McGrath, 2002). Within bilingual educational contexts, it involves pedagogical knowledge on the teaching approach (mainly CLIL), but also considering other aspects, such as the linguistic policy of the community and of the specific school.
Since the textbook is considered one of the key elements in learners’ success, the choice and its analysis are crucial in order to avoid the use of inadequate materials. Therefore, teachers need to know the principles of text evaluation, an “essential practice in the field of teaching” as Ahmadi and Derakshan (2016, p.260) observe.
The use of checklists to evaluate textbooks
Current research advocates for a checklist as the most commonly used tool to evaluate textbooks, since it has been used for decades (Mukundan & Ahour, 2010). Checklists are easy to use and have been considered useful for stakeholders, teachers, and pre-service teachers who can get practical information of what a coursebook is like. Checklists complement what has been called the impressionistic method, consisting of evaluating a textbook from a general impression (McGrath, 2002, AbdelWahab, 2013). Even though they do not provide a detailed description of the textbook features, they have also been considered useful to discard unsuitable textbooks (Skierso, 1991).
A review of the checklists used to evaluate ELT textbooks shows differences in categories and criteria. This happens because checklists are not meant to be exhaustive, they are always in process, and items can be eliminated or added depending on each context. In addition, “what one teacher considers an advantage in a textbook, another teacher may consider a disadvantage” (Graves, 2000, p. 175); therefore, creating a checklist that fitted all contexts would be impossible. As AbdelWahab summarizes, referring to ELT textbooks: “neither could [exist] a universally appropriate English language evaluative checklist” (2013 p. 59).
In the field of ELT, the closest one to CLIL contexts, the textbook evaluation checklists available present differences in the number of key features and in the grouping of them. However, a set of criteria with certain similar features can often be found. For example, checklists usually include physical and utilitarian attributes such as layout, design, attractiveness, or visuals (Sheldon, 1988, Cunningsworth, 1995); items referring to learner factors, such as age range and learners’ needs, and to teacher factors, such as language competence and experience of teaching, among others (McGrath, 2002). Finally, the availability of supplementary materials and the treatment of language skills are also frequently included (Garinger, 2002).
As for the number of criteria, checklists, though comprehensive, need to be made up of a number of items that are not too extensive for the users to complete (Swales, 1980). For instance, Cunningsworth’s checklist (1995), one of the best-established, is made up of 45 criteria grouped in 8 categories: “aims and approaches, design /organization, language content, study skills, topic, methodology, teacher’s book, and practical considerations” (in AbdelWahab, 2013, p.57).
Regarding the grouping of criteria, this study follows the classification established by Mukundan, Hajimohammadi, and Nimehchsilasen (2011) in their extensive review of the textbook evaluation checklist within four decades (1970-2008). Their study takes into consideration aspects such as the number of items, categories, and length of statements, among others. They observed that most researchers prefer to devise qualitative than quantitative checklists but point out that quantitative checklists “are more reliable instruments and more convenient to work with, especially when team evaluations are involved” (p. 21). Since this study replicates theirs, the checklist devised and provided in this paper is also quantitative.
Finally, within the current literature on ELT checklists, the checklist elaborated by Alemi and Mesbah (2013) has an organization and structure very close to the one presented in the Appendix. Their checklist was created to analyse language textbooks adhering to the American Council on the Teaching of Foreign Languages (ACTFL)’s standards. These standards contain five main components: communication, culture, connection, comparison, and community, which can be compared to the four Cs in CLIL (content, communication, cognition, and culture). The teachers taking part in their study observed the merits and the shortcomings of each “C”. In a similar vein, one category per each “CLIL C” is provided in the Appendix in order to be individually analysed.
The 4 Cs in CLIL, considered as the key components/principles of the approach, are all interconnected, and their integration involves successful learning outcomes (Coyle, Hood, & Marsh, 2010). As these authors point out, these components/principles indicate that even though the content is key in CLIL subjects, it must be supported by communication within a culture/community (learning as a social process) and by the appropriate cognitive demands in the different stages of the learning (reflection, analysis, evaluation, etc.).
Method
The present study is the second part of a three-stage project; the main objective of this phase is to refine the tentative checklist created in the previous part of the study (López-Medina, 2016). In the first part of the study, a thorough revision of the existing instruments to evaluate ELT textbooks was carried out, and a tentative checklist was drawn up, considering two features:
- checklists previously created to evaluate ELT textbooks (CLIL materials use principles/techniques of foreign language teaching, Steirt & Masser, 2011)
- criteria for producing quality CLIL learning materials drawn up by Mehisto (2012)
The tentative checklist included a list of items considering general criteria (organization, layout, price, available resource pack, etc.), linguistic criteria (sequencing of vocabulary, aids for pronunciation, etc.), the 4 Cs of CLIL (content, communication, cognition, and culture), and a final criterion related to the consistency of the textbook to the principles of CLIL. The final list comprised 54 criteria. In that stage, we considered that marking the presence or absence of each criterion would be enough to have an approximate impression of the main features of the book. The list was conceived to cover the essential elements of a CLIL textbook trying to avoid ELT terminology, even though ELT checklists had been considered as referents.
In this part of the study, and following Mukundan, Nimehchsilasen, and Hajimohammadi (2011), a focus group (n=5) was given the tentative checklist from the previous stage of the study in order to discuss the relevance of the inclusion of each item in the list and in the corresponding categories and, on the other hand, to consider the potential addition of items not included in the first version of the list. As a result of this process, a new list was created based on the contributions of the group. Further research is in process in order to add the weight of each item from the list through factor analysis (third stage of the process). As the referent authors suggest, in this stage, it is necessary to increase validity by checking that the criteria are representative enough for each section of the checklist. It is also necessary to improve reliability by simplifying statements for those users with less expertise. Finally, regarding practicality, the final number of criteria should be manageable. In this respect, the number of words of the new checklist (n=384) can be considered moderate since, in the analysis of textbook evaluation checklists by Mukundan and Ahour (2010), the running words ranged between 113 (Tucker, 1975) and 4553 (Skierso, 1991).
Participants
The focus group included five CLIL teachers of primary education (4 female and 1 male), aged 25-30, with previous experience teaching in bilingual programs (3-7 years) in different schools in Spain. They all had experience in selecting CLIL textbooks and EFL textbooks for their students, but had never used checklists in order to select or evaluate their coursebooks. All the participants had had the same training (Degree in Teaching Primary Education and a recently finished master’s degree in bilingual education, where they had taken two compulsory subjects on CLIL). These two features (working as CLIL teachers and same postgraduate specific training) were essential in the process of recruiting participants in order to make up a group with similar characteristics. The participation in the focus group was proposed to the former students of the master’s degree above-mentioned, and it was voluntary. Informed consent was obtained from all participants.
The one hour and forty minutes discussion of the focus group took place online via a free web-based group videoconference. The administrator took notes on the suggestions and recommendations provided by the group, and the videoconference was also recorded. Following Mukundan, Nimehchsilasen, and Hajimohammadi (2011), the discussion was divided into two sections: unstructured interview and structured interview. In the unstructured interview, participants brainstormed their ideas on the general features they considered important in an effective CLIL textbook. In the structured interview, they were provided with the tentative checklist created for the first stage of the study (see López-Medina, 2016) and commented on the clarity of the items, the relevance of the criteria, and the categories the criteria had been included in. The focus group could use CLIL textbooks throughout the whole discussion and could also incorporate examples of certain items from the checklist when necessary. Addition, deletion, or modification of items of the tentative list was carried out with the contributions from the group as each item was presented and discussed.
Unstructured Interview Results
This section includes the items suggested by the focus group which had not been included in the tentative checklist and which came up in the first part of the interview. The new items are listed in this section, discussed individually, and included in the new checklist. The information on the location in the new checklist is provided in brackets.
(1) It provides support for the parents helping with homework (added as item 9, part IB)
(2) It offers online activities to promote autonomous learning (added as item 10, part IB)
(3) The teacher guide provides extra activities for extension (added as item 26, part II)
(4) The teacher guide provides extra activities for reinforcement (added as item 27, part II)
Regarding the first two items, and in contrast to monolingual academic contexts, the focus group indicated that support for the students’ families is essential in CLIL contexts: All participants observed that parents at their schools found difficulties when trying to help their children with homework, since nowadays, in Spain, the level of FL of their students’ parents is not always high enough to understand instructions of exercises provided in the students’ assignments. Certain cues or sporadic translations provided in workbooks might come in useful in order to help with exercises done at home. In this vein, four teachers mentioned the importance of online activities that most textbooks provide nowadays, which help students reinforce the contents provided in class.
The third and fourth items refer to the teacher guide: all the teachers pointed out the importance of a resource pack or a section included in the teacher guide where the teachers could have easy access to activities to be used for extension and reinforcement. The difficulty of attending classes through an additional language requires the reintroduction of key concepts more often than in monolingual contexts, and online activities offer more exposure to new terms.
Structured Interview Results
In the second part of the discussion, the focus group was provided with the above-mentioned tentative checklist for CLIL textbooks evaluation. The purpose of this phase was to adjust and modify the list according to the teachers’ needs. The categories of the checklist and all the items were discussed in order to analyse if they needed rewriting, merging, or elimination. At the end of the discussion, participants were invited to suggest new items/criteria to be added. As a result of this phase, 6 items from the previous list (1, 3, 4, 6, 7, 9, and 10) were revised (mainly reworded), 2 were deleted (items 2 and 11) and 3 were added (5, 8, and 12). In the list comprising these items, the inserted words are underlined, and the deleted words have been stricken through.
(1) It provides guidance for non-native content teachers teaching through L2 (item 8, part I.B)
(2) It provides support for language assistants (deleted)
(3) It contains appropriate pictures, diagrams, tables sufficient visual content (pictures, diagrams, tables …) (item 13, part IC)
(4) Learning outcomes for learning are specified (item 18, part II)
(5) HOTS and LOTS are present through the unit/module (added as item 37, part III)
(6) It stresses communicative competence in activities (interaction) Activities promote interaction (item 39, part IV)
(7) Activities are developed to encourage teacher-student and student-student communication (item 41, part IV)
(8) Instructions are straightforward (added as item 42, part IV)
(9) There is appropriate sequencing of vocabulary (load and re-entry) (item 55, part VI)
(10) There is appropriate sequencing of vocabulary (load and re-entry) (item 56, part VI)
(11) The number of words in each module is appropriate to the students’ level of L2 (deleted)
(12) The length of texts is appropriate to the students’ L2 level (added as item 57, part VI)
The first two items are related to the main teachers and their language assistants respectively. The first item was rephrased following the suggestion provided by the whole group: To change the concept non-native content teachers teaching through L2 to content teachers teaching through L2, since this latter term focuses on their identity as content teachers and not on a feature related to them as speakers of L2. The second item, regarding language assistants, was deleted since the presence of language assistants in the class is scarce at times. Therefore, their tasks are limited and guided by the headteacher depending on the group needs. Finally, a specific section in a coursebook suggesting activities/tasks where language assistants could help was not considered useful, and the item included in the list was dropped out.
The third item of the list was reworded by grouping the terms pictures, diagrams, and tables into the generic term visual aids, including specific examples (graphic organizers or mindmaps, among others). In CLIL contexts, the variety of teaching methods and multimodality is emphasized, and the use of visual aids is key to help understand content when the language proficiency is limited; therefore, the presence of visual aids is more frequent in CLIL contexts than in language classes (Çekrezi, 2011, Mehisto, 2012).
Participants observed the need to introduce high and low thinking skills in the list. According to them, an item compiling the presence of HOTS (high order thinking skills) and LOTS (low order thinking skills) was needed. HOTS and LOTS are part of the theoretical bases of CLIL and constitute one of the 4Cs –cognition-, upon which the approach is based (the other three –communication, culture, and content- are also categories present in the checklist).
The group observed differences in the syntactic patterns of the items. In order to simplify sentences and use parallel structures when possible, items 4, 6, and 7 were syntactically simplified by reducing wording in the statements.
Two participants pointed out the importance of clear instructions in textbooks. The item instructions are straightforward was introduced in the category of communication.
Regarding items 9 and 10, related to vocabulary, in the initial checklist, they were merged: there is appropriate sequencing of vocabulary (load and re-entry). However, since items must not have coordinate sections, this item was split into two. Item 11 on the layout of the book was considered redundant and, therefore, deleted. Finally, a new item regarding the length of texts was introduced: All the teachers pointed out the need to include texts in the FL coursebooks shorter than those included in the textbooks written in the L1; the reading speed, the fluency, and also the understanding of texts differs in the mother tongue and in the additional language(s).
Results and Discussion
As a result of the suggestions provided by the focus group, and after analysing their recommendations, the new checklist was created. It is made up of 60 items grouped under 6 headings, increasing the number of items from the tentative checklist (54), and reducing the categories (7). The number of items is within the range of the checklists from the field: Farr and Tulley (cited in Chambliss, 1994) observed that in the 70 checklists of English Language Textbooks they had reviewed, the number of criteria ranged from 42 to 180. The users’ rating can be indicated on the right column, on a Likert scale from 0 to 4 to indicate satisfaction (4=Excellent, 3=Good, 2=Adequate, 1=Weak, and 0=Totally lacking).
After the discussion of the focus group, the most significant changes to the list (new items added) took place in the sections of supplementary materials, content, and language. This might be due to the lack of materials in CLIL (section supplementary materials) and the dual nature of the approach (content and language).
As for the number of words, the new list has 502 words instead of 480, despite the six new items added to the list; the increase of only 22 words shows an effort to simplify syntax and reword items. Finally, the focus group did not find any category which could be inapplicable to CLIL contexts; consequently, none was dropped out.
Conclusion
The objective of this phase of the study was analysing a tentative evaluation checklist for CLIL textbooks in order to evaluate the clarity of items and to discuss their inclusiveness (to what extent they still pertain and if new items were needed). Conclusions obtained after the focus group’s analysis revealed the need for a new checklist that fitted the CLIL teachers’ needs better, but without showing significant differences from the one proposed (tentative checklist). The version can be considered as final since the future third phase of the study will not alter the number of current items or modify their content. Instead, it will be focused on the areas of validity and reliability in relation to each individual criterion from the checklist.
Among the limitations of the study, we can point out the composition of the focus group, made up completely of primary teachers. We believe that even if some criteria might change depending on the specific contexts, CLIL contexts in primary and secondary education share a significant number of factors, and most of the evaluative criteria included in the checklist are valid for most CLIL contexts. Therefore, in the next phase of the study both educational levels will be considered. As for the suggestions provided to modify the tentative list, we observe a significant emphasis on the technical criteria than on the rest, with frequent references from the focus group to the choice of typeface or too colourful illustrations; in this respect, since the use of checklists allows teachers to add extra criteria to the list in order to tailor them to their needs, the physical layout might have to be considered as a relevant feature for CLIL teachers.
The updated version provided in the Appendix can be useful for those teachers and materials developers working with CLIL textbooks. Even though the full testing for reliability and validity has not been carried out at this stage, the checklist can be used as a reflection tool. In this respect, it can be of help to novice teachers in the area or to those experienced teachers whose previous experience with textbooks has not been successful.
Its main significance is covering the gap in textbook evaluation, mainly in bilingual contexts where the CLIL approach is used.
About the Author
Beatriz López-Medina has extensive teaching experience in the area of Applied Linguistics both at undergraduate and postgraduate levels. Her research interests include bilingual education and the CLIL approach. She received the Second National Award on Educational Research (Spain). She also works in the area of Quality Assurance in Higher Education.
To cite this article:
López-Medina, B. (2021). On the Development of a CLIL Textbook Evaluation Checklist: A Focus Group Study. Teaching English as a Second Language Electronic Journal (TESL-EJ), 25(1). https://tesl-ej.org/pdf/ej97/a17.pdf
References
AbdelWahab, M. M. (2013). Developing an English language textbook evaluative checklist. IOSR Journal of Research & Method in Education, 1, 55–70. https://doi.org/10.19030/cier.v4i6.4383
Ahmadi, A., & Derakhshan, A. (2016). EFL teachers’ perceptions towards textbook evaluation. Theory and Practice in Language Studies, 6(2), 260-267. http://dx.doi.org/10.17507/tpls.0602.06
Alemi, M., & Mesbah, Z. (2013). Textbook evaluation based on the ACTFL standards: The case of top notch series. The Iranian EFL journal, 9(1), 162-171. https://www.mjselt.com/iefl-journals/iefl-2013/textbook-evaluation-based-on-the-actfl-standards-the-case-of-top-notch-series/
Ansary, H., & Babaii, E. (2002). “Universal characteristics of EFL/ESL coursebook: A step towards systematic coursebook evaluation. The Internet TESL Journal, 2, 1-8. http://iteslj.org/Articles/Ansary-Textbooks/
Banegas, D. L. (2012). CLIL teacher development: Challenges and experiences. LACLIL. Latin American Journal of Content & Language Integrated Learning, 5(1), 46–56. https://doi.org/10.5294/laclil.2019.12.2.3
_____ (2016). Teachers develop CLIL materials in Argentina: A workshop experience. LACLIL. Latin American Journal of Content & Language Integrated Learning, 9(1), 17-36. 10.5294/laclil.2016.9.1.2
Ball, P., Kelly, K., & Clegg, P. (2015). Putting CLIL into practice. Oxford University Press.
Byrd, P., & Schuemann, C. (2014). English as a second/foreign language textbooks: How to choose them —how to use them. Celce-Murcia, M., Brinton, D., & Snow, M.A. (Eds.), Teaching English as a second or foreign language (4th ed.), (pp. 380-393). Heinle.
Chambliss, M. J. (1994). Evaluating the quality of textbooks for diverse learners. Remedial & Special Education, 15(6), 348-362. https://doi.org/10.1177/074193259401500604
Cox, J. (2008). As I see it! – Emerging trends in textbooks – Libraries will feel the effect. Against the Grain, 20(2), 74-75. https://doi.org/10.7771/2380-176X.2761
Cunningsworth, A. (1995). Choosing your coursebook. Macmillan: Heinemann.
Çekrezi, R. (2011). CLIL and teacher training. Procedia Social and Behavioral Sciences, 15; 3821–3825 https://doi.org/10.1016/j.sbspro.2011.04.379
Cortazzi, M., & Jin, L. (1999). Cultural mirrors: Materials and methods in the EFL classroom. In Hinkel, E. (Ed.) Culture in Second Language Teaching and Learning. Cambridge University Press, 196-219.
Coyle, D., Hood, P., & Marsh, D. (2010). CLIL. Content and language integrated learning. Cambridge University Press.
Ellis, R. (1997). SLA research and language teaching. Oxford University Press.
Garinger, D. (2002). Textbook selection for the ESL classroom. ERIC Digest.
Gholami, R., Noordin, N., & Rafik-Galea, S. (2017). A thorough scrutiny of ELT textbook evaluations: A review inquiry. International Journal of Education and Literacy Studies, 5(3), 82–91.
Graves, K. (2000). Designing language courses, a guide for teachers. Heinle.
Hedge, T. (2000). Teaching and learning in the language classroom. Oxford University Press.
Jebahi, K. (2009). Using a commercially developed ESP textbook: A classroom dilemma, The Asian ESP Journal, 5(2), 75-92. https://www.asian-esp-journal.com/
Kelly, K. (2014). Ingredients for successful CLIL. British Council, Teaching English. http://www.teachingenglish.org.uk/article/keith-kelly-ingredients-successful-clil-0
López-Medina, B. (2016). Developing a CLIL textbook evaluation checklist. LACLIL. Latin American Journal of Content & Language Integrated Learning, 9(1), 159-173. https://doi.org/10.5294/laclil.2016.9.1.7
Marsh, D. (2003). The relevance and potential of content and language integrated learning (CLIL) for achieving MT+2 in Europe. ELC Information Bulletin, 9 – http://www.celelc.org/archive/Information_Bulletins/2003_marsh/index.html
Martín del Pozo, M. A., & Rascón Estébanez, D. (2015). Textbooks for content and language integrated learning: Policy, market and appropriate didactics? Foro de Educación, 13(18), 123-141. http://dx.doi.org/10.14516/fde.2015.013.018.007
McGrath, I. (2002). Materials evaluation and design for language teaching. Edinburgh University Press.
Mehisto, P. (2012). Criteria for producing CLIL learning material. Encuentro, 21, 15-33. http://hdl.handle.net/10017/14581
Merikivi, R., & Pietlä, P. (2014). Vocabulary in CLIL and in mainstream education. Journal of Language Teaching and Research, 5(3), 487-497. doi:10.4304/jltr.5.3.487-497
Miekley, J. (2005). ESL textbook evaluation checklist. The Reading Matrix 5(2), 9-17.
Montijano Cabrera, M. P. (2014). Textbook use training in EFL teacher education. Utrecht Studies in Language & Communication, 27, 267-286. https://doi.org/10.1163/9789401210485_016
Morton, T. (2013). Critically evaluating materials for CLIL: Practitioners’ practices and perspectives, In Gray, J. (Ed.) Critical Perspectives on Language Teaching Materials, 111–136. Palgrave Macmillan.
Mukundan, J., & Ahour, T. (2010). A review of textbook evaluation checklists across four decades (1970-2008). In B. Tomlinson, & H. Masuhara. (Eds.), Research for materials development in language learning: Evidence for best practice. Continuum.
Mukundan, J., Hajimohammadi, R., & Nimehchisalem, V. (2011). Developing an English language textbook evaluation checklist. Contemporary Issues in Education Research. 4(6), 21-27. https://doi.org/10.19030/cier.v4i6.4383
Mukundan, J. H., Nimehchisalem, V., & Hajimohammadi, R. (2011). Developing an English language textbook evaluation checklist: A focus group study. International Journal of Humanities and Social Science, 1(12), 100-105.
Navés, T. (2009). Effective content and language integrated learning (CLIL) programmes. In Y. Ruiz de Zarobe, & R. Jimenez Catalan (Eds.), Content and language integrated learning: Evidence from research in Europe, (pp. 22-40). Multilingual Matters.
Nimehchisalem, V., & Mukundan, J. (2015). Refinement of the english language teaching textbook evaluation checklist. Pertanika Journal of Social Sciences & Humanities (JSSH), 23(4), 761–780.
Pavón Vázquez, V. (2014). Enhancing the quality of CLIL: Making the best of the collaboration between language teachers and content teachers. Encuentro 23, 115-127. http://hdl.handle.net/10017/21615
Pavón Vázquez, V., & Ellison, M. (2013). Examining teacher roles and competences in content and language integrated learning (CLIL). Lingvarvm arena 4: 65-78.
Razmjoo, S.A. (2007). High schools or private institutes textbooks? Which fulfill communicative language teaching principles in the Iranian context?. The Asian EFL Journal Quarterly, 9(4), 125-139. https://asian-efl-journal.com/December_2007_EBook.pdf
Riazi, A. M. (2003). What textbook evaluation schemes tell us? A study of the textbook evaluation schemes of three decades. In W. A. Renandya (Ed.), Methodology and Materials Design in Language Teaching, (pp.52-68). SEAMEO Regional Center.
Shatery, H., & Azargoon, M. (2012). Investigating the degree of correspondence between topics of reading passages in general English course books and students’ interest. The Iranian EFL Journal, 8(4), 261–272. https://www.mjselt.com/wp-content/uploads/2014/01/Volume-8-Issue-6.pdf
Sheldon, L.E. (1988). Evaluating ELT textbooks and materials. ELT Journal 42 (4), 237-246. https://doi.org/10.1093/elt/42.4.237
Simsek, M. R., & Dündar, E. (2018). Particularised checklists in materials evaluation: developing contextually relevant criteria for Turkish EFL classes. Journal of Language and Linguistic Studies, 14(3), 154–189.
Skierso, A. (1991). Textbook selection and evaluation. In M. Celce-Murcia (Ed.), Teaching English as a second or foreign language, (pp. 432-453). Heinle.
Steirt, C. & Masser, U. (2011). Guidelines for evaluating and developing CLIL materials. In S. Ioannou-Georiou & P. Pavlou, Guidelines for CLIL Implementation in Primary and Pre-primary Education, (pp. 98-113). PROCLIL. Online publication. https://tinyurl.com/ybowq3o8
Tomlinson, B., Dat, B., Masuhara, H., & Rubdy, R. (2001). ELT courses for adults. ELT Journal, 55(1), 80-101. https://doi.org/10.1093/elt/55.1.80
Tucker, C. A. (1975). Evaluating beginning textbooks. English Teaching Forum 13, 355-361.
Williams, D. (1983). Developing criteria for textbooks evaluation. ELT Journal 37(3), 251–255. https://tinyurl.com/y2dp5mw5
Zohrabi, M., Sabouri, H., & Kheradmand, M. (2014). Comparative study of Interchange 1 and English book 1 of Iranian high schools. Education International Journal of English, 3(2), 95-104. 10.7575/aiac.ijalel.v.3n.3p.58
APPENDIX
Tentative checklist for CLIL textbooks
Criteria | |
I. GENERAL | |
A. Structure | |
1. The organization is consistent with the curriculum | ⓪ ① ② ③ ④ |
2. The objectives are specified explicitly | ⓪ ① ② ③ ④ |
3. It provides summarizing items at the end of the modules | ⓪ ① ② ③ ④ |
4. It includes self-evaluation at the end of the modules | ⓪ ① ② ③ ④ |
B. Supplementary materials | |
5. The resource pack is complete | ⓪ ① ② ③ ④ |
6. The resource pack includes varied ICT resources | ⓪ ① ② ③ ④ |
7. It offers supplementary materials for under- /over- achievers | ⓪ ① ② ③ ④ |
8. It provides guidance for content teachers teaching through L2 | ⓪ ① ② ③ ④ |
9. It provides support for the parents helping with homework | ⓪ ① ② ③ ④ |
10. It offers online activities to promote autonomous learning | ⓪ ① ② ③ ④ |
C. Physical and utilitarian features | |
11. It shows quality in editing and publishing | ⓪ ① ② ③ ④ |
12. Its layout is attractive | ⓪ ① ② ③ ④ |
13. It contains enough visual aids (pictures, diagrams, tables …) | ⓪ ① ② ③ ④ |
14. It is easily available | ⓪ ① ② ③ ④ |
15. It is durable | ⓪ ① ② ③ ④ |
16. It is cost-effective | ⓪ ① ② ③ ④ |
II. CONTENT | |
17. It covers the contents of the curriculum | ⓪ ① ② ③ ④ |
18. Learning outcomes are specified | ⓪ ① ② ③ ④ |
19. The content is appropriate for the students’ age | ⓪ ① ② ③ ④ |
20. The content is relevant to students’ experiences | ⓪ ① ② ③ ④ |
21. The order of sequences is flexible | ⓪ ① ② ③ ④ |
22. It provides support to simplify content (scaffolding) | ⓪ ① ② ③ ④ |
23. The visual content is functional | ⓪ ① ② ③ ④ |
24. The activities suggested for practicing the content are varied | ⓪ ① ② ③ ④ |
25. The activities suggested for practicing the content are enough | ⓪ ① ② ③ ④ |
26. The teacher guide provides extra activities for extension | ⓪ ① ② ③ ④ |
27. The teacher guide provides extra activities for reinforcement | ⓪ ① ② ③ ④ |
28. There is authentic material at an appropriate level | ⓪ ① ② ③ ④ |
III. COGNITION | |
29. It allows breaking down tasks / activities to make them more manageable (scaffolding) | ⓪ ① ② ③ ④ |
30. It relates written work to structures and vocabulary practiced orally | ⓪ ① ② ③ ④ |
31. The activities are cognitively appropriate for the content | ⓪ ① ② ③ ④ |
32. It caters the needs of different learning styles | ⓪ ① ② ③ ④ |
33. Activities activate previous knowledge | ⓪ ① ② ③ ④ |
34. Activities are challenging | ⓪ ① ② ③ ④ |
35. Activities are motivating | ⓪ ① ② ③ ④ |
36. Activities include projects | ⓪ ① ② ③ ④ |
37. HOTS and LOTS are present through the unit/module | ⓪ ① ② ③ ④ |
IV. COMMUNICATION | |
38. It provides support to simplify language (scaffolding) | ⓪ ① ② ③ ④ |
39. Activities promote interaction | ⓪ ① ② ③ ④ |
40. Activities enable students to use the L2 outside the classroom situations | ⓪ ① ② ③ ④ |
41. Activities encourage teacher-student and student-student communication | ⓪ ① ② ③ ④ |
42.- Instructions are straightforward | ⓪ ① ② ③ ④ |
43. Activities are balanced between individual response, pair work and group work | ⓪ ① ② ③ ④ |
V. CULTURE | |
44. It relates content to the learners’ culture and environment | ⓪ ① ② ③ ④ |
45. It guides students in developing cultural awareness | ⓪ ① ② ③ ④ |
46. The content is relevant to the socio cultural environment | ⓪ ① ② ③ ④ |
47. The content involves culture-specific items | ⓪ ① ② ③ ④ |
48. The content is free from stereotypical images | ⓪ ① ② ③ ④ |
49. The visuals relate to the students own culture | ⓪ ① ② ③ ④ |
50. Cultural sensitivities have been considered | ⓪ ① ② ③ ④ |
VI. LANGUAGE | |
51. It considers proficiency level of L2 | ⓪ ① ② ③ ④ |
52. The language is authentic | ⓪ ① ② ③ ④ |
53. It presents vocabulary in appropriate contexts and situations | ⓪ ① ② ③ ④ |
54. It gives guidance in the presentation of vocabulary | ⓪ ① ② ③ ④ |
55. There is appropriate sequencing of vocabulary (load) | ⓪ ① ② ③ ④ |
56. There is appropriate sequencing of vocabulary (re-entry) | ⓪ ① ② ③ ④ |
57. The length of texts is appropriate to the students’ L2 level | ⓪ ① ② ③ ④ |
58. It gives practice in guided composition in early stages | ⓪ ① ② ③ ④ |
59. It suggests aids for pronunciation | ⓪ ① ② ③ ④ |
VII. INTEGRATION | |
60. It is consistent with the principles of CLIL | ⓪ ① ② ③ ④ |
[back]
Copyright rests with authors. Please cite TESL-EJ appropriately. Editor’s Note: The HTML version contains no page numbers. Please use the PDF version of this article for citations. |