{"id":16183,"date":"2021-01-16T21:15:10","date_gmt":"2021-01-16T12:15:10","guid":{"rendered":"http:\/\/www.tesl-ej.org\/wordpress\/?page_id=16183"},"modified":"2021-02-01T21:03:00","modified_gmt":"2021-02-01T12:03:00","slug":"ej96a2","status":"publish","type":"page","link":"https:\/\/tesl-ej.org\/wordpress\/issues\/volume24\/ej96\/ej96a2\/","title":{"rendered":"Constructing Analytic Rubrics for Assessing Open-Ended Tasks in the Language Classroom"},"content":{"rendered":"

February 2021 – Volume 24, Number 4<\/h2>\n

Mary Lou Vercellotti<\/strong>
\nBall State University
\n<mlvercellott\"atmark\"bsu.edu><\/p>\n

Dawn E. McCormick<\/strong>
\nUniversity of Pittsburgh
\n<mccormic\"atmark\"pitt.edu><\/p>\n

\n

Abstract<\/h3>\n

With the increased emphasis in communicative language learning and task-based language teaching, classroom assessment increasingly includes performance-based assessments (e.g., essays, speeches, projects, presentations), which require careful planning and can be challenging to implement and assess. The ability to design and use well-designed assessments for language performance tasks is a critical skill for classroom instructors. A rubric is a tool to more objectively measure the quality of the language performance when assessing language use in open-ended tasks. Analytic rubrics, with multiple categories and descriptions reflecting various levels of performance, help the instructor evaluate the effectiveness of instruction, document evidence of learner progress, and give feedback to learners. This article synthesizes theoretical and empirical scholarship in order to describe how to construct well-designed analytic rubrics for classroom language assessment. Four main steps are described: 1) establishing categories, 2) describing levels of performance, 3) reviewing the components of the rubric before implementation, and 4) evaluating the effectiveness of the rubric after implementation.<\/p>\n

Keywords<\/em>: rubrics, scoring scales, assessment, grading, performance-based, task-based<\/p><\/blockquote>\n

Classroom Assessment<\/h3>\n

Classroom assessment documents evidence of learner progress, gives feedback to learners, and allows teachers to evaluate the effectiveness of instruction. Instructors should be able to design and implement \u201cvaried and valid assessments…to support student learning\u201d (TESOL International Association, 2018, p. 60). Classroom instructors may choose a variety of assessments, from objective tests, such as form-focused multiple choice tests, to open-ended performance-based tasks, with many tasks in between these two sides of the assessment continuum. Assessing what a learner knows about<\/em> a language can be relatively straightforward. For instance, the learner supplies answers on an objective test (e.g., vocabulary matching quiz) whose questions have expected answers, and the instructor can score the learners\u2019 work reliably with an answer key. For classroom assessment with specific form-focused objectives, other assessment tools, such as detailed grading criteria, checklists, and tallies may be useful. Learner knowledge about the language, though, does not reflect what the learner can do<\/em> with that knowledge (Van Gorp & Deygers, 2014), and the results of objective tests do not reflect the learner\u2019s language skills, such as the ability to speak or write the language. With increased emphasis on communicative competence and task-based language teaching, language instructors have begun to assign more open-ended language tasks as pedagogical activities and as assessments (Purpura, 2016; Van Gorp & Deygers, 2014). Given the variety of assessment types and assessment tools, a crucial issue in assessment is the alignment between the content to be assessed and the assessment procedure. In a performance-based (or task-based) approach to language assessment, the learner\u2019s performance in open-ended tasks is seen as evidence of the learner\u2019s knowledge, skills, and ability (Purpura, 2016). With an open-ended language production task (e.g., essays, speeches, projects, presentations), the learner has the opportunity to choose the vocabulary and grammatical structures in response to the prompt or assignment. For instance, one learner may choose to use more complex grammar to express an idea while another may choose to create simpler but accurate structures. These open-ended performance-based assessments are vital in classroom language assessment because they more directly measure productive skills (e.g., speaking, writing). Performance-based assessments, however, require careful planning and can be challenging to implement. Instructors may not yet have the requisite knowledge to assess these language performances. Because of these challenges and others, instructors may borrow an existing assessment from another context (e.g., another course, another proficiency level), but using a rubric designed for another context is a potential misuse of the assessment tool (Purpura, 2016). Accordingly, it is important for instructors to create well-designed assessments for their classroom. In fact, designing valid assessments is a key teaching principle (TESOL International Association, 2018).<\/p>\n

This article describes how to construct an analytic rubric for classroom language assessment, informed by theoretical and empirical scholarship. After introducing rubrics as a tool for assessing student performance of open-ended tasks, we then describe how to construct an analytic rubric by identifying categories and describing levels of performance for each category. The third and fourth sections provide recommendations for pre-use and post-use reviews, critical steps in the assessment process. Last, we provide a checklist that summarizes steps in constructing analytic rubrics for open-ended tasks and provides practical suggestions.<\/p>\n

Performance-based Assessments and Rubrics<\/h3>\n

Classroom instructors choose from a range of assessments, with consideration of the link between course objectives and the assessments themselves. Open-ended tasks focus on course-level communication objectives, rather than on specific grammar or vocabulary of any particular lesson, thus requiring an assessment tool that measures the quality of the language produced during these tasks. Analytic rubrics are one tool for open-ended tasks in an instructor’s assessment tool kit, thus instructors must determine whether a rubric is the right tool for the assessment. The concepts discussed below provide guidance to determine if a rubric is an appropriate assessment tool.<\/p>\n

Open-ended tasks often are evaluated through the rater\u2019s impressions or judgements (Green & Hawkey, 2012), which are inherently subjective, even when learners are to be evaluated by how well they meet stated assessment objectives. The ratings could differ because different raters can have different impressions of the performance (Green & Hawkey, 2012), but even a single rater may have difficulty scoring consistently. For instance, scoring may become more conservative (or more generous) as fatigue increases (Van Moere, 2014), or a rater might start assessing quite strictly and lower expectations after viewing multiple performances. Raters can also be influenced by the previous language performance during the grading process (Upshur & Turner, 1995). After scoring a really strong essay, for instance, the rater may score the next essay lower than if it had followed a weaker essay. Having clearly ranked descriptions of language performance will help increase the consistency when assessing, and so rubrics have been increasingly used to assess the quality of the performance in open-ended tasks, such as speaking and writing.<\/p>\n

The term \u201crubric\u201d has sometimes been used to refer to any grading criteria. In this article, the term rubric refers more narrowly to a specific assessment tool with descriptions of various levels of the quality of the performance. Goldberg (2014) has defined a rubric as \u201ca scoring guide that outlines features of work at different levels of performance\u201d (p. 1). Holistic rubrics and analytic rubrics are two types of common tools for language performance. With holistic rubrics, the descriptions of the levels of performance include multiple traits, resulting in a single score. Holistic rubrics can be more practical when only an overall score is needed (e.g., a quick placement test, a proficiency test) (Brown & Abeywickrama, 2019) as a measure of learning that has happened (Brookhart, 2018). Since holistic rubrics provide less specific information, they are less useful for classroom assessment. Analytic rubrics, on the other hand, are more useful during the learning process (Brookhart, 2018). Analytic rubrics list multiple traits or categories separately for the rater to rank (e.g., good, better, best) based on descriptions of levels of performance. Analytic rubrics strengthen the reliability of the assessment for language produced during such tasks (Green & Hawkey, 2012). Since the instructor can give separate ratings for each category, the learner receives specific feedback about what they can do and what skills need more improvement (Brown, 2018). Additionally, the results of the assessment with an analytic rubric provides the instructor information about the strengths and weaknesses of the learners.<\/p>\n

Table 1 shows an author-created analytic rubric for an individual informational speech in a language class for beginners to serve as an example. It is designed to assess three language skills: vocabulary, grammar, and pronunciation at three possible levels: meets standard, approaching standard, and needs improvement. In Table 1\u2019s format, the lowest performance is at the bottom and each higher row has descriptions of higher level of performance.<\/p>\n

Table 1. Analytic Rubric for Informational Speech- City; High Beginner Learners<\/strong><\/p>\n\n\n\n\n\n\n
Criterion<\/strong><\/td>\nVocabulary<\/strong><\/td>\nGrammar<\/strong><\/td>\nPronunciation<\/strong><\/td>\n<\/tr>\n
Meets Standard<\/strong><\/td>\nSpeaker used specific, descriptive words to create a visual picture.<\/td>\nSpeaker created simple sentences using present tense.<\/td>\nSpeaker was clear, comprehensible and listener could understand meaning.<\/td>\n<\/tr>\n
Approaching Standard<\/strong><\/td>\nSpeaker used appropriate but general descriptive words.<\/td>\nSpeaker created simple and incomplete sentences using present tense.<\/td>\nSpeaker was comprehensible but listener had to focus to understand.<\/td>\n<\/tr>\n
Needs Improvement<\/strong><\/td>\nSpeaker used incorrect (or missing) descriptive words.<\/td>\nSpeaker created incomplete sentence with base form verbs.<\/td>\nSpeaker was generally not comprehensible even with listener effort.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

In summary, language instructors often assign communicative language tasks, but these open-ended assignments are challenging to assess because the resulting language performances are all different. Analytic rubrics are an appropriate choice for instructors when assessing open-ended extended language performances because they provide a structure for consistent assessment and feedback.<\/p>\n

In the following sections, we review how to construct an analytic rubric for teacher-implemented classroom assessment. Author-created rubrics are provided to model the concepts and steps presented. These are illustrative and not intended to be applied to specific classroom contexts. The imagined context for the following rubric is an adult, mixed-gender, integrated-skills English as an additional language (EAL) class in an English-speaking country. The students\u2019 levels range from high-intermediate to low advanced [i<\/a>] with a minimum of 52 on the TOEFL iBT or 5.5 on the IELTS to meet the entrance requirement. The students identify their motivation for language learning as academic, professional, and\/or personal. The learning goals focus on speaking and listening, and reading, writing, grammar, and pronunciation skills are developed explicitly when they support speaking and listening. We focus on a rubric for an individual speaking activity. Students engage in language practice that facilitates transfer to language use in everyday, work, and academic settings. Although a specific class situates the example rubric, the concepts and steps are applicable across multiple teaching contexts.<\/p>\n

Constructing Analytic Rubrics<\/h3>\n

Analytic rubrics have two main parts: categories to evaluate and ranked descriptions of performance (Brookhart, 2018), usually set up like a table, with the categories either listed in the columns (shown in Table 1) or the categories set up in the rows (shown in Table 2). Both the categories in the analytic rubric and levels of student performance are designed for a specific context, a specific course with a specific student population (Moskal & Leydens, 2000). The following two sections describe how to identify categories and determine performance descriptors for specific classroom assessments.<\/p>\n

Categories<\/h4>\n

Each category (i.e., evaluation criteria) in the analytic rubric assesses a different aspect of the learner\u2019s skill (e.g., Popham, 1997; Van Moere, 2014). In other words, the categories of an analytic rubric are independent. For classroom assessments, each learning objective to be assessed can be represented as a category in the rubric (Brookhart, 2018). The rubric in Table 2, for instance, has categories focused on linguistic skills (i.e., vocabulary, grammar structures, pronunciation, fluency). These categories reflect the skills required for effective communication. The rubric also includes a category on content to represent a learning objective of presenting factual information with supporting statements. The inclusion of discourse-level categories, such as content and fluency, reflects the understanding that effective communication is more than adherence to sentence-level forms that is captured by the other categories. In Table 2\u2019s format, the categories are along the leftmost column.<\/p>\n

The categories in the rubric must focus on observable and measurable skills of the performance and be directly linked to the assessment\u2019s learning objectives. For instance, perhaps the students are asked to complete the task, \u201cPresent a \u2264 4-minute prepared speech,\u201d in order to meet objectives that include:<\/p>\n

    \n
  1. Use topic-appropriate vocabulary<\/li>\n
  2. Use a variety of grammatical structures, including compound and complex sentences, and transitions that support meaning<\/li>\n
  3. Produce segmentals and suprasegmentals that are comprehensible to listeners<\/li>\n
  4. Use appropriate pacing and pausing<\/li>\n
  5. Present original or given information with relevant supporting ideas<\/li>\n<\/ol>\n

    The five categories in Table 2 relate to the students\u2019 language learning goals of the unit and reflect the course learning objectives, and the instructor can hear the evidence (i.e., observable) and evaluate performance (i.e., measurable) during the speech.<\/p>\n

    A rubric with many categories may become impractical because it is difficult to assess all of the language skills listed (Schreiber et al., 2012). Generally, five categories may be the maximum a rater can attend to (Green, 2014; Popham, 1997). No assessment can assess every skill; a rubric\u2019s categories must be carefully selected from the learning objectives for that particular assessment.<\/p>\n

    Table 2. Analytic Rubric Categories: Formal Informational Speech<\/strong>
    \n
    \n<\/p>\n

    Since productive language skills tap language knowledge, analytic rubrics to assess writing and speaking may have the same linguistic categories of vocabulary and grammar. Despite these similarities, the grammar patterns and vocabulary use differ in written text and in speech because written language can be revised (Vasylets et al., 2017; Vercellotti, 2018).<\/p>\n

    In some rubrics for assessing speaking, instructors may include nonverbal behaviors (e.g., eye-contact, gestures) to evaluate that aspect of oral communication. Raters have been found to attend to nonverbal behavior in dialogues, even when that category is not included in the rubric (May, 2009). Nevertheless, research with native English speakers has shown that nonverbal behaviors do not correlate with speech grades (Schreiber et al., 2012), which indicates that those skills do not seem to be integral to effective monologic speaking. Accordingly, nonverbal communication could be included when the course (or unit) objectives specifically include development of such skills, particularly in interactive communication, but a category for nonverbal behavior during speaking performances may not be necessary to assess speaking skill.<\/p>\n

    Levels of Performance<\/h4>\n

    After identifying each category in the analytic rubric, the next step is to create descriptions for different levels of performance in each category. The description of each level of performance should convey the criteria for that category to help the rater identify which description most aligns with the learner\u2019s performance (Goldberg, 2014). In other words, the descriptions should help the rater place the performance at a particular level for each category. It is often helpful to first describe the expected performance at the \u201cMeets Standard\u201d level because the expectations of the stated standard is most evident. After describing the expectation for the standard, a description of the level of performance in that category at one level higher or one level lower can be developed. The differences between each level of performance should be distinct (Moskal, 2002). Labeling each level can help focus the rating process around the standard (Suskie, 2009). Such labels are also useful to anchor the descriptions around the standards when drafting the ranked descriptions (e.g., Exceeds Standard, Meets Standard, Approaching Standard). The example rubric in Table 3 includes descriptive labels as well as the corresponding letter grade (used in schools in the United States). It is important to note that the rubric\u2019s top level of performance should not expect learners to have a native-like performance; learners should not be expected to have a perfect performance to receive full points.<\/p>\n

    Importantly, although the labels for the levels of performance may be evaluative terms (such as \u201cexcellent\u201d or \u201cexceeds standards), the descriptions differentiating the levels of performance should not use evaluative terms. Only \u201cdescriptive language helps students envision where they are in their learning and where they should go next\u201d (Brookhart, 2018, p. 2) and \u201cprovide a clear description of the learning goal\u201d (Brookhart, 2018, p 5). In addition, the descriptions should predominantly focus on what the learner demonstrates rather than what is absent. This recommendation does not limit the descriptions to \u201ccan do\u201d statements. Some negative descriptions might be used, but without describing what is missing. For instance, the phrase \u201cword choice may sometimes be too specific or vague<\/em>\u201d (rather than \u201cdid not use a variety of words\u201d<\/em>) focuses on what the learner produced.<\/p>\n

    Since rubrics are designed to evaluate the quality of the language performance, the descriptions must focus on distinguishing quality, rather than measuring quantity. Descriptions which differentiate levels of performance primarily with quantifiers (e.g., few, some, many) should be avoided also because it is difficult to consistently measure \u201cfew\u201d of any aspect of a language performance. For instance, rather than a focus on the number of errors, the descriptions for a category for grammatical accuracy in a language classroom can include phrases about whether the errors interfere with understanding the speaker\u2019s message. Likewise, descriptions should not attempt to differentiate levels of performance with relativistic or comparison terms (e.g., exceptional, stronger) because such terms may encourage scoring in comparison to others rather than to the stated objective. Further, since the categories in the rubric focus on the language skills (e.g., grammar, vocabulary) that contribute to a successful performance, the categories should not include requirements of the assessment itself, such as a word count or time fulfillment. In fact, student learning can be hindered by rubrics that focus on the assignment requirements (or directions) rather than describing the quality of the work (Brookhart, 2018).  Failure to meet the requirements of the assessment can be penalized, ideally outside of the rubric\u2019s categories because rubrics are a tool to assess the quality of the language performance.<\/p>\n

    It is best practice to have only \u201cas many scale points as can be well defined and that adequately cover the range\u201d of performances (Perlman, 2002, p. 8). Generally, rubrics have between three and five levels of performance (Brookhart, 2018; Suskie, 2009). With too many ranked levels, the task of matching a learner\u2019s performance to the most similar level becomes difficult and more time-consuming. Additionally, the number of categories in the rubric may influence the number of levels. For instance, an analytic rubric with a higher number of categories may, in turn, have fewer levels of performance, in order to keep the rubric a manageable size. The example rubric in Table 3 has four levels of performance arranged with the lowest performance in the column next to the categories and each higher level of performance builds up to the highest performance in the rightmost column.<\/p>\n

    Table 3. Analytic Rubric: Formal Informational Speech<\/strong><\/p>\n\n\n\n\n\n\n\n\n
    Category<\/strong><\/td>\nIn Progress<\/strong>
    \n(below D)<\/strong>
    \n3 points<\/em><\/strong><\/td>\n    		Approaching Standard (C-)<\/strong>
    \n4 points<\/em><\/strong><\/td>\n    		Meets Standard<\/strong>
    \n(B+)<\/strong>
    \n4.5 points<\/em><\/strong><\/td>\n    		Exceeds Expectations <\/strong>
    \n(A)<\/strong>
    \n5 points<\/em><\/strong><\/td>\n<\/tr>\n
    Vocabulary<\/strong><\/td>\nWords often unsuitable for task; errors interfere with meaning<\/td>\nWord choice is limited or reduces effective expression of meaning<\/td>\nVariety of words; word choice may sometimes be too specific or vague but meaning is clear<\/td>\nVariety of words;
    \nword choice supports meaning & appropriate for task<\/td>\n<\/tr>\n
    Grammar Structures<\/strong><\/td>\nLimited variety of structures & transitions; errors interfere with meaning<\/td>\nVariety of useful structures & transition words;
    \nsome choices
    \nreduce effective expression of meaning<\/td>\n    		Variety of structures; useful transition words; grammar choices may be generic but appropriate for meaning<\/td>\nVariety of complex structures & effective transition words; grammar choices support meaning<\/td>\n<\/tr>\n
    Pronunciation<\/strong><\/td>\nSubstitution of sounds, suprasegmental patterns interfere with meaning; difficult to understand by all audiences<\/td>\nSounds substitutions and suprasegmentals occasionally slow down comprehension but overall meaning is understandable to people familiar w\/ENL speech<\/td>\nMinor, predictable sound substitutions which are understandable to people unfamiliar with ENL speech<\/td>\nSegmentals and suprasegmental patterns are understandable to all audiences<\/td>\n<\/tr>\n
    Fluency<\/strong><\/td>\nPacing, pausing, and\/or fillers interferes with comprehen-sibility<\/td>\nPacing, pausing and\/or fillers slows down comprehensibil-ity<\/td>\nAppropriate pacing and pausing<\/td>\nPacing and pausing enhance message<\/td>\n<\/tr>\n
    Content<\/strong><\/td>\nTopic not clear; information may not be factual or relevant to topic<\/td>\nSpeech is focused on a single unstated topic; information was factual but incomplete, general, or less relevant<\/td>\nTopic was explicitly stated; information was factual and relevant to the topic and generally supported<\/td>\nTopic was explicitly stated; information was factual and relevant to the topic; specific details enhance message<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    The descriptions of the levels of performance could be created based on previously scored learner performances, for example, by sorting the graded performances and identifying the features that differentiate the quality at each level for each category. Often, however, the instructor must create the descriptions at each performance level based only on the course expectations and previous experience with the teaching context.<\/p>\n

    Points and Weighting<\/h4>\n

    When designing classroom assessment, considerations for assessing student work and assigning a score to the assessment must be addressed. In Table 1, we presented a rubric that described student performance without assigning a point value, but instructors and students may benefit from assigning points to performance because assigning points allows the instructor to grade the performance and the learner has concrete feedback of the distance between their performance and the expected standards of the activity. In other words, the levels in a rubric serve to assign a score and provide feedback to the learner (Brown, 2018; Goldberg, 2014).  In this section, we explain setting the point value of each performance level in relation to other levels and the weight of each category in relation to other categories. At the end of this section, we explore the question of using a range of scores for each performance level.<\/p>\n

    When assigning point values, the difference between point values should represent the difference in quality between the levels. For instance, in Table 2, the point values in the \u201cMeets Standard (B+)\u201d level is only slightly lower than the \u201cExceeds Expectation (A)\u201d level because the quality of the performances described is only slightly less successful. It is also important to consider how the resulting grade percentage and\/or letter grade align with the descriptions of performance. For instance, if the second-best level of performance equates to a \u201cB\u201d performance, the numbers assigned at that level should equal a percentage in the B range. Note also that according to the rubric in Table 2, meeting the standard is not sufficient to earn full points, which may not be appropriate in some teaching contexts. The point values also make sense mathematically and do not penalize students unintentionally. For instance, assigning point values of 1, 2, 3, and 4 when there are four levels of performance may be incongruous for grading, unless the descriptions reflect vastly different performances which should earn 25%, 50%, 75%, and 100%.<\/p>\n

    Additionally, analytic rubrics allow the instructor to weight the categories (Green, 2014; Popham, 1997), which means that a specific category can be worth more points in the analytic rubric, based on the course focus, unit focus, or learner level. For instance, the rubric shown in Table 4 lists five total categories with the categories of fluency and content worth half the value of the other categories, indicating the relative importance of those categories. Ballard (2019) found that the arrangement of the categories may inadvertently influence raters\u2019 perception of which categories are most important, in that the leftmost column may be perceived as most important while the rightmost column may be perceived as the least important. This finding suggests that weighted categories should be placed left-most or topmost to align with rater\u2019s expectations, to avoid causing a conflict in expectations.  Finally, when weighting categories, it is more efficient to list the point values in the individual cells, which can be circled, rather than having a multiplication notation (e.g., x2 for a category) because the notation requires another calculation for the rater and is often confusing to (or overlooked by) learners.<\/p>\n

    Table 4. Weighted Analytic Rubric: Formal Informational Speech<\/strong><\/p>\n\n\n\n\n\n\n\n\n
    Category<\/strong><\/td>\nIn Progress<\/strong>
    \n(below D)<\/strong><\/td>\n    		Approaching standard<\/strong>
    \n(C-)<\/strong><\/td>\n    		Meets Standard<\/strong>
    \n(B+)<\/strong><\/td>\n    		Exceeds Expectations <\/strong>
    \n(A)<\/strong><\/td>\n<\/tr>\n
    Vocabulary<\/strong><\/td>\nWords often unsuitable for task; errors interfere with meaning
    \n(6)<\/td>\n    		Word choice is limited or hinders effective expression of meaning
    \n(8)<\/td>\n    		Variety of words; word choice may sometimes be too specific or vague but meaning is clear
    \n(9)<\/td>\n    		Variety of words;
    \nword choice supports meaning & appropriate for task
    \n(10)<\/td>\n<\/tr>\n
    Grammar Structures<\/strong><\/td>\nLimited variety of structures & transitions; errors interfere with meaning
    \n(6)<\/td>\n    		Variety of useful structures & transition words;
    \nsome choices hinder effective expression of meaning
    \n(8)<\/td>\n    		Variety of structures; useful transition words; grammar choices may be generic but appropriate for meaning
    \n(9)<\/td>\n    		Variety of complex structures & effective transition words; grammar choices support meaning
    \n(10)<\/td>\n<\/tr>\n
    Pronunciation<\/strong><\/td>\nSubstitution of sounds, suprasegmental patterns interfere with meaning Difficult to understand by all audiences
    \n(6)<\/td>\n    		Sounds substitutions and suprasegmentals occasionally hinder but overall meaning is understandable to people familiar w\/ENL speech
    \n(8)<\/td>\n    		Minor, predictable sounds substitutions and suprasegmentals which do not interfere with meaning; understandable to people unfamiliar with ENL speech
    \n(9)<\/td>\n    		Segmentals and suprasegmental patterns are understandable to all audiences
    \n(10)<\/td>\n<\/tr>\n
    Fluency<\/strong><\/td>\nPacing, pausing, and\/or fillers interferes with
    \ncomprehensibility
    \n(3)<\/td>\n    		Pacing, pausing and\/or fillers hinders comprehensibility
    \n(4)<\/td>\n    		Appropriate pacing and pausing
    \n(4.5)<\/td>\n    		Pacing and pausing enhance message
    \n(5)<\/td>\n<\/tr>\n
    Content<\/strong><\/td>\nTopic not clear; information may not be factual or irrelevant to topic
    \n(3)<\/td>\n    		Speech is focused on a single unstated topic Information was factual but incomplete, general, or less relevant
    \n(4)<\/td>\n    		Topic was explicitly stated; information was factual and relevant to the topic and generally supported
    \n(4.5)<\/td>\n    		Topic was explicitly stated; information was factual and relevant to the topic; specific details enhance message
    \n(5)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    Some rubrics include a range of points within each level of performance (e.g., 18-19-20). This option adds flexibility to the scoring, but it counters one of the main purposes of the analytic rubric, namely to describe the features of each level of performance for the rater and for the learner. As Perlman (2002, p. 8) has stated \u201ceach point on the scale should be clearly labeled and defined\u201d. Therefore, a range of scores within a level lowers the reliability because the rubric does not include guidelines for awarding the points within each level; thus, the instructor may not consistently score student work as an 18 vs. 19 vs. 20 when each value is connected to the same description of performance. Additionally, such rubrics are less helpful for the student to know how to improve or for the instructor to know what to teach. <\/a> Accordingly, a range of available points within levels of performance should be used sparingly in classroom assessment. An alternative solution is to describe the levels of performances for specific point values (e.g., 20, 18, 16, and 14) and leave scores in-between (e.g., 19, 17, 15) unspecified. The rubric can note that the scores in-between the described levels of performance would represent a performance slightly stronger than the lower score but not quite fulfilling the description of the higher score. (See Appendix B<\/a> for an example.) This formatting allows performances to be placed in a single point in the scoring according to how it maps onto the descriptions or in between the stated descriptions. This option may be the best balance between reliability and practicality (e.g., time, effort) when assessing tasks where the performances will vary greatly and when differentiating slight differences in quality is desired.<\/p>\n

    Pre-use Review<\/h4>\n

    As with all materials development, rubrics require a careful review and an iterative revision process, particularly the descriptions of the levels of performance. This revision process, labeled as pre-use review, involves 1) ensuring that the expectations in the descriptions match the teaching context, 2) checking for consistency in the performance descriptors across levels, 3) considering the practicality of the rubric, and 4) improving the beneficial consequences of the rubric on student assessment and language development. These pre-use review components are described below.<\/p>\n

    First, the descriptions of level of performance should describe expectations appropriate to the context. Recall that descriptors must focus on the quality of performance, allow the rater to identify the appropriate level of the student performance, and provide the student with insight into their current and potential learning. The descriptions can sometimes unintentionally muddle the underlying construct represented in the learning objective that the category is meant to evaluate. For instance, Isaacs (2014) warned about including allusions to accentedness because learners of a language will commonly have an accent, but the accent itself does not prevent listeners from comprehending the speaker. In fact, the descriptions should not expect native-speaker-like performance (Green, 2014) because in most contexts learners should not be expected to have native-like performance, for instance in pronunciation (Isaacs, 2014; Purpura, 2016).<\/p>\n

    Second, the descriptions across the levels of performance should display parallelism, including consistency in language and syntax (Goldberg, 2014) so that the learners\u2019 work can be consistently assessed across the levels. Further, the descriptions should be concise without \u201cdysfunctional detail,\u201d which includes long descriptors that try to describe every possible nuance within each level of performance (Popham, 1997, p. 74).<\/p>\n

    Third, the resulting rubric should be practical. Well-crafted descriptions of levels of performance will make the grading process more efficient (Ambrose et al., 2010) when the levels are easily compared. Practicality is further improved when the descriptions help the rater focus on the specific features which differentiate levels.  Font enhancements, such as bolding, may be useful, when used judiciously (Goldberg, 2014). Then, extraneous and redundant wording perhaps can be eliminated. Depending on the context, phrases (rather than sentences) may be sufficient and efficient to describe the performances. When fewer words are used in the descriptions, the grading can be quicker because the instructor has less to compare to identify the most appropriate level. It is also important to review if the rubric\u2019s format is easy to use. Analytic rubrics can be visually crowded, so spacing and formatting should be carefully considered. At the same time, a one-page rubric is likely easier to use than having categories or descriptions of levels of performance across pages.<\/p>\n

    Fourth, a well-constructed rubric also improves the beneficial consequences of the assessment because classroom assessments should be in service to learning. An analytic rubric can also clarify the instructor\u2019s expectations, serving as a blueprint for success for the learners, a benefit of the assessment process known as positive washback (Green, 2014). Accordingly, the categories and the descriptions of the levels of performance should be understandable to all users, the instructors who will assess and the learners who will be assessed (Moskal, 2002; Pui et al., 2020), so that the grading process is transparent and the scores provide feedback to the learner (Brown, 2018; Goldberg, 2014). While reviewing the rubric for transparency, think about how to explain the rubric\u2019s components and scoring procedure to the students. In addition to giving a score for each of the categories, instructors may give individual feedback on the language performance. The rubric formatting can facilitate feedback by including a place for comments, either for each category, or general comments, perhaps below the rubric.<\/p>\n

    Post-use Evaluation<\/h4>\n

    After using any assessment tool, an evaluation and reflection of the assessment process is necessary so that the instructor can reflect on the teaching implications (Ambrose et al., 2010) and the opportunities for improving the assessment. Despite careful design, no rubric will be perfect. Through a review process, both the rubric and the assessment process can be improved. The following content presents specific, feasible suggestions for reviewing a rubric after using it for classroom assessment.<\/p>\n

    The post-use review should include an investigation of the resulting scores. The class mean (average) and the range (the lowest and highest scores) for the assessment should be calculated to consider whether the scores are meaningful and useful for understanding the learners\u2019 skills. It is also useful to review the mean and the range of scores for each section of the assessment. If the learners\u2019 performances were organized in rank order, lowest to highest score, they should reveal a corresponding increase in quality. Any discrepancies can be reviewed to improve the assessment\u2019s scoring. Additionally, any two or more performances which were given the same score in the stated scoring system should have equivalent quality. If the post-use review suggests that they are not, the scoring system may be revised to reflect the difference in quality. Of course, any changes should align with the purpose of the assessment and the assessment\u2019s stated learning objectives.<\/p>\n

    Both the categories and the descriptions of the levels of performance of analytic rubrics should be carefully reviewed after using the rubric. A key principle of an analytic rubric is that the categories are distinct with each giving valuable information about the learner\u2019s skills (e.g., Youn, 2015). The independence of the rubric categories can be reviewed by checking if any two categories pattern together. For instance, speakers who are rated as \u201cmeets expectations” in category X have also been rated \u201cmeets expectations” in category Y; and speakers rated \u201cinsufficient\u201d in X have also been rated as \u201cinsufficient\u201d in Y. If this is the case, those categories may be confounded or intertwined based on how the descriptions are written. With confounded categories, learners who are deficient (or superior) in one category will not be able to score higher (or lower) in another category; a rubric with confounded categories is incongruent with a main advantage of analytic rubrics, which is assessing various skills separately. To remedy this issue, the descriptions can be revised so that the categories are independent, ensuring that the skill level in one category does not limit the speaker\u2019s opportunity to demonstrate skill in another category. Another possibility is that the categories are inherently connected. For instance, productive vocabulary use and collocation use both increase with proficiency as part of vocabulary development (Bonk, 2011). If the two categories are inherently related, those categories could be combined in the rubric, or one category could be eliminated, knowing that the other category has been shown to effectively measure the construct.<\/p>\n

    Research has shown that raters sometimes evaluate language performances using their own ideas of what is relevant (Fulcher, 2003), perhaps implicit, unstated expectations (Moskal & Leydens, 2000). Consequently, it is important to honestly consider whether the scoring followed the descriptions in the rubric. Consider possible sources for any inconsistency and how the assessment could help the rater evaluate more consistently. Further, the best learners\u2019 performances can be reviewed for shared features; such a review may suggest features that better differentiate quality of performances. If it was difficult to place student work into one of two levels of performance in a certain category, those descriptions of level of performance can be clarified or more finely separated to capture the differences among the learners\u2019 performance in that particular context.<\/p>\n

    A post-use review of an analytic rubric should also look carefully at the number of students placed at each level of performance. If the majority of learners have not met the stated level of performance, further instruction\/practice is needed so that the learners are meeting the objective or the stated expectations in the rubric may need to be adjusted to the teaching context. Conversely, if most learner performances earned the same score in one category in the rubric, that category fails to differentiate between levels of performance. That result may, in fact, be appropriate, such as when all learners have met the stated standard. With a standards-based assessment, each learner\u2019s work is assessed to the stated standard (not to the other learners), which means potentially every student can pass or even get full points (Green, 2014).<\/p>\n

    Alternatively, if all students appear to perform at the same level, this result may indicate that the descriptions of levels of performance for that category are too broad. If so, an adjustment should be made to better reflect the context, specifically differences in proficiency for those learners. Typically, the full range in the rubric should be used, sorting learners into each of the rubric\u2019s levels of performance because<\/em> the classroom assessment is designed for those specific learners. If, for instance, a post-use evaluation reveals that no performance was rated as \u201cinadequate\u201d for any category, the instructor should consider deleting that level when none of the performances of your particular learners fit the description for that level. On the other hand, an unused lowest level of performance can be used to shield learners from negative affect. Even though the ratings will accurately describe the learner\u2019s performance, the rubric format can allow every learner to be above the \u201cworst\u201d description.  Likewise, the same inquiry can be considered within each category, where superfluous descriptions can be deleted. Generally, when the rubric can be simplified by deleting unnecessary descriptions, practicality is strengthened. The practicality can also be evaluated with a review of how easy it was to grade the student work with the rubric. If the rubric (or a section of the rubric) was difficult to use, consider how the assessment could be improved. Often, simply adjusting the order of the evaluation categories (to match the order during the assessment process) or adjusting the physical layout of the rubric improves practicality.<\/p>\n

    Summary<\/h3>\n

    <\/a>
    \nAn analytic rubric is a tool to more objectively and reliably measure the quality <\/em>of the language performance; it is a tool to assess language production in open-ended tasks where the learner has freedom to decide what vocabulary and grammar constructions to use. Despite the recognition of the importance of assessing productive language skills, the assessment of language performance has several challenges. No rubric is perfect; therefore, revision before and after using it is always necessary. The steps listed below provide a summary of the process for constructing strong analytic rubrics for classroom use. (Note: A more detailed list in provided in     Appendix A<\/a>):<\/p>\n