THE ORIGINS OF CA

Get Complete Project Material File(s) Now! »

Chapter 3. Methodology

Ellis (2012) points out two principal research paradigms for classroom-based L2 research: the normative paradigm that “seeks to test hypotheses drawn from an explicit theory of L2 teaching or learning and typically involves some form of experiment,” and the interpretive paradigm that “seeks to describe and understand some aspect of teaching by identifying key variables and examining how they interrelate” (p.x). The present study obviously fits into the interpretive paradigm because it sought to describe the participant teachers’ CA practices in depth, holistically, and in context, so as to reveal the typical patterns of these practices in their EFL speaking classrooms.
To achieve such a purpose, this study adopted a multiple-case study approach. This chapter, after the presentation of the research questions, will discuss the rationale for the case-study approach. What follows is a description of the identification of the cases, the data-collection instruments and procedures, and the process of data analysis.

Research questions

This study aimed to uncover and crystallize EFL teachers’ CA practices in naturalistic classroom settings. As discussed in chapter 2, CA, as an umbrella term, is not only about grading students but also about enhancing teacher teaching and student learning. In other words, CA includes both SA and FA. While SA practices should be salient in this EFL context since the educational culture in China is still examination-dominant (Carless, 2011; Cheng & Curtis, 2010b), this study also made an effort to capture the teachers’ FA practices, especially those remained oblivious to the teachers but fit the definition of CA. Therefore, this study addressed the following research questions.
1) How did the teachers assess their students in their oral English courses?
1a) What were the recognized CA practices, including both SA and FA, in these oral English courses? What was each practice like?
1b) Were there any unrecognized CA practices in the teachers’ oral English courses? If yes, what were they like?
2) Why did the teachers assess their students the way they did?
3) What were the students’ perceived impacts of their teachers’ CA practices?
3.2 Rationale for the case-study approach
In essence, this study was an exploratory and descriptive study, aiming to uncover and crystallize EFL teachers’ CA practices in their own oral English classrooms in China. Because the literature suggests that CA is a multi-faceted multidimensional phenomenon, performing multiple functions in localized classroom contexts, a case study approach is the best choice to capture such a complex phenomenon, thanks to its advantages such as allowing contextualized, in-depth and holistic understanding of a complex issue and allowing for mixed methods and multiple research paradigms (Bassey, 1999; Dornyei, 2007; Mackey & Gass, 2005; Merriam, 1988; Patton, 2002; Yin, 2009). As Nunan and Bailey (2010, p. 158) points out, a case study is a “detailed, often longitudinal, investigation of a single individual or entity”, and “as a type of naturalistic inquiry,” it does not “involve any sort of treatment” but allows the researcher to “learn what is happening.” These features just serve the purpose of the present study well.
The decision to carry out a multiple-case rather than a single-case study was based on Yin’s (2009) suggestion as well as the real situation in China. Yin (2009) said “single-case designs are vulnerable … because you will have put ‘all your eggs in one basket.’ More important, the analytic benefits from having two (or more) cases may be substantial” (p.61). In China, although oral English is widely offered to university students, there is a wide range of variations from university to university and from department to department concerning textbooks used, course requirements, course syllabus, students’ English levels, teachers’ teaching experience, etc.. Therefore, it was difficult to justify any one case as a representative case for this particular context. Consequently, a multiple-case approach could allow the researcher to select a few cases in contrasting situations, which could enhance the generalizability of the study findings (Dornyei, 2007).

Identifying the cases

In case studies, defining the case is of paramount importance (Simons, 2009). Although traditionally a case is defined as an entity, such as a person, a classroom, a policy, a process, etc., researchers have pointed out decisions about case selection also concern specific aspects of the selected entity to be investigated, since it is impossible to study every aspect of a selected entity (Duff, 2008; Yin, 2009). Since a case is a “bounded instance” (Nunan & Bailey, 2010, p. 161), it is important to set boundaries so that the study remain focused and how the study is related to readers can be made explicit (Merriam, 1998; Yin, 2009). Following Yin’s (2009) suggestion that the defining of a case should be directed by the research purpose and research questions, the present study regarded a university EFL teacher’s CA practices in his/her oral English course in China as the unit of analysis in this study. More specifically, this study examined: EFL teachers, not ESL or EAL teachers; tertiary-level teachers, not primary school or middle school teachers; oral English teachers, not teachers teaching other subjects such as reading or writing; the teacher’s CA practices, not his/her teaching strategies, although some of his/her CA practices might also function as teaching strategies such as incidental assessment opportunities (Hill, 2012).
At tertiary level in China, there are various types of universities such as comprehensive universities, foreign language universities, universities of science and technology, universities with specialties such as law, forestry, and medicine. In spite of such variations, almost every university has both English-major programs, where students have English as their major and spend most of their time learning and perfecting their English language skills, and non-English-major programs, where students have their own majors and they spend much less time learning English. In addition, while most universities still employ Chinese teachers to teach oral English, more and more native English speakers are being hired to teach oral English, especially to English-major students. Moreover, with society valuing a person’s oral communication skills more and more in China, not only English-major students but also non-English-major students attach great importance to their oral English courses.
Bearing such variations in mind, the researcher made an attempt to carry out “maximum variation sampling” (Miles & Huberman, 1994, p. 28). She approached 16 teachers from ten different universities in Beijing. In the end, given the strong commitment this study required on the part of the participants, three teachers from two universities were recruited. As explained above, the intention was to recruit teachers who varied according to their personal characteristics, their educational background and the kind of courses they were teaching to achieve maximum variation sampling. As the profiles in Table 3.1 show, this variation was achieved to a large extent.

Data collection

To address the research questions, the present study adopted a variety of data-collection methods, mainly qualitative in nature, conducted throughout the whole semester of the selected courses. Miles and Huberman (1994) pointed out that qualitative research has three strengths: a strong handle on what ‘real life’ is like, a strong possibility for understanding latent, underlying, or non-obvious issues, and strong potential for revealing complexity (p.10). Such strengths serve the purposes of the present study well, that is, to gain an in-depth and comprehensive understanding of a complex and contextualized phenomenon.
A small-scale one-case pilot study was first conducted in January 2010, aiming to check and refine the data-collection instruments and rehearse the procedures for carrying out such fieldwork studies. The pilot study was conducted in an advanced-level ESL speaking course in a New Zealand university from Jan. 5, 2010 to Jan. 19, 2010. This was a summer course, with four sessions each week (100 minutes for each session). Every four sessions of this 24-session intensive course were devoted to one unit. After each unit, two sessions were scheduled for an end-of-unit formal assessment. This pilot study focused on the first unit of the course, together with its end-of-unit formal assessment, which altogether lasted for one week and a half. This class was very small, with only six students, all immigrants from Asian and Pacific countries. The teacher, a native New Zealander, and the six students participated in the pilot study. Based on this pilot study, revisions were made concerning the instructions for student journals, the interview guides, and the way of conducting stimulated recall, which, to avoid misunderstanding, was renamed as stimulated retrospective interviews because they were not conducted the way a typical stimulated retrospective interview is usually conducted (Gass & Mackey, 2000). These revised and improved versions of the instruments were used in the main study, which will be described below.
In what follows, I will first describe each of the data-collection instruments used in the main study, and then describe the data-collection procedures at each site.

Data-collection instruments

Richards and Morse (2007), when discussing the design of a qualitative study which will be as valid as possible, pointed out that the researcher should try to achieve a high level of alignment between the research questions, the data collected, and the methods used to collect the data. Therefore, guided by the research questions and existing literature and taking into consideration the actual settings, this study collected four types of data through four data-collection instruments (see Figure 3.1), as an effort to maximize the validity of the present study.
In what follows, the data collection instruments will be described in turn.
Documents
Documents were a valuable source of data “not only because of what can be learned directly from them but also as stimulus for paths of inquiry that can be pursued only through direct observation and interviewing” (Patton, 2002, p. 294). This research method was considered valuable for the present study due to the nature of this study: an exploratory and descriptive study of EFL teachers’ CA practices in context. As CA practices are always contextualized, the researcher analyzed the following publicly accessible official documents including the National Curriculums for English Majors and Non-English Majors and website information about each teacher’s school and university, paying special attention to the university/school guidelines and/or principles regarding assessment of student learning. The analysis of this group of documents helped situate the present study in a larger context, especially what was mandated concerning CA, which served as a basis for the practices that actually happened in classroom context.
In addition to analyzing the publicly accessible documents, the researcher also analyzed the teachers’ textbooks, course descriptions, teacher group meeting minutes, test papers, marking schemes, lesson plans, if available. The analysis of this group of documents not only provided a starting point for the identification of the teachers’ CA practices, but also informed other methods of data collection such as the fieldwork observations, the interview questions posed to the participants, and student journal topics. They were also used together with data obtained from observations and interviews to reveal the connection between stated and actual CA practices.
Classroom recording and observation To capture the instruction-embedded FA practices, many researchers have adopted the method of classroom discourse analysis which helped them analyze classroom recording data and generate patterns from the analysis (Cowie & Bell, 1999; Leung & Mohan, 2004; Rea-Dickins, 2001, 2006; Torrance & Pryor, 1998). As the present study took a very broad view of CA, in order to reveal all the existing CA practices, including those embedded in daily teaching and learning activities, the researcher recorded some of the participant teachers’ lessons, which served as the working data for discourse analysis in order to reveal the hidden CA practices. To strike a balance between the maximum amount of data collected and the minimum interference on a teacher’s normal teaching, the researcher recorded four weeks’ lessons from each teacher, including daily teaching activities as well as part of the midterm test in Mary’s case and some marked assignments in Linda’s and Andrew’s cases.
While video-recording is more powerful than audio-recording in uncovering the subtle reality of classroom life, it tends to cause more anxiety on the participants’ part and consequently may distort the natural data (Zuengler, Ford, & Fassnacht, 1998).
Therefore, to make the data-gathering less intrusive, audio-recording instead of video-recording was adopted. During the observed lessons, the researcher usually came earlier and put a recorder inside the teacher’s pocket or on the teacher’s desk or near where the teacher would usually sit, to ensure that the teachers’ voices were clearly recorded. The researcher did not put another recorder among the students so as to minimize any potential anxiety that might be caused by the recorder. As the recorder is guaranteed to capture sounds within 40 meters, it successfully captured the sounds when students talked individually in class, but when students talked in small groups, sometimes it was too noisy to distinguish who was talking what.
To complement the audio-recordings, the researcher also conducted classroom observation. It has been pointed out that the way to conduct classroom observation can vary along two dimensions. Firstly, the researcher’s role in the classroom may vary from “participant” to “non-participant” (Dornyei, 2007, p. 177). Because the purpose of the present study was to obtain naturally occurring data, the “non-participant” stance was adopted. The researcher tried her best to minimize the influence of her presence on the observed classes by observing classes from a position located towards the back of the room, close to students but not so close to attract undue attention.
In addition, the observation data also vary from “structured” to “unstructured” (ibid, pp. 177-178). Considering that FA practices may be highly integrated within classroom teaching activities and may vary in form, time and size (cf. chapter 2), no a priori classroom observation scheme, such as the Communicative Orientation of Language Teaching (COLT) (Spada & Frohlich, 1995), was used. Instead, a loosely-structured observation instrument was used, which consisted of four columns: time (starting time and ending time of a classroom activity), teacher’s words and
behaviours, students’ words and behaviours, and the observer’s comments and reflections (Appendix 1). The time column helped trace the classroom activities on my recording. The two middle columns were used to remind the researcher of what an activity or an interaction episode was about afterwards. The last column was the researcher’s real-time judgment of whether an activity or an interaction episode was potentially a CA practice. If an activity was noted as a potential CA practice according to the field notes, the researcher generally paid more attention to this activity during the follow-up stimulated retrospective interviews.
Teacher interviews
To find out teachers’ explanations of their own assessment practices, teachers’ self-report data about their beliefs on teaching, learning, and assessment, as well as their explanations of their own CA practices should be sought. To obtain such type of data, the interview has been a widely used method.
The interview is considered a highly interactive means of gathering data and is valuable because not everything can be observed (Fontana & Frey, 2003; Patton, 2002). It is not possible to directly observe feelings, thoughts, and intentions. Nor is it possible to observe “how people have organized the world and the meanings they attach to what goes on in the world” (Patton, 2002, p. 341). Therefore, interview data can supplement and triangulate with observation data. Out of this concern, interviews with the teachers were conducted in this study.
In this study, three interviews were conducted with each participant teacher. First, a baseline interview was conducted at the beginning of the semester to find out each teacher’s educational background, teaching contexts, current teaching practices, and beliefs on teaching, learning, and assessment. At the end of this interview, the arrangements for classroom observations were also made (Appendix 2).
Second, at the end of the semester, each teacher was interviewed again to find out how they would summarize and reflect on his/her overall teaching experiences, with special attention given to their CA practices (Appendix 3).
Both the baseline interview and the end-of-semester interview followed a semi-structured interview guide for the following reasons. First, by listing the key questions and issues to be explored in an interview, the guide ensured that the most important issues were covered during the limited time available in an interview situation. Second, because this was a multiple case study, the guide might “ensure that the same topic lines of inquiry are pursued with each person interviewed” (Patton, 2002, p. 343). Third, although the key issues had been listed in the guide, the researcher still had the freedom to explore, probe, and ask questions that could illuminate this complex phenomenon: teachers’ CA practices.
In addition to the above two interviews, each teacher had a stimulated retrospective interview right after one observed lesson. The purpose was to capture the teacher’s thought processes involved in carrying out a task or activity where assessment was involved.
Generally, to obtain this kind of data, stimulated recall (SR) is a commonly used method (Gass & Mackey, 2000). However, during the pilot study the researcher noticed that almost all the activities in the oral English course might involve assessment and it was very time-consuming to conduct each SR interview by replaying the recording for each of the 100-minute sessions. Therefore, during the interview, the researcher just summarized all those activities or episodes according to the field notes and used the summaries as cues rather than replaying the recordings, which was somewhat different from the suggested practices for doing SR interviews (Gass & Mackey, 2000; Lyle, 2003). Therefore, it should be more appropriate to call this method a stimulated retrospective interview. Appendix 4 describes how the researcher prepared each teacher for this kind of interview and presents the generic questions asked during such stimulated retrospective interviews with each teacher.
Student questionnaires
To answer the third research question, students’ self-report data were needed about their sense of achievement and their feelings about how the CA practices adopted in their courses affected their motivation, self-efficacy, effort, and learning. Considering the fairly large number of students in each class, three ways were adopted to collect data from the students: student questionnaires, student journals, and student interviews. This part will describe student questionnaires only. The other two methods will be described afterwards.
At each site, all the students from the selected teacher’s oral English class were asked to complete two questionnaires, one at the beginning of the semester and one near the end. The two questionnaires were used in this study for the following two reasons. First, because the researcher was only able to follow a limited number of students extensively in each setting, the convenience of a questionnaire survey (Bryman, 2001; Dornyei, 2003; Neuman, 2003) enabled the researcher to obtain data from more students, and thus get a more comprehensive picture. Second, the questionnaire data could complement and triangulate with those qualitative data obtained through classroom observations, student journals, and interviews, and thus enhance the internal validity of the study.
Aiming to get baseline information about the students, the beginning questionnaire (Appendix 5) consisted of two main parts: students’ demographic information and their perceptions of their oral English learning including their self-evaluation of their oral English ability (items 1-7), their goal orientations in learning oral English in the selected course (items 8-16)6, their anxiety in speaking English (items 17-19) and their effort in improving their oral English (items 20-23). The second part of the questionnaire was derived from empirical studies (Brookhart & DeVoge, 1999; Brookhart & Durkin, 2003; Brookhart et al., 2006; Wang & Cheng, 2010), which suggest that CA practices may influence students’ perceived self-efficacy, motivation (specifically goal orientations), anxiety, effort, and overall achievement. Because students’ perceived self-efficacy is usually associated with specific assessment tasks (Brookhart & Devoge, 1999) and the questionnaire for the present study was not directed towards any specific assessment practices, items on self-efficacy were not included in the questionnaire. For the second part, students were asked to indicate the degree to which they felt each statement was true on a five-point Likert scale (1 = strongly disagree, …, 5 = strongly agree). This part appeared both in the beginning questionnaire and in the end-of-semester questionnaire in order to see if students reported any changes in these aspects over one semester.
Near the end of their oral English course, all the students were asked to do a questionnaire survey again (Appendix 6). This end-of-semester questionnaire consisted of three parts. In addition to the two parts contained in the beginning questionnaire, this questionnaire had an additional part to find out students’ perceptions of some typical CA practices, their frequencies and their usefulness to their learning. The typical CA practices were designed according to Wiliam and Thompson’s (2008) model of FA (Figure 2.6). Specifically, this part tried to find out if teachers clarified and shared with students the learning intentions and criteria for success (items 1-5 and 20-22), how teachers elicited evidence of learning (items 6-8, 18, 19), how teachers provided feedback to students (items 9-15), and whether teachers involved students in CA (items 16, 17). Students were also given the space to add other CA practices that were not included in the questionnaire. The inclusion of this part III was intended to not only elicit students’ general impressions of the CA practices in the selected courses, but also to see if there might be some relationship between their perceptions of the CA practices and their motivation, anxiety, effort, or overall self-evaluation. Such findings could also be used to triangulate with findings obtained through qualitative methods such as classroom observations, student interviews, and student journals. At the very end of the questionnaire, students were also invited to express their opinions and feelings about their oral English course.
English versions of the two questionnaires were tried out in the pilot study. However, because of the small number of student participants (only 6 students) and the short time interval between the two questionnaires (only one week and a half), only the wording of some items was improved based on the respondents’ feedback.
Since the present study was conducted in China, to ensure that students understood the questionnaires accurately, the researcher translated all the questionnaires into written Chinese and then asked a Chinese colleague who is proficient in English to translate them back into English. The translated versions and the original versions were compared and all the discrepancies were discussed and revised to make sure the items in Chinese were accurate translations of the items in English.
Before being used in my main study, the Chinese version of the end-of-semester questionnaire was tried out with 48 first-year English-major freshmen in one university in China. Because of the time constraint, the researcher was unable to try out both questionnaires with the right time interval in between, so only the end-of-semester questionnaire was tried out since it incorporated the initial questionnaire, aiming to check if the wording of the items was clear to the potential students. Based on the respondents’ feedback, the researcher made revisions accordingly. Appendixes 5 and 6 contain the two revised questionnaires used in the main study.
However, the analysis of the questionnaire data from Andrew’s and Mary’s classes showed that on the one hand, the data were not suitable for inferential analysis due to the small number of students from each class, and on the other hand, students from the same class gave very different responses regarding their teacher’s classroom practices, indicating that the questionnaire failed to provide an accurate picture of a teacher’s CA practices (cf. Table 4.13). Therefore, the end-of-semester questionnaire was further modified (Appendix 7), aiming to get more straightforward answers from the students about their understanding of what constituted CA practices in their class (Part II), their perceptions of improvement, if any, regarding their overall self-evaluation, goal orientations, anxiety, and effort (Part III), and if they perceived any relationship between CA practices and their improvement (Part IV). Most of the CA practices listed in Parts III and IV were the same as those in the previous version (Appendix 6) but the revised items were specifically related to Linda’s classroom activities. This revised questionnaire was used in Linda’s class.
Student journals
To go deeper into students’ perceptions of how they had been engaged in daily CA practices and their perceived impacts of such practices on themselves, volunteer students from each selected class were invited to write journals after each observed lesson.
Journal writing as a data-collection method usually refers to “solicited diaries”, that is, accounts produced specifically at the researcher’s request by an informant (Bell, 1999; Dornyei, 2007; Mackey & Gass, 2005). This method allows the researcher “an unobtrusive way of tapping into areas of people’s lives that may otherwise be inaccessible” and allows the researcher to “study time-related evolution or fluctuation within individuals by collecting data on many occasions from the same individuals” (Dornyei, 2007, p.157). Since the present study lasted for one semester, inviting students to write journals was thought to be an appropriate and effective way to accomplish this research purpose.
In the pilot study, the researcher tried out a journal template that consisted of three prompts: the students’ experiences and feelings of the classroom activities in an observed lesson, what the students had learned about how to improve their speaking skills, and what had brought about such learning. After talking with the student participants and analyzing the data collected in the pilot study, the researcher realized that students could not recall a lesson clearly and their comments were very general and sometimes vague. In order to elicit more specific responses from the students, for each journal the researcher gave students a list of questions concerning particular activities based on the classroom observation and required students to give more specific responses concerning their feelings. An example can be found in appendix 8. It should be noted that the journal topics were written in English with Chinese translations, and students had the freedom to respond either in English or in Chinese.
Student interviews
Based on a similar reason to that for using student journals, some volunteer students also had a stimulated retrospective interview after one observed lesson. The way these interviews were conducted was similar to the stimulated retrospective interviews with the teachers. Appendix 9 describes how the researcher prepared each student for this kind of interview and presents the generic questions asked to each student.
In addition, the volunteer students were also interviewed at the end of the semester, to find out their overall impressions about this course, their perceptions of improvement and the contributing reasons, and their understanding and their feelings of the CA practices in this course (Appendix 10).

Table of contents
ABSTRACT
DEDICATION
ACKNOWLEDGEMENTS
LIST OF TABLES
LIST OF FIGURES
CHAPTER 1. INTRODUCTION
1.1 CLARIFYING THE TERMS
1.2 RESEARCH BACKGROUND
1.3 STUDY PURPOSE
1.4 SIGNIFICANCE OF THE STUDY
1.5 OVERVIEW OF THE THESIS
CHAPTER 2. LITERATURE REVIEW
2.1 THE ORIGINS OF CA
2.2 DEFINING CA
2.3 MODELS / FRAMEWORKS OF CA
2.4 RESEARCH ON CA IN L2 ASSESSMENT FIELD
2.5 FRAMEWORK FOR THE PRESENT STUDY
2.6 CHAPTER SUMMARY
CHAPTER 3. METHODOLOGY
3.1 RESEARCH QUESTIONS
3.2 RATIONALE FOR THE CASE-STUDY APPROACH
3.3 IDENTIFYING THE CASES
3.4 DATA COLLECTION
3.5 DATA ANALYSIS
3.6 SUMMARY
CHAPTER 4. ANDREW’S CA PRACTICES
4.1 ANDREW’S TEACHING CONTEXT
4.2 ANDREW’S CA PRACTICES
4.3 STUDENTS’ GENERAL PERCEPTIONS
4.4. SUMMARY
CHAPTER 5. MARY’S CA PRACTICES
5.1 MARY’S TEACHING CONTEXT
5.2 MARY’S CA PRACTICES
5.3 STUDENTS’ GENERAL PERCEPTIONS
5.4. SUMMARY
CHAPTER 6. LINDA’S CA PRACTICES
6.1 LINDA’S TEACHING CONTEXT
6.2 LINDA’S CA PRACTICES
6.3 STUDENTS’ GENERAL PERCEPTIONS
6.4. SUMMARY
CHAPTER 7. COMPARATIVE ANALYSIS AND DISCUSSION
7.1 COMMON FEATURES OF THE THREE TEACHERS’ CA PRACTICES
7.2 VARIATIONS CONCERNING THE THREE TEACHERS’ CA PRACTICES
7.3 RELATIONSHIP BETWEEN SA AND FA
7.4 SUMMARY
CHAPTER 8. CONCLUSION
8.1 MAJOR FINDINGS
8.2 IMPLICATIONS
8.3 LIMITATIONS AND SUGGESTIONS
REFERENCES
GET THE COMPLETE PROJECT