Measuring Japanese EFL Student Perceptions of Internet-Based Tests with the Technology Acceptance Model

August 2016 – Volume 20, Number 2

Gilbert Dizon
Himeji Dokkyo University, Japan.
<gilbert.dizon.jrgmail.com>

Abstract

The Internet has made it possible for teachers to administer online assessments with affordability and ease. However, little is known about Japanese English as a Foreign Language (EFL) students’ attitudes of internet-based tests (IBTs). Therefore, this study aimed to measure the perceptions of IBTs among Japanese English language learners with the technology acceptance model (TAM), the most influential and widely employed model to predict a user’s adoption of new technologies (Lee, Kozar, Larsen, 2003). However, TAM has yet to be utilized in the context of English education in Japan. Thus, the study also sought to determine the correlational relationships between three TAM variables – perceived usefulness (PU), perceived ease of use (PEOU), and the behavioral intention (BI) to use IBTs. An adapted TAM questionnaire was administered to 80 Japanese university students. The results showed that the relationships between PU, PEOU, and BI were positively and highly correlated. More importantly, students’ attitudes towards the PU, PEOU, and BI to use IBTs were found to be moderately favorable, with PU and BI more favorably perceived than PEOU. The findings verified that the four proposed hypotheses were supported, providing evidence that IBTs are viewed positively by Japanese EFL students.

Keywords: technology acceptance model, internet-based test, online assessment, L2 assessment

Introduction

As Roever (2001) noted, there has been considerable interest in internet-based tests (IBTs) in the language testing community due to their advantages over traditional paper-based tests (PBTs). Not only do IBTs provide immediate feedback, improved scoring accuracy, and a richer interface (Noyes & Garland, 2008), but they also offer more flexibility and convenience (Roever, 2001). However, as this interest grows and IBTs become more widespread in second language classrooms, so does the need to examine students’ attitudes towards them as helpful and easy to use tools for assessment. This study utilizes the technology acceptance model (TAM), the most widely used theoretical model to explain a user’s technological acceptance. However, TAM has yet to be employed in Japan to study English as a Foreign Language (EFL) students’ views of a given technology. Therefore, the purpose of this study is twofold: first, to examine the correlational relationships between three TAM variables, perceived usefulness (PU), perceived ease of use (PEOU) and behavioral intention (BI) to use IBTs; and second, to measure Japanese EFL student perceptions of IBTs with a TAM questionnaire.

Literature Review

Computer-based and internet-based tests

The increase in the usage of technology in language assessment has provided insight into L2 students’ attitudes towards computer-based tests (CBTs) and IBTs. Much of this research has focused on standardized exams, namely, the TOEFL test. One such study is Striker and Wilder’s (2001) investigation of examinees’ opinions of the TOEFL CBT. A total of 689 test takers in three cities, Buenos Aires, Cairo, and Frankfurt, were surveyed in the spring and summer of 1999. The results of the study indicated that the students had moderately favorable attitudes about the exam. These findings confirm the results from previous studies on other CBTs (Powers & O’Neill, 1993; Schmitt, Gilliland, Landis, & Devine, 1993; Schmidt, Urry, & Gugel, 1978). In all of these studies including Striker and Wilder’s (2001), favorable attitudes were found once the test takers had completed testing. Additionally, in Powers and O’Neill’s (1993) study, students were also asked about their views of CBTs prior to testing and were found to have more favorable attitudes after completion compared to their pre-test opinions. As Striker and Wilder (2001) pointed out, this suggests that their test taking experiences on the computer may have been better than they had anticipated.

One study which has measured test takers’ opinions of IBTs is Nobandegani’s (2012) investigation of the internet-based TOEFL exam. One hundred examinees of the TOEFL iBT in Iran were asked to specify their level of agreement to statements based on research of the advantages and disadvantages of CBTs. According to her findings, the disadvantages slightly outweighed the advantages; specifically, computer familiarity, computer anxiety, and difficulty with computer equipment were all perceived as limitations of IBTs. On the other hand, the participants indicated that IBTs offered several benefits such as administrative flexibility, accuracy and immediacy of scoring, as well as greater standardization. While IBTs may have significant advantages, these findings illustrate that their disadvantages need to be carefully addressed before implementing them into a teaching context.

In a much larger study of the TOEFL iBT, Striker and Attali (2010) measured the attitudes of 762 examinees in China, Columbia, Egypt, and Germany. They found that test takers in three of them had moderately favorable attitudes towards the test, with participants in Germany viewing it neutrally. This is in contrast with Striker and Wilder’s (2001) prior study on students’ attitudes towards the computer-based TOEFL exam in which all three areas held positive views of the test. Striker and Attali (2010) also looked at the test takers’ opinions of the different sections of the exam. Although the participants in all four of the countries had favorable views of the listening, reading, and writing sections, the speaking portion was perceived to be less favorable or unfavorable.

At the classroom level, there have been relatively few studies examining L2 students’ opinions of IBTs or online assessment. One exception is Hirshel’s (2012) investigation of Japanese university students’ views on online forums, glossaries, and quizzes with regard to potential opportunities for learning and interaction, awareness of lexical and grammatical patterns, as well as comprehensible output. A total of 23 EFL students participated in the study. An online survey including seven open-ended questions was distributed, and students were also interviewed by the researcher to obtain more detailed responses. Based on his findings, the perception of online quizzes was generally positive, with one student remarking “I think that [the vocabulary quizzes are] very useful for me. Because there are various kinds of questions or divide some parts. It make me master how to use the certain vocab” (p. 105).

Research framework: Technology acceptance model

A wide-range of theoretical models have been introduced to measure a user’s degree of technological acceptance. However, TAM has emerged as “the most influential and commonly employed theory for describing an individual’s acceptance of information systems” (Lee, Kozar, & Larsen, 2003, p. 752). Illustrating the impact that it has had on the information systems (IS) field, Lee et al. (2003) found that by 2003, the first two articles concerning TAM by Davis (1989) and Davis, Bagozzi, and Warshaw (1989) had received 698 journal citations in the Social Science Citation Index. Because of its prominence in IS and its established validity and reliability, TAM was chosen as the framework for the present study.

TAM is an adaption of the Theory of Reasoned Action (TRA) by Fishbein and Ajzen (1975). While TRA attempts to explain the reasons for behavior in a variety of domains, TAM aims to understand an individual’s behavioral intention (BI) towards the use a technology. The final version of TAM developed by Venkatesh and Davis (1996) consists of four variables: perceived usefulness (PU), perceived ease of use (PEOU), external variables, and BI (Figure 1). Davis (1989) refers to PU as “the degree to which a person believes that using a particular system would enhance his or her job performance” (p. 320). According to Davis (1989), PEOU is “the degree to which a person believes that using a particular system would be free of effort” (p. 320). A user’s BI is jointly determined by his or her PU and PEOU of the technology. External variables include factors such as user training, user involvement in design, system characteristics, and the nature of the implementation process (Venkatesh & Davis, 1996).

Figure 1. Technology acceptance model

Of the two variables that directly affect BI, PU has been found to be more influential. Lee et al. (2003) investigated the relationship between the three main variables in TAM (PU, PEOU, and BI) in 101 studies and found that 74 of them revealed a significant relationship between PU and BI. In contrast, the researchers found that only 58 studies showed a significant relationship between PEOU and BI. Nevertheless, PEOU was found to be a significant antecedent of PU with 69 of the studies showing a significant relationship between PU and PEOU. As Lee et al. (2003) stated, this suggests that PEOU can indirectly impact the acceptance of technology through PU.

While numerous studies have investigated learners’ willingness to adopt e-learning systems using TAM (e.g., Hsu, Wang, & Chiu, 2009; Lee, Cheung, & Chen, 2005; Ong, Lai, & Wang, 2004; Park, 2009; Saadé & Bahli, 2005), fewer studies have employed the model in the context of second language learning. One recent study is Soleimani, Ismail, and Mustaffa’s (2014) examination of mobile assisted language learning (MALL) among postgraduate ESL students in Malaysia. The twenty-five participants were asked nine questions concerning their level of agreement according to 5-point Likert scale and the results were analyzed using descriptive statistical analysis. Soleimani et al. (2014) found that the mean values of PEOU (m=3.68), PU (m=3.40), and BI (3.13) were high, indicating a positive perception towards MALL by the students.

Another TAM study conducted in a L2 context is Mah and Er’s (2009) investigation of the correlational relationships between PEOU, PU, and BI with Pearson’s correlation coefficient. The study consisted of a target sample of 58 participants from a total population of 918 Malaysian ESL students. Prior to the study, a pilot test involving 37 students was performed which verified the reliability of the 15-item questionnaire with Cronbach’s alpha. All three of the variables analyzed, PEOU, PU, and BI, had Cronbach’s alpha values of over 0.9, indicating a very high level of internal consistency. Mah and Er (2009) found that there were strong links between the three relationships examined (PEOU and PU; PU and BI; PEOU and BI) to write web logs in the ESL classroom, thereby providing validity to TAM questionnaires as research tools in second language educational contexts.

Research hypotheses

Based on the research objectives and the aforementioned literature, the following four hypotheses were proposed:

H1: There is a positive and significant relationship between PU and BI to use IBTs among Japanese EFL students.
H2: There is a positive and significant relationship between PEOU and BI to use IBTs among Japanese EFL students.
H3: There is a positive and significant relationship between PEOU and PU of IBTs among Japanese EFL students.
H4: There is a positive perception among Japanese EFL students concerning the usefulness, ease of use, and BI to use IBTs.

Methodology

Research design

Data were collected with a questionnaire containing items related to the PU, PEOU, and BI to use IBTs. The correlational relationships of the TAM variables were measured with Pearson’s correlation coefficient using SPSS. Per Boone and Boone’s (2012) recommendation concerning Likert scale data, a descriptive statistical analysis detailing the mean and SD of each construct was carried out to measure the level of acceptance of IBTs among the students.

Participants

The participants in the study included 80 male and female first-, second-, third-, and fourth-year students who were enrolled in one or more English language classes taught by the researcher during the 2014 spring semester at Himeji Dokkyo University, representing a non-random sample of the target population. The students’ L2 English proficiency levels ranged from beginner to intermediate. More than half of the students (n=45) were part of the Faculty of Foreign Studies. The remaining 35 students belonged to the Faculty of Pharmacology and were required to take an English for Specific Purposes (ESP) class as a departmental prerequisite. Only students who took both the midterm and final exams were included in the study.

Online Assessments

All students in the present study had summative assessments (i.e., midterm and final exams) administered through the Internet via classmarker.com (Figure 2). The testing website was chosen because of the key features it offered, specifically, multiple interface languages including Japanese, access to exams via password-protected links, as well as the ability to randomize test items and answers. Exams were created by the researcher and were comprised multiple choice, cloze, grammar, and essay questions based on the content of the respective classes. Although the medium of the exam items and answers was English, the medium of the website was Japanese in order to enhance ease of use and avoid any misunderstandings regarding the navigation and use of the website. Summative assessments were monitored and conducted in class due to the stakes involved and to reduce the likelihood of cheating.

Figure 2. ClassMarker homepage

Learner training

As Hackbarth, Grover, and Yi (2003) noted, “Users generally perceive a system easier to use as they gain more knowledge and confidence through direct experience in using the system” (p. 221). Therefore, students were given several opportunities to familiarize themselves with the website used for testing. A week prior to taking the midterm exam, a brief orientation was conducted in each class to explain how to access and navigate the exams as well as to detail some of the perceived student benefits of IBTs such as increased scoring accuracy and immediate feedback (Noyes & Garland, 2008). Following the orientation, the students took a non-graded practice quiz which allowed them to become comfortable with the test interface. Finally, formative assessments were administered to further increase familiarity and help them review for the midterm exams. Formative assessments were also administered a week before the final exams for these same reasons.

Instrument

A 10-item self-report questionnaire measuring PU, PEAU, and BI was used in the study (Appendix A). All items were adapted from existing TAM literature (Appendix B), thereby ensuring validity. While there may be some concerns regarding validity when using recycled items, Hyman, Lamb, and Bulmer (2006) asserted that the degree of validity is likely to be high if the questions have been pre-tested to confirm that they are accurate measures of the concepts of interest. The reliability of the questionnaire was verified with Cronbach’s alpha (α) using SPSS. Because the values of PU (α= .849), PEOU (α= .863), and BI (α= .729) are higher than 0.7, the internal consistency of the three subscales is considered acceptable. Table 1 shows the Cronbach’s alpha of each construct.

Table 1
Cronbach’s alpha of PU, PEOU, and BI Constructs

Construct	Number of items	Cronbach’s alpha (α)
Perceived Usefulness (PU)	4	0.849
Perceived Ease of Use (PEOU)	4	0.863
Behavioral Intention	2	0.729

Respondents were asked to indicate their level of agreement or disagreement to the items according to a seven-point Likert scale ranging from strongly disagree (1) to strongly agree (7). A seven-point Likert scale was utilized due to research showing that construct validity tends to be higher for Japanese when there are seven choices rather than four (Lee, Jones, Mineyama, & Zhang, 2002). English was designated as the language of the questionnaire because it was the medium of the classes taught by the researcher. The questionnaire was distributed via individualized class surveys through surveymonkey.com, and was taken after the students had completed their final exams online. Each class survey was time sensitive and closed at the end of every class. In addition, IP addresses were tracked so that the questionnaire could only be taken one time per device. The survey was taken once by each student, even if a participant was enrolled in multiple classes taught by the researcher. These students were given discreet instructions to not take the questionnaire a second time. Moreover, their in-class computer activities were carefully observed through student monitoring software on the teacher PC to ensure that they did not take the survey again.

Results and discussion

Table 2
Pearson’s Correlation Matrix for PU, PEOU, and BI

	PU	PEOU	BI
PU	1
PEOU	.791^**	1
BI	.947^**	.946^**	1
Note. ** p<.01, one-tailed.

Table 2 above shows the Pearson’s correlation coefficients (r) between PU, PEOU, and BI. According to Taylor (1990), a Pearson’s correlation coefficient between 0.68 and 1.0 is considered high with r-values ≥ 0.90 indicating a very high correlation. There was very high positive correlation between PU and BI (r= .947), signifying a strong positive relationship between the variables. Thus, the first hypothesis has failed to be rejected – There is a positive and significant relationship between PU and BI to use IBTs among Japanese EFL students.

According to the results, there is also a strong positive relationship between PEOU and BI (r= .946). This demonstrates that the second hypothesis has failed to be rejected – There is a positive and significant relationship between PEOU and BI to use IBTs among Japanese EFL students. These findings coincide with Lee et al.’s (2003) meta-analysis that PU is a stronger predictor of BI than PEOU.

While not as strong as the other two relationships studied, there is a positive relationship between PEOU and PU (r= .791). Therefore, the third hypothesis has failed to be rejected – There is a positive and significant relationship between PEOU and PU of IBTs among Japanese EFL students. This reinforces the statement by Lee et al. (2003) that PEOU can indirectly affect technological acceptance through PU.

These results demonstrate the importance of PU and PEOU in determining the behavioral acceptance of a technology. Given this, instructors must be able to effectively convey the benefits of IBTs or any other computer-assisted language learning (CALL) task in order for it to be considered meaningful and worthwhile. More importantly, the technology must also have critically useful functionality for it to be accepted by users (Davis, 1989). Furthermore, teachers must give learners sufficient training to ensure they have a clear understanding of all the features and functions of an online test interface. If not, this lack of computer familiarity can have a negative effect on test performance, particularly in the context of Japanese students (Kirsch, Jamieson, Taylor, & Eignor, 1998), as well as indirectly influence the PU of IBTs through PEOU (Lee et al., 2003).

Table 3
Mean and SD of PU, PEOU, and BI

Construct	Mean	SD
PU	5.4	1.6
PEOU	5.1	1.5
BI	5.4	1.4

As shown in Table 3, the mean of PU is 5.4. This number falls roughly between somewhat agree (5) and agree (6), indicating that the Japanese EFL learners in this study had moderately favorable views of the usefulness of IBTs. According to Noyes and Garland (2008), IBTs offer many benefits over PBTs, such as immediacy of feedback, improved scoring accuracy, and greater standardization. These advantages along with the administrative flexibility that they provide (Roever, 2001) demonstrate that IBTs can be useful not only for students but for teachers as well.

In relation to PEOU, the mean is 5.1; thus, the students’ views of the PEOU of IBTs were also moderately positive, although less than that of PU. It is important to note, however, that the mean of BI to use IBT still remained moderately high (5.4). This supports the notion by Davis (1993) that users are willing to tolerate some difficulties related to ease of use as long as the technology provides critical functionality that helps them complete a task.

Finally, in terms of BI, the mean is 5.4. This shows that the students had moderately favorable views regarding their intention to use IBTs. English education in Japan largely revolves around the usage of PBTs, most notably in the context of university entrance exams (Hirasawa, 2010). However, the students still had favorable perceptions to use online testing in the future, which suggests that exposure to IBTs can lead to acceptance despite a lack of prior experience with the testing format. In fact, in their meta-analysis of literature concerning paper- vs. computer-based tasks, Noyes and Garland (2008) concluded that “user preference certainly seems to support the use of online assessment” (p. 1370). Therefore, more research needs to be done to assess whether or not Japanese students prefer IBTs to PBTs.

The results illustrate that Japanese students have generally favorable views of the PU, PEOU, and BI to use online assessments in the EFL classroom. As a result, the fourth hypothesis – There is a positive perception among Japanese EFL students concerning the usefulness, ease of use, and BI to use IBTs – has failed to be rejected. These findings lend support to previous studies which have shown that students have generally favorable attitudes towards CBTs and IBTs (Powers & O’Neill, 1993; Schmidt, Urry, & Gugel, 1978; Schmitt, Gilliland, Landis, & Devine, 1993; Striker & Attali, 2010; Striker & Wilder, 2001).

There are some issues that need to be addressed regarding online testing and PEOU. As previously mentioned, computer familiarity can have an impact on ease of use and test performance. In addition, Brown (1992) suggested that computer anxiety may be another potential disadvantage, an assertion that was supported by Nobandegani (2012). This is important because computer anxiety has been shown to fully mediate the effect that user experience has on ease of use (Hackbarth, Grover, & Yi, 2003). In other words, when users gain more experience with a particular technology, their computer anxiety decreases which in turn also enhances PEOU. Therefore, L2 learners should be given sufficient time to become accustomed to the unique interface of IBTs, especially if they have little to no experience with the testing format.

Moreover, although students may view IBTs positively, there are many other factors that must be considered in order to successfully implement them into the classroom. Potential challenges such as a lack of computer familiarity, computer anxiety, and cheating may inhibit their usefulness. The reliability of computers as well as confidentiality concerns have also been raised as possible disadvantages (Noyes & Garland, 2008). In other words, the delivery medium of IBTs does not necessarily make them more effective than PBTs at assessing students’ L2 competence or subject-specific performance (Roever, 2001). Thus, it is critical to understand whether or not online assessments are appropriate for a particular teaching context and to address all potential issues before their utilization.

Conclusion

Based on the results of this study, the first three proposed hypotheses concerning the correlational relationships of PU, PEOU, and BI to use IBTs have been validated. Each of the relationships studied had correlation coefficients of over 0.68, thereby confirming that the constructs were positively and highly correlated at a significant level. Therefore, the study has successfully used TAM to examine Japanese EFL students’ attitudes towards IBTs, the first study of its kind in the context of English education in Japan.

In regards to student attitudes, the perceptions of all three constructs have been found to be moderately favorable, with the PU and BI to use IBTs being viewed more positively than PEOU. As a result, the fourth hypothesis has also been validated. Given this, the study advocates the usage of IBTs in EFL classrooms due to their perceived usefulness and ease of use by students, two important variables in determining the behavioral intent and acceptance of technology. Furthermore, they offer several distinct advantages that make them useful for students as well as teachers.

Nevertheless, there are several limitations regarding the usage of TAM in the present study. First, the researcher did not attempt to measure the role, if any, that gender has on the technological acceptance of IBTs. Furthermore, other external variables such as computer familiarity, computer anxiety, and learner training should also be studied to find out how these factors impact the PU, PEOU, and the BI to use IBTs. It is also important to reiterate that the findings of this study were taken from a non-random sample to obtain a snapshot of the target population. Therefore, additional research is needed to assess the validity of the results, preferably among EFL students at multiple institutions throughout Japan.

About the Author

Gilbert Dizon is a lecturer at Himeji Dokkyo University, Japan. He holds a Master of Arts in Applied Linguistics from the University of Massachusetts Boston. His current research interests include computer-assisted language learning, language assessment, and collaborative learning.

References

Boone, H. N. & Boone, D. A. (2012). Analyzing Likert data. Journal of Extension, 50(2). Retrieved from joe.org/joe/2012april/tt2.php

Brown, J. D. (1992). Using computers in language testing. Cross Currents, 19, 92-99.

Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319-340. Retrieved from jstor.org/stable/249008

Davis, F. D. (1993). User acceptance of information technology: System characteristics, user perceptions and behavioral impacts. International Journal of Man-Machine Studies, 38, 475-487. Retrieved from deepblue.lib.umich.edu/bitstream/handle/2027.42/30954/0000626.pdf

Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989). User acceptance of computer technology: A comparison of two theoretical models. Management Science, 35(8), 982-1003. Retrieved from home.business.utah.edu/actme/7410/DavisBagozzi.pdf

Fishbein, M., & Ajzen, I. (1975). Belief, attitude, intention and behavior: An introduction to theory and research. Reading, MA: Addison-Wesley.

Hackbarth, G., Grover, V., & Yi, T. Y. (2003). Computer playfulness and anxiety: Positive and negative mediators of the system experience effect on perceived ease of use. Information & Management, 40(3), 221-232. doi:10.1016/S0378-7206(02)00006-X

Hirasawa, H. (2010). Understanding English education in Japan. CATESOL 2010 Proceedings. Retrieved from catesol.org/11hirasawa.pdf

Hirshel, R. (2012). Moodle: Students’ perspectives on forums, glossaries, and quizzes. The JALT CALL Journal, 8, 95-112. Retrieved from journal.jaltcall.org/articles/8_2_Hirschel.pdf

Hsu, M. K., Wang, S. W., & Chiu, K. K. (2009). Computer attitude, statistics anxiety and self-efficacy on statistical software adoption behavior: An empirical study of online MBS learners. Computers in Human Behavior, 25(2), 412-420.

Hyman, L., Lamb, J., & Bulmer, M. (2002). The use of pre-existing survey questions: Implications for data quality. Proceedings of Q2006 European Conference on Quality in Survey Statistics. Cardiff, UK. South Wales, UK: Office for National Statistics. Retrieved from epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/Pre-existing%20surveys

Kirsch, I., Jamieson, J., Taylor, C., & Eignor, D. (1998). Computer familiarity among TOEFL examinees. (TOEFL Research Report No. 59). Princeton, NJ: Educational Testing Service. Retrieved from ets.org/Media/Research/pdf/RR-98-06.pdf

Lee, J. W., Jones, P. S., Mineyama, Y., & Zhang, X. E. (2002). Cultural differences in responses to a Likert scale. Research in Nursing & Health, 25, 295-306.

Lee, M. K. O., Cheung, C. M. K., & Chen, Z. (2005). Acceptance of internet-based learning medium: The role of extrinsic and intrinsic motivation. Information & Management, 42(8), 1095-1104.

Lee, Y., Kozar, K. A., & Larsen, K. R.T. (2003). The technology acceptance model: Past, present, and future. Communications of the Association for Information Systems, 12, 752-780. Retrieved from aisel.aisnet.org/cais/vol12/iss1/50

Mah, B. Y., & Er, A. N. (2009). Measuring students’ perception of writing web logs in ESL classroom employing the Technology Acceptance Model. Conference on Scientific & Social Research 2008 & 2009 (CSSR 08’09) (p. 3304707). Melaka, Malaysia: Research Management Institute (RMI) Universiti Teknologi MARA (UiTM). Retrieved from researchgate.net/publication/234113541_Measuring_students_perception_of_writing_web_logs_in_ESL_classroom_employing_the_Technology_Acceptance_Model

Nobandegani, P.A. (2012). Students’ perceptions of computerized TOEFL test. Language Testing in Asia, 2(2), 73-101. Retrieved from languagetestingasia.com/content/pdf/2229-0443-2-2-73.pdfOng

Noyes, J. M., & Garland, K. J. (2008). Computer- vs. paper-based tasks: Are they equivalent? Ergonomics, 51(9), 1352-1375.

Ong, C. S., Lai, J. Y., & Wang, Y. S. (2004). Factors affecting engineers’ acceptance of asynchronous e-learning systems in high- tech companies. Information & Management, 41(6), 795–804.

Park, S. Y. (2009). An analysis of the technology acceptance model in understanding university students’ behavioral intention to use e-learning. Educational Technology & Society, 12(3), 150-162.

Powers, D.E., & O’Neill, K. (1993). Inexperienced and anxious computer users: Coping with a computer-administered test of academic skills. Educational Assessment, 1, 153-173.

Roever, C. (2001). Web-based language testing. Language Learning & Technology, 5(2), 84-94. Retrieved from llt.msu.edu/vol5num2/roever/default.html

Saadé, R.G., & Bahli, B. (2005). The impact of cognitive absorption on perceived usefulness and perceived ease of use in on-line learning: An extension of the technology acceptance model. Information & Management, 42(2), 317-327.

Schmidt, F.L., Urry, V. W., & Gugel, J. F. (1978). Computer assisted tailored testing: Examinee reactions and evaluations. Educational and Psychological Measurement, 38, 265-273.

Schmitt, N., Gilliland, S. W., Landis, R. S., & Devine, D. (1993). Computer-based testing applied to selection of secretarial applicants. Personnel Psychology, 46, 149-165.

Soleimani, E., Ismail, K., & Mustaffa, R. (2014). The acceptance of mobile assistance language learning (MALL) among post graduate ESL students in UKM. Procedia – Social and Behavioral Sciences, 118, 457-462. Retrieved from sciencedirect.com/science/article/pii/S1877042814015924

Striker, L. J., & Attali, Y. (2010). Test takers’ attitudes about the TOEFL iBT. (TOEFL iBT No. 13). Princeton, NJ: Educational Testing Service. Retrieved from ets.org/Media/Research/pdf/RR-10-02.pdf

Striker, L. J., & Wilder, G. Z. (2001). Examinees’ attitudes about the TOEFL-CBT, possible determinants, and relationships with test performance. (Research Report 01-01). Princeton, NJ: Educational Testing Service. Retrieved from ets.org/Media/Research/pdf/RR-01-01-Stricker.pdf

Taylor, R. (1990). Interpretation of the correlation coefficient: A basic review. JDMS, 1, 35-39. Retrieved from sagepub.com/salkind2study/articles/05Article01.pdf

Venkatesh, V., & Davis, F. D. (1996). A model of the antecedents of perceived ease of use development and test. Decision Sciences, 27(3), 451-481.

Appendix A

Internet-based Testing Questionnaire

Please read each statement and choose the answer that best matches your opinion. Thank you for completing the questionnaire.

1. I was able to answer questions more quickly on the Internet.