How Native Language and Level of English Proficiency Affect the Structure of the Test of English as a Foreign Language (TOEFL).

The aim of this study was to appraise the effect of native language and level of English proficiency on the structure of the TOEFL(`). The interrelations among TOEFL items, using all of the information provided by the various responses to the items (the four alternatives, omitted, and not reached), were analyzed by three-way multidimensional scaling for samples of examinees systematically varying in native language and level of English proficiency. Four dimensions were identified: three corresponded to the sections of the test, and the fourth was an end-of-test phenomenon. The dimensions were predominantly defined by easy items and were most salient for low-scoring examinees. Native language had little influence on the results. Major conclusions were that the TOEFL's construct validity is supported, the test's interpretation varies with the examinees' English proficiency, easy and difficult items differ in their potential for diagnosis and globa! screening, and the dimensionality of the TOEFL and of competence in English depends on the examinees' English proficiency. The Test of English as a Foreign Language (TOEFL; Educational Testing Service, 1985) consists of three sections, Listening Comprehension, Structure and Written Expression, and Vocabulary and Reading Comprehension, and provides scores for each section as well as a total score. The test is intended to assess the ability of nonnative speakers to understand spoken English, to comprehend reading materials, and to recognize correct structural, grammatical, and lexical usage. Responses on the TOEFL may reflect both the influence of the examinees' native language and their level of English proficiency. Work thus far has not appraised the independent influences of native language and level of English proficiency on TOEFL performance, and these variables are confounded in most of this research. The purpose of this study was to appraise the influence of examinees' native language and level of English proficiency on the structure of the TOEFL. More specifically, the aim was to assess the interrelations among TOEFL items for groups of examinees that systematically varied in native language and level of English proficiency, going beyond the usual right versus wrong scoring to use all the information provided by the various responses to the items. Method examinees and Test Form The data were drawn from the 53,169 examinees who took the TOEFL in the May 1985 international administration and had complete information. The form had 146 operational items. Twenty-one subsamples of examinees, comprising seven language groups (Arabic, Chinese, Greek, Japanese, Korean, Malay, and Spanish) and three levels of performance on the TOEFL ( High -total scores on the TOEFL of 543 and above; Mediumscores of 483 to 540; and Lowscores of 480 and below) were randomly drawn from the total sample. All language groups with approximately 400 or more examinees at each of the three performance levels were included. (The three levels were determined by trichotomizing the score distribution for the total sample.) Each subsample consisted of 400 examinees, except for 397 in the low-scoring Greek subsample. f Jacobson, 1976) among the items was computed. This coefficient, a measure of association between two nominal variables, indicates (on a scale from 0 to 1) the extent to which one variable is predictable from the other, and vice versa. In this analysis, each item is a nominal variable with six categories (the four alternatives, omitted, and not reached), and the tau between a pair of items is computed from the resulting 6 x 6 contingency table. A three-way, metric multidimensional scaling analysis of the 21 tau matrices was carried out, using SINDSCAL (Pruzansky, 1975). Three-way scaling allows for variation among in the salience of the dimensions (the individuals in the present application are the 21 subsamples). The results of the scaling were subjected to two hierarchical cluster analyses (Ward, 1963), one on the 146 items and one on the 21 subsamples, to identify regions in the multidimensional space where items and subsamples formed groupings.