ELI Retreat, May 23, 2012
Discussion of Recommendations of the Testing Committee
Discussion leader: Mary Beth Worrilow
Recorder: John Milbury-Steen
Participating: Walt Babich, Nonie Bell, Leslie Criston, Michael Fields, Bob Palmer, Randall Pennington, Mikie Sarmiento and Erik Flanigan
In regard to recommendation 1: “that we examine data from standardized test results to draw inferences for the purpose of informing administrators and teachers:”
We all agreed that was a good idea: “it goes without saying.” We noted one obvious inference: low achievement in Saudis; also, the correlation between the COMPASS reading test and the ELI grade is not very good. One of us said that perhaps the test emphasizes top-down skills too much and is, therefore, unreliable for low level students. We discussed whether this might be true. We agreed that certainly students in Basic should not take the test. Walt noted that many faculty are untrained in test design and writing, and therefore current in-class testing may not be leading to ideal results in terms of validity and reliability. We wondered what effect the new mode of instruction in the lower levels would have on COMPASS testing outcomes. Walt predicted an improvement in scores.
In regard to recommendation 2: “that we work with level coordinators to insure that new teachers are adequately trained and mentored in assessment:”
We agreed that norm referencing helps to insure this, especially the part of the training done by the level coordinator. Mikie noted that some coordinators provide tests used by all teachers, as well as finals. We discussed whether this was an instance of micromanagement: most of us thought it was. We noted that there is a tricky balance between the need to enforce standards and the need to allow the teacher his or her autonomy. Walt thought perhaps we should have a workshop on test construction, including the details of making multiple choice tests : creating false attractors and other techniques. There was discussion about critiquing tests at level meetings. Back to the test writing workshop, Randall suggested the speaker be J.D. Brown of the University of Hawaii. Unanimous, enthusiastic approval that, instead of inviting him here, we all go there on a chartered plane.
In regard to recommendation 3: “that we explore the possibility of creating an integrated skills test as a final assessment for EAP VI students:”
This would be for graduating students ready to matriculate into UD. It would involve an academic reading passage, listening to a related lecture, and writing and speaking about them. Michael asked whether graduation depends primarily on getting a 6.5 on the final essay. Walt said that that was, of course, important and listed the other requirements. Michael thought an integrated skills test was an idea that ought to be explored. General agreement. John noted that this might be hard to organize: we’re already hard pressed during finals week. We didn’t speculate about how it might be managed.
In regard to recommendation 4: “that we work with lower level teachers to refine the placement process so that initial placements into Basic, Level I and Level II are more standardized:”
Walt gave some background: that some teachers felt that not enough students were being placed into Basic. We all agreed that it’s important to get lower level teachers represented in interviewing and essay scoring. One of us noted that some interviewers, faced with a very low level student, cut the interview short and automatically place the student in level I. Mikie floated the idea of pre-screening students for interviews: based on a short conversation, send the low level students to an interviewer who teaches a low level. This was the “specialist approach.” The other idea was to have all interviewers thoroughly normed in interviewing so they could discriminate among the three lowest levels. After all, a lower level teacher might not be present during interviewing.
In regard to recommendation 5: “that we closely examine the speaking finals, level by level, to insure standardization and to make sure that the finals reflect realization of language learning outcomes and that they are not overly content-based:”
Walt provided helpful background again: that some university professors have complained that they can’t understand our students when they speak. We all agreed that speaking finals in content courses like The Sixties or Drama should not be content-based, but skills-based. Mary Beth observed that in the business track, speaking finals have to be content-based. We granted business classes an exception. We considered the possibility (and feasibility) of double-rating speaking tests. For whom? For all students, or for CAP students ready to graduate? John again brought up the problem of the increased workload. There was no objection to double-rating speaking tests, but no enthusiasm, either. Michael said that he disagreed with the whole premise: judging students incompetent because they cannot speak well. The student may still be communicatively competent. The professor may be overgeneralizing. After all, he is not an ESL specialist. The truism was noted that comprehensibility in speaking is important, yet it was pointed out that the TOEFL does not even measure a student’s pronunciation directly.
In regard to recommendation 6: “that we work with administrators to assess students referred to ELI for language proficiency assessment by UD Admissions such as those who have taken the SAT but who scored below the required 90 on the iBT:”
Background from Walt: the students in question are “generation 1.5:” they have gone to an American high school but they did not pass the TOEFL. We would work with our own administrators to assess how much more ESL instructions they require. This seemed reasonable to the group.
P.S. We also added further recommendations for 318 as an administrative center.
- Mikie pleaded for mail to get reliably delivered to 318: physical campus mail. It seems there is no system in place.
- There was also discussion about providing facilities for Muslims to do their ablutions in preparation for prayer.
r prayer.