Interpreting Feedback and Evaluations

Jim Wilkinson, Derek Bok Center

Although faculty and Teaching Fellows sometimes voice distrust of student evaluations, nearly two decades of research support the idea that they offer valuable feedback on teaching and learning. But like any data, evaluations require interpretation. Whether these are online Q evaluations in a large course or informal "minute papers" in a Freshman Seminar or section, student responses are not always easy to understand. Nor is it always evident what teaching strategies should be adopted to address the problems raised. What follows is an attempt to offer some suggestions.

At the outset we should distinguish between two kinds of feedback — (a) determining that a problem exists, and (b) diagnosing just what the problem might be. Like medical patients, students are better at identifying (a) than (b). They are generally correct when they claim that something is amiss, but often less reliable when attempting to identify the precise cause of the problem. This should come as no surprise. For one thing, few have been trained to assess teaching techniques, and thus most focus on effects, rather than on causes. For another, most students lack a precise vocabulary for talking about teaching, and tend to invoke familiar grounds for complaint, even when they do not apply.

The inability to understand non-native English speakers is one such ground. I know of a case in which students complained about the English competence of an Asian graduate student. When the complaint was investigated, it turned out that the graduate student in question had been born in Akron, Ohio. It may be that an Ohio accent was difficult for some students to follow. Far more plausible is the assumption that there was a genuine problem in the class that students were unable to identify, and that English-language competence thus became a convenient (if false) peg on which to hang their (real) dissatisfaction.

Does this mean that student complaints (or, in happier circumstances, student praise) are based on misinformation or subjective whim? While bias or inattention may affect some student evaluations, the vast majority provide a reliable starting point for interpretation — a necessary, though not a sufficient condition for understanding what is going on in a class.

Interpretive Challenges

There exist a number of general challenges in interpreting student evaluations. One is that not all students think alike. At Harvard, where the diversity of background and preparation among students is pronounced, what works for some will fail to reach others. In large classes, especially, it is often true that two or more subgroups exist among the students, each with different expectations and needs. Lecture material that seems too advanced for one will appear "just right" or too elementary for others. Thus one rule for interpreting student evaluations is to expect diversity of opinion, mirroring the diversity of the student body as a whole.

Another general challenge is the degree to which students personalize instruction. Faculty are generally attracted to their disciplines by the subject matter. Their intellectual curiosity is stimulated by specific types of problems, or dealing with specific types of data, be they medieval texts or regression analyses or animal behavior. Students are far more likely to be attracted to a field by the people who teach it; they are more likely to be dissuaded from studying a topic for the same reason.

In short, students are finely attuned to the emotional attitudes they perceive their instructors adopting towards them. They will work far harder and more readily for a demanding but caring instructor than for a demanding and impatient one. Putting students down in front of peers, using sarcasm or innuendo in written paper or exam comments, imputing laziness or lack of interest to students are all signs of instructor impatience that students take personally and interpret as hostility towards them, thus beginning a vicious cycle where impatience breeds resentment which in turn breeds behavior that fuels impatience.

A third challenge is when balancing positive and negative comments. Some instructors ignore all but the positive ones; many more do just the reverse. Since it is rare for students to be unanimous in their evaluation of a course, it is important to keep the inherent complexity of their reactions in mind, while deciding which comments to emphasize.

Finally comes the difficulty of knowing what the general teaching norms might be in your field, or College-wide. Is a 4.1 on a 5-point scale good or bad? Do student complaints about your syllabus mean you should fix it, and if so, how? Teaching Fellows and faculty tend to have high standards for their own performance, without always having a clear idea of just what standard is appropriate for their particular stage in teaching. Comments that would be negative when applied to someone in their fifth year of teaching might be complimentary to someone in their first. It is therefore often a good idea to seek the advice of a second observer — a colleague, friend, partner, or the Bok Center.

Bok Center staff have collectively reviewed thousands of evaluations over the past three decade, and tend to have a good sense of the norms by which to judge individual performance. Moreover, they also have a highly developed repertoire of possible remedies for the problems diagnosed on the basis of student feedback. If your syllabus needs fixing (and it might), they will have some ideas of how to proceed.

Profiles of Problem Classes

To paraphrase Tolstoy, happy classes may be all alike, but every problem class is problematic in its own way. Yet it remains true as well that problems experienced by instructors tend to recur. Though the pedagogical mix between instructor and a particular class of students is unique, some instructors seem to affect different classes in roughly similar ways. What follows are four common syndromes and the written comments that typically announce them.

  1. "Nice But Can't Teach"
    In this case students divide their comments between positive evaluations of the character and motivation of instructor, but give low scores for teaching skills. They characterize such teachers as accessible, friendly, caring, even knowledgeable, and at the same time confusing, disorganized, and/or passive. How can we interpret what they are saying? The problem is most often teacher inexperience — one of the easiest teaching problems to correct, since such instructors in this family are often at the start of their careers, motivated to improve, and able to benefit from standard training such as that offered at the Bok Center's Fall and Winter Teaching Conferences and microteaching sessions.
  2. "Doesn't Care"
    Students report not getting papers back on time, receiving minimal comments (my favorite is the one-word comment, "vague"), their instructors being late for office hours or failing to show up entirely, not returning emails, not posting materials on the course web site in a timely manner, and so forth. In most cases these complaints point directly to a lack of commitment on the part of the instructor. They are concrete and concern matters about which students are the most expert. Though they may not be reliable about an instructor's accent, they know exactly how many days have elapsed since they turned in their papers.
  3. "Over My Head"
    Students report that the "over my head" instructor is unaware or misinformed about the true state of student preparation for the course, seems intent on presenting material at a high level of difficulty, and has no idea about the problems they encounter with the lectures or readings. Often this syndrome results when beginning TFs and faculty feel the need to demonstrate their proficiency in their fields, and conduct their classes more like graduate seminars than undergraduate classes. Correction requires the willingness to sound students out on the true state of their preparation and to reorient the course or section so that it is neither too recondite nor too elementary.
  4. "Impatient and Hostile"
    Students report that some instructors are dismissive, abrupt, even contemptuous — the negative opposite of the disengaged "doesn't care" instructor profiled above. Comments include reports of being humiliated by the instructor in front of one's classmates, paper comments such as "haven't you learned to write yet?" and an unwillingness to tolerate elementary questions in class. Given the importance that students assign to the personal aspect of teaching, this syndrome is the most likely to trigger a spirited negative response on their part. It is also one of the most difficult to correct, since it involves trying to alter an instructor's ingrained expectations of how rewards and punishments affect students. Verbal aggression, often a staple of teacher-student exchanges in Europe, simply does not work with American students.

It is true that some classes are simply tough to teach — not because of the content, but because of the students. Most everyone who has been a section leader can remember teaching back-to-back sections with identical content and very different results. One goes swimmingly, the other sinks. But far more often it is the instructor's attitude, not the students', that sets a tone. In some cases the cause is related to instructor expectations that students should be capable of more advanced work, and incomprehension at their poor preparation. The Bok Center has often found it useful to invoke the distinction between intelligence and knowledge in counseling faculty on how to deal with disappointing classes. Especially those international faculty who are accustomed to rigorous national systems of education — which the United States signally lacks — may find it difficult to understand why 19-year-olds in the U.S. are ignorant of things they themselves had mastered at 15 or 16. But so it is.