Enhancing Course Evaluations with Three Subscales

I love learning! Humanity’s history is so vast with debates about profound questions and achievements through our creativity that only makes sense once you learn about. I love being able to teach college, sharing our collective wisdom and helping students build skills and knowledge for the future. Like many professors, I came to this calling for idealistic reasons. Yet when we're judged, it’s with short term course evaluations presenting us more like service providers with course objective "products" to sell. No wonder so many of us dislike evals. There has to be a better way.

We could have pre- and post- class activities measuring students’ knowledge and skills with the subject so we can observe what it learned overed a semester. I personally do this and it’s quite helpful comparing how learning rates vary different semesters in light of pedagogical choices I make. But it took me quite a lot of effort to create the measures, I have the methodological skills to devise measures, and I don’t feel pressure to “teach to the test” I create because my measures are mainly for my own growth.

Unfortunately, we’re seemingly stuck with course evaluations because administrators feel a need to have a measure applicable across fields that could document success and flag potential issues. So I would like to make a modest proposal. We eat our students! Of course not. We eat our administrators! Seriously, we can at least mitigate teaching eval’s flaws and make course evaluations a little more valuable for faculty. In particular, we can make course evaluations more useful by creating sub-scales so we can see if we excel or struggle in particular elements of teaching.

To propose new item’s, we conducted an extensive literature review of course evaluation instruments, both K-12 and college. Synthesizing items across dozens of measures, there is remarkable consistency. Items tend to fall into three kinds: instructor caring, instructor clarity, and instructor encouraging insight.

Wording of items is usually similar. We considered items functionally identical if we could not imagine a teacher being honestly rated high with one item but low with the other. With a team of undergraduate research assistants, we reviewed items to remove redundancies (functionally equivalent items). We refined wording seeking simplicity and breath while being free of jargon.

Keeping with the general consensus of measures, we use a 5 point Likert scale with all items phrased so agreement is a positive reflection on teaching. However, 5 points were only somewhat more common than 7 points, so either choice is reasonable. We found measures intended for college typically used the term “instructor” while measures intended for K-12 used “teacher;” the terms can be exchanged depending on what is most intuitive to your students. We arrived at eight distinct items for each subscale, the Multidimensional Comprehensive Course Evaluation of Student Impressions (figure 1). When administering to students to gauge their impressions, items are randomly ordered and subscale names are removed.

In the literature, three concerns are identified repeatedly as possible issues with validity. Women tend to be rated lower than men and persons of color tend to be related lower (Lawrence, 2018). Instructors of classes with lower grades tend to be rated lower, instructors intuitively recognize this, and feel pressure toward grade inflation (Eiszler, 2002).. The proposed measure hopefully helps by highlighting it's not a measure of objective reality, but as its name implies, we're measuring students' impressions. Substantively, by using 8 items with each dimension, each institution can identify items with the least differences by your demographics and lowest correlations with grades With a smaller chosen subset, administrators can share the overall the size of gaps (e.g., Women are average score 0.2 points lower than men) allowing stakeholders to mentally tweak ratings up or down for a plausibly fairer comparison. However, using all 24 items is reasonable; since each item takes a student thoughtfully responding about 10 seconds, the entire scale takes only about 4 minutes to administer.

Having subscales of equal weight prevents administrators from over-valuing a dimension without being consciously aware of their choice. For example, when I compare my institution's course eval system with the three sub-scales, I can see our items are overwhelmingly about clarity. Maybe teachers presenting material methodically with very clear assessments is what an institution values most. Regardless of what we choose, the three subscales make our choice intentional.

To me, the biggest advantage of three subscales is the ability of faculty to observe different aspects of how they’re perceived by their students. It’s actionable. If somebody just gets low evals, what do you do? But if your evaluations are mediocre because of a low caring subscale score, you can focus your attention on how you show empathy and kindness. Similarly, a faculty member with low insight subscale score might realize they need to increase the depth of material they cover.

Why not more subscales? Maybe we should have more. In fact, when we conducted literature reviews, we specifically looked at general course evaluations and not students' impressions of many other instructor qualities, Many studies have reliable measures, such as the use of humor, self-disclosure, or off-task choices ). But these measures were created with more specific constructs in mind, so we did not synthesize them.

I wouldn’t pretend the Multidimensional Comprehensive Course Evaluation of Student Impressions solves our problems with faculty accountability and growth. It’s genuinely a modest proposal. But this small step can mean a lot. We can start undoing racial and gender biases, reduce the incentive for grade inflation, and provide faculty more ‘actionable’ feedback.

Eiszler, C. F. (2002) College students’ evaluations of teaching and grade inflation, Research in Higher Education, 43(4), 483-501.

Lawrence, J. W. (2018). Student evaluations of teaching are not valid. Academe, 104(6), 16-18.