Ditch the Demo

Teaching faculty interview processes vary between institutions, but the centerpiece is usually a live teaching demonstration. Sometimes this is to an audience of faculty or a mix of faculty and students. Other times the candidate presents to an actual meeting of an ongoing course. Sometimes the talk is on a topic of the candidate’s choosing. Other times the choices are limited by the search committee. And when presenting to an actual class, the candidate may need to cover whatever was scheduled for that day.

But—variations aside, a live teaching demonstration is a core part of the vast majority of teaching faculty interviews, where it replaces the research-focused job talk given by research faculty candidates.

At Illinois, we restructured our teaching faculty interview process to be more effective and equitable by deemphasizing performative pedagogy. Under my leadership, beginning with our Spring 2022 interview cycle, we replaced the live teaching demonstration with an open-ended presentation on effective pedagogy. In what follows, I’ll explain why we made this change, and discuss some of the benefits I’ve noticed over the past few years.

To begin, let me briefly justify focusing on the talk component of the interview. It’s true that most interviews also include one-on-one or smaller-group conversations with faculty and students. And so some will say: We evaluate all those things that the teaching demonstration misses through our one-on-one meetings.

The importance of the job talk in teaching faculty interviewing is also a reflection of the research faculty interview process. It is certainly not uncommon for a research faculty candidate who looks brilliant on paper to not receive an offer because they gave a bad talk. That’s not saying that their talk wasn’t well-delivered and well-structured. But research faculty candidates are expected to use their talk to put their work in context and underline their future potential. A candidate who has done brilliant work, but doesn’t seem to understand what is important about it, or can’t plot an exciting course toward new discoveries, will probably not be competitive for top-tier research faculty positions. Perhaps the talk component of teaching faculty interviews doesn’t deserve such weight. But as long as teaching faculty are interviewed alongside research faculty, the talk will probably continue to be emphasized.

All to say: The talk is the single most important component of the teaching faculty interview. So its format matters! Let’s proceed.

Live teaching demonstrations have two significant weaknesses. Most importantly, they place too much emphasis on performative pedagogy—a candidate’s live teaching abilities. But it’s also worth mentioning that, even if your goal is to emphasize live classroom instruction, a live teaching demonstration is a poor way to evaluate that ability.

The topic causes more trouble. Allowing the candidate to choose may help them select material that they feel comfortable with or that aligns with their other interviews, but can also result in candidates making poor choices or accidentally presenting something in a way that varies from the “institutional style” or some other instructor’s favorite approach. It also makes it harder to compare applicants. Choosing a topic for the candidate creates the complementary set of problems.

Placing a candidate in an actual course might initially seem like a great idea. But in practice it forces candidates to present on a very limited subset of topics, and requires that they deal with whatever context or instructional approach they inherit from the primary instructor. And you can also worry about the wisdom of mandating involuntary student involvement in the interview process, particularly when a candidate might present an important idea in an incorrect or confusing way.

Let’s also acknowledge the fact that any decent candidate will over-prepare their teaching demonstration. This is predictable and, in some ways, a good thing—preparation is a big part of effective teaching, and we want colleagues who prepare well. But the problem is that a candidate generally has a lot more time to prepare a one-off interview teaching demonstration than they’ll have to prepare the many lectures they’ll need to give each term⁽¹⁾.

Even if you could address these issues, there’s a more fundamental problem at play. Specifically, studies show that even forewarned interviewers are almost incapable of not being overly biased by first-hand observation of a candidate—even if that observation contradicts better data they have available.

A great anecdote on that topic comes from another high-stakes hiring process: drafting basketball players⁽²⁾. As related by Michael Lewis in “The Undoing Project”—which I highly recommend—and I’ll quote this at length due to its relevance:

Before the draft, the Rockets would bring a player in with other players and put him through his paces on the court. How could you deny the chance to watch him play? But while it was interesting for his talent evaluators to see a player in action, it was also, Morey began to realize, risky. A great shooter might have an off day; a great rebounder might get pushed around. If you were going to let everyone watch and judge, you also had to teach them not to place too much weight on what they were seeing. (Then why were they watching in the first place?) If a guy was a 90 percent free-throw shooter in college, for instance, it really didn’t matter if he missed six free throws in a row during the private workout.

Morey leaned on his staff to pay attention to the workouts but not allow whatever they saw to replace what they knew to be true. Still, a lot of people found it very hard to ignore the evidence of their own eyes. A few found the effort almost painful, as if they were being strapped to the mast to listen to the Sirens’ song. One day, a scout came to Morey and said, “Daryl, I’ve done this long enough, I think we should stop having these workouts. Please, just stop doing them.” Morey said, Just try to keep what you are seeing in perspective. Just weight it really low. “And he says, ‘Daryl, I just can’t do it.‘”

I should note that I have served on both research and teaching faculty hiring panels for many cycles at multiple institutions, and I have never heard any discussion of this kind of cognitive error, much less how to avoid it. Interviewing is a well-studied human decision-making process, but little of that knowledge is shared with faculty hiring committees. Mea culpa: I did not discuss this with committee members when I chaired our hiring process. Next time.

Overall, once you combine this source of bias with all the other sources of noise enumerated above—the audience, the topic, preparation—it’s hard to conclude that a live teaching demonstration is accomplishing anything beyond identifying particularly unqualified applicants. If you want to evaluate live teaching ability, there are better ways. I’ll return to this point.

But the deeper problem with the centrality of teaching demonstration in teaching faculty hiring is not that they measure poorly, it’s what they don’t measure at all.

Creating and offering a successful course requires excellence at a diverse set of skills and abilities, including, but not limited to:

1. Determining what to cover.
2. Selecting or creating a textbook or other third-party tools and materials.
3. Sequencing the content.
4. Developing effective explanations for core ideas.
5. Choosing which concepts to emphasize and which to deemphasize.
6. Identifying and addressing limitations and gaps in student prior knowledge.
7. Designing effective and engaging assignments and assessments.
8. Developing grading rubrics and ensuring fair and useful marking.
9. Structuring the content and other activities to keep students on track.
10. Designing grading strategies that align student desires for good marks with genuine increases in understanding and capability.
11. Ensuring that academic integrity policies are communicated and enforced.
12. Crafting course policies that support equitable learning while limiting staff overhead.
13. Training new course staff members—a never-ending challenge given constant turnover.
14. Managing course staff, many of whom have other priorities and obligations.
15. Building and maintaining any necessary course infrastructure, or supervising other staff members as they do so.
16. Learning and effectively utilizing any educational tools and resources necessary to support the course—learning management systems, forums, grading platforms, and so on.
17. Coordinating with other instructors, both those teaching the same course, those teaching adjacent courses, and others in the same program.
18. Analyzing data to identify flaws with the course and opportunities for improvement.
19. Completing incremental improvements to the course each semester, while also planning and executing on more transformative changes that may span multiple courses, in cooperation with other stakeholders—including departmental leadership.
20. Fostering an equitable and inclusive course community among staff and students.
21. Staying current with the latest changes to both the content and pedagogical approaches.

Phew! Did I miss anything⁽³⁾? Ah, yes, sorry:

22. Front-of-the-room teaching and classroom management.

The only thing evaluated by a live teaching demonstration.

Accomplishing this is not that difficult. I’ll describe our implementation at Illinois, but I’m sure that there are variants of our process that achieve similar goals.

Starting in the Spring 2022 interview cycle we modified our interview process for teaching faculty candidates. Specifically, we removed the live teaching demonstration. This was not a particularly hard decision. There had been lingering dissatisfaction with this component of the interview, and nobody stood up to defend it. I’ll admit that we had not necessarily been utilizing the best implementation of the live teaching demonstration, and perhaps that aided our transition.

Two new components of the interview process replace the live teaching demonstration. First, we request that candidates prepare a prerecorded demonstration of technical content. This is a 10–15 minute video explaining a core computer science concept which they can upload to their video sharing platform of choice. In this way, we preserve our ability to gain some insight into how the candidate explains technical concepts.

Even the pre-recorded demonstration alone has already been an improvement on the previous live demo. Because it’s asynchronous, it’s easier for committee and other faculty members to watch on their own time, which leads to increased viewership. And because it’s recorded, it’s also easier to pore over when needed. I know I’ve returned to a candidate’s prerecorded technical demonstration for a second or third viewing, particularly when resolving disagreements about the effectiveness of their explanation. Technically this was possible previously because teaching demonstrations were recorded, but I find myself more likely to review this content now, probably because it’s shorter. We also get it before the candidate’s synchronous interview, and the ability to view it before talking to them is another advantage.

At this point an astute reader may be thinking: Wait, you just told me that this kind of one-off over-prepared demonstration was invalid! And yes, that’s true! There’s a better way to do this, which I’ll come back to at the end. We’re just not there yet.

During the interview day itself, in place of the live teaching demonstration we have candidates give a presentation on effective pedagogy. Here’s a link to the official description, which also includes information about the prerecorded explanation of technical content discussed above.

As you can see, the talk is intentionally open-ended, giving candidates the opportunity to provide a high-level overview of a variety of ways that they support effective instruction in their courses. Everything I listed in the laundry list above can be incorporated, and at this point we’ve candidates discuss pretty much every one of those topics—although obviously not all at once.

We’re only a few years into this experiment, but I’ve already noted several things I consider successful about this format.

I also think that the talk sets up later conversations with candidates much more effectively. Those conversations were never going to be a continuation of the teaching demonstration⁽⁴⁾. But now that candidates are providing us with a lot more context and a higher-level overview of their thinking, that provides a better starting point for a more in-depth one-on-one or group conversation later.

I’ve also found the opportunity to explore higher-level aspects of a candidate’s educational philosophy very useful. As an example, there are instructors who seem to consider it axiomatic that only a certain percent of a class will succeed in a given semester. Considering that instructor versus one whose goal is for every student to succeed, that’s an easier choice for me, and definitely outweighs weaknesses in their prerecorded demonstration of technical content. To be sure, it’s always critical that a candidate’s demonstration be clear and accurate. But an instructor driven to achieve high levels of student success will find and fix their mistakes, whereas one inured to student failure may not. What we believe is possible affects what we strive to achieve.

Finally, one unintentional but significant benefit of this talk format is that it helps research faculty grasp the challenges and complexities of effective instruction.

If anything, the current demonstration-based talk format for teaching faculty is probably a reflection of the limited way that many research faculty view teaching: I just show up in class and read the slides to the students. And if I’m more animated and crack a few jokes, maybe my student evaluations go up a bit. Given the limited number of instructional faculty at many institutions, it’s likely that current teaching faculty hiring processes were in fact designed and written by research faculty⁽⁵⁾.

Effective instructors know that there is so much more to the job than what happens behind the podium. But if we want to move past second-class citizenship in our departments, we need to make sure that our research faculty colleagues know that as well. From that perspective, centering instructional hiring around a live teaching demonstration sends exactly the wrong message—that the performance is all or most of what there is to it.

While I’m generally wary of comparisons with research faculty, it’s worth pausing to note the significant difference between the talks at the heart of the research faculty and teaching faculty interview process. Specifically: The research faculty job talk is not a live demonstration of the candidate doing research.

That idea might have provoked a laugh. But why is it silly? It’s silly because there’s general acknowledgement that effective research has many different components—from developing new ideas, to refining them through conversation and collaboration, to evaluating them rigorously and communicating them to a broader audience. The research faculty job talk also provides candidates with the opportunity to present prior work, discuss how their thinking about various problems has evolved, and present at least a tentative plan for the future. Teaching faculty candidates deserve the same opportunity to present their thinking and accomplishments, along with everything that goes in to effective teaching.

Overall I consider our current interview format to be a successful change. But, of course, there are still areas for improvement. In a future post, I’ll discuss the most significant weakness in our current hiring process, and one that I suspect affects many other departments as well—a lack of evidence.

As a brief coda leading in the same direction as the follow-on post, what’s a better way to evaluate live teaching performance? Why not review lecture captures from the candidate’s previous courses? Even just a single instance immediately addresses many of the weaknesses of the live teaching demonstration enumerated above. Actual students. Content in context. Realistic levels of preparation. If needed, concerns about accidentally choosing a poor example can be alleviated by asking the candidate to select a few examples for the committee to choose from.

This sounds like a great idea. But today, too often candidates are unprepared to provide these kinds of materials. That’s the lack of evidence, a significant challenge to the teaching faculty hiring process that I’ll return to.