Quantifying CS1

Recently Mark Guzdial published a piece describing how he evaluates teaching portfolios. It’s a good article, and you should certainly read it. I found myself nodding along with the following points:

• He doesn’t think much of student evaluations of teaching
• He points out that overreliance on assistance from the instructor or course staff can indicate problems with the course design and materials
• He looks for evidence of instructors creating a positive and inclusive climate
• He supports building and evaluating new pedagogical tools

But by the end I was struck less by what was included in his criteria than by what was excluded: Any consideration of actual results and data. While his post does focus on process and best practices, merely mimicking things that might have been shown to work elsewhere without validating them in your own environment dangerously skirts the definition of cargo cult science.

So in this essay I’m going to list some of what I consider important quantitative measures for evaluating CS1 instruction. Some of these are specific to CS1, others are more general. These are all ways that I would want my own course to be evaluated. Uncoincidentally, they are all metrics I’ve used to assess my own attempts to improve my class. I’ll share some of those results as we go.

To start, let me send off a few common objections to quantifying CS1 and data-driven analysis.

First, none of these measures should be taken out of context. A common response by people who aren’t collecting data seems to be a fear that, if the data existed, other people wouldn’t understand it. Well, to start, until the data exists, nobody can understand it, or act on it. And second, usually it’s the instructor that is going to collect, interpret, and present it—allowing them to place it in proper context. I would certainly expect this to be true on a promotion portfolio. Overall if you are really sure that any attempts at data analysis will be used against you in bad faith, you probably have other problems to focus on—like finding a new job.

Most of the metrics I list below can and should be based on data collected and analyzed by individual instructors. A few do require some level of departmental or multi-instructor coordination. But lacking those shouldn’t prevent us from analyzing the data we do have. This also has the effect of putting the instructor in the driver’s seat, which is likely to make them more comfortable and help assure that the data is not taken out of context, as it might be if analyzed by a third party.

Without further ado, what measurements should we use to quantify CS1? Here are my suggestions:

• What and who is the course trying to teach? This is not always quantifiable, but is measurable and comparable to a degree, and important to put other numbers in context.
• How many students are meeting the learning goals?
• What performance gaps exist between cohorts?
• Do students succeed in downstream courses?
• How much programming practice are students receiving?

Let’s go through each in turn.

What and who is the course trying to teach?

What and who is the course trying to teach?

Of the metrics I’ve listed, this is the one that is the hardest to quantify. However, it’s also probably one of the most important, and should be considered before examining other data points.

To be fair, there is a range of potential speeds at which CS1 can be taught, even if some seem clearly too fast or too slow. Choosing the right one requires considering a lot of different factors, including the strength of the student population and expectations of downstream courses. It also frequently requires making tradeoffs between multiple objectives. For example, for courses that teach both CS majors and non-majors, the majors may be able to move much faster at the risk of increasing frustration among non-majors. And the course may need to prepare majors for later coursework, unlike non-majors who may not usually proceed past CS1.

Simply making CS1 easier and easier seems too often to be the approach to these tradeoffs. But it’s not a solution. It simply shifts the problem downstream, as continuing students arrive unprepared to succeed in later courses.

These tensions and the decision-making surrounding them deserve separate treatment, and I’ll return to this topic. But for now, consider an understanding of both the audience and ambitiousness of CS1 to be critical for contextualizing the metrics that follow.

For my own CS1 course, this calibration has been one of the bigger challenges. The problem here is exacerbated by Illinois’ selectivity gap—a wide difference in admissions rates between our CS majors and the rest of the university population. Illinois admits 62% of its applicants. But Computer Science at Illinois admits only 15%!⁽¹⁾ This 47% selectivity gap is more than twice that of other elite CS programs: including CMU (17%), Berkeley (8.5%), University of Washington (18%), and the University of Texas (21%).

My course serves both as the first course for majors and potential minors and as a service course for non-majors with a strong interest in the material. The tension is palpable. The vast majority of my students—over 80% each year—are non-majors. But the course is taught in computer science and has a responsibility to prepare students for the rest of our program. The drop rate among majors is near zero, whereas among non-majors it’s non-zero. I suspect I could cover much more without affecting the majors. But at the price of increasing the non-major drop rate further.

I’ve done my best to find a pace for my course that balances these competing objectives. I’ve also worked outside my course to improve the early curriculum for non-majors—specifically by opening the follow-on programming course to all students, not just majors. But this is a tough challenge and an optimization problem that I’m not convinced has a stable solution.

It’s also important to note that we do offer many other slower-paced CS1-like options, including one in Python, one designed specifically for engineers, another that focuses on data science—and all those just in computer science. And a bunch of other ones that seem to crop up across the university like weeds. Still, my course is the elephant in the room, with the highest profile and largest enrollments—over 1200 for Fall 2020 and over 1400 for Fall 2021.

How many students succeed?

How many students succeed?

I define a course success rate as the number of students who are earning a satisfactory grade in the class. Note that this does not mean the number who are passing, since usually a department does not consider a D to be a suitable grade in CS1. Sure, you can move on, and nobody can stop you. Ds get degrees! But most advisers look for a stronger signal early that a student is prepared to succeed. C’s and D’s in CS1 are very likely to turn into F’s in CS2 and later courses, putting a student’s academic progression at risk. Better to hold them back early than let them fail later.

My understanding is that my departmental advisers will contact any student who receives lower than an A- in my CS1 course. Not all of them will retake the class. But a B grade or lower is seen as an early sign of trouble and a reason to check in. What happens next depends on a student’s circumstances, but it’s not uncommon for these students to retake early courses to make sure that they have a solid foundation before continuing with the program.

As a result, I define the success rate for my course as the percentage of students who receive an A- or above, with the denominator the number of students in the course two weeks into the semester. But that definition may vary depending on your program’s expectations.

It’s fairly obvious how to interpret success rates. However, this is one of the measures I’ve included that is more openly subject to instructor manipulation, since they are assigning grades. It’s worth keeping that in mind. The success rate should also be considered in the context of what the course is trying to accomplish. Courses that introduce more material may end up having lower success rate than a course that doesn’t end up covering that much or effectively prepare students for success in later coursework.

Note that success rates are not the same as 1 - DFW, where DFW is the drop-fail-withdraw rate. While students that earn a C in my course have passed, I don’t consider them to have succeeded. On some level, for many students a C or even a B can be more damaging than a drop, since it ends up on their transcript and affects their GPA, which we’re now apparently requiring students to maintain at incredibly inflated levels. So it seems strange to consider a C a better outcome than a drop when students themselves don’t share that assessment.

And, as it turns out, it’s also possible for the success rate to increase even as the DFW rate goes up. But that’s a story for later.

For my CS1 course our Fall 2020 success rate was 69% and in Spring 2021 it was 63%, both during a year when the course was offered entirely asynchronously online. Success rates for CS majors are much higher: over 95 percent in most semesters.

I think that these numbers are decent, given the objectives that the course is charged with accomplishing; the variation in the student population; and that my students can walk linked lists, recurse over binary trees, implement Quicksort partition, and definitely finish off FizzBuzz. They’ve also done some simple Android development. But I’ll admit that I don’t have a lot of comparison points here. If you have numbers you’d like to share, please get in touch.

It’s also worth considering the rate at which your course produces bad outcomes: specifically low grades and failures. Again, I don’t necessarily consider drops to be a bad outcome, although they do get more costly as the semester progresses. Most of the students who leave my class drop, which is in large part due to our transparent and incremental grading policies. Only 7% in Fall 2020 and 13% in Spring 2021 earned a C grade or below. If we calculate the same metrics used in A Longitudinal Evaluation of a Best Practices CS1, we achieved a Fail Rate (D or below) of 3.2% in Fall 2020 and 6.5% in Spring 2021, both substantially lower than the 9.9% fail rate considered a success in that study.

Finally, given that these data were collected during the COVID year, it’s important to consider the potential impact of university grading policies. In both semesters of the 2020–2021 academic year the University of Illinois provided students with a P/F option. However, this option was provided after the drop deadline in Fall 2020 but before the Spring 2021 semester began. I suspect that this may have had a role in reducing our success rate in Spring 2021, with more students willing to take the pass and not do all of the work required to get an A. At the same time, pretty much all of our overselected majors take my course in the fall, and numbers from that semester always look a bit better due to their influence.

As a side note, I would love to permanently offer my course with either a pass-fail option or entirely pass-fail. But the people in charge tell me that I can’t.

What performance gaps exist between different cohorts?

What performance gaps exist between different cohorts?

While grades are inherently subjective, looking at the performance of different cohorts within the same course can be extremely illuminating. Because while faculty can and do inflate (and deflate) student grades, manipulating the performance of different cohorts requires adjustments that I’m pretty sure would be flagged as capricious grading pretty much anywhere.

Gaps between students with more and less prior experience are harder to interpret. If a CS1 course welcomes students with no prior experience, they need to be able to succeed. However, it’s also unrealistic to expect students with prior experience to have to work as hard as complete beginners. I took advanced beginning French in college. There were people in that class who clearly came in already speaking some French. They did better than I did!⁽²⁾

So small experience gaps are probably acceptable. Larger ones start to create questions about whether the course is really calibrated properly for its student population, or perhaps not teaching the material well enough to support students without prior experience.

One of the things that I’m the most proud of with my CS1 course is how we’ve closed our gender gap over the past few years. A median final grade gap that was 8 percent in Fall 2018 dropped to zero by Fall 2019 and has been at most a point over the past few semesters. Drop rates are also equivalent across genders, so this isn’t something that we’re achieving through attrition.

Our experience gap is also quite small: only 1% between students that self-identified as 1 (no prior experience) versus 2–5 (some to a lot of prior experience) during Spring 2021. I think that this has a lot to do with the fact that we’ve created excellent materials, and don’t create unnecessary competition between students for grades.

Do students succeed downstream?

Do students succeed downstream?

This might be the hardest for a single instructor to measure. It also introduces a bunch of different potential confounders. But it’s probably one of the more valuable metrics to keep an eye on.

Unfortunately gaining access to this data requires some help—either from a downstream instructor, or from your department. My colleagues are always quite willing to share their data with me, and I’ve engaged in a few useful collaborations surrounding cross-course design based on this information. And it can be helpful to work with raw grading data, rather than with the letter grades themselves, where a lot of precision has been lost. But it’s also great if someone in your department has this data or is paying attention to these kinds of cross-cutting metrics. If you work in a department like this, please get in touch! I’d love to know more.

If you can do this, it also helps to have a sentinel course. For CS1 this is frequently the downstream data structures and algorithms class, which in many departments represents the end of the early programming sequence and a gateway into the rest of the program.

In my case I’ll admit that I was not tracking this measure, despite excellent relationships with the faculty teaching the downstream data structures course. I need to do better here.

I did finally get to see some of this data when we assembled some numbers for a site visit to support a grant application. My heart was racing as I process the raw data. But it turned out that I was happy with what I discovered. Since I took CS1, the DFW rate for CS majors in our data structures course dropped from 12% to 6%.

However, recall that our CS students do take a second-semester programming course before proceeding to data structures. And that course itself had gone through some substantial changes. So I was further gratified to see that the DFW rate for non-majors had also fallen by from 19% to 13%.

Clearly there is a lot of additional context you have to bring to these analyses, some of which is missing here. For example, an excellent CS1 course may inspire less-prepared students to continue their studies in CS, and they may end up struggling in later courses. This is particularly true if there is a big difference in experience or general academic preparation between CS majors and non-majors, which is true here and I’m sure also at least some other universities. It can also take a while to observe changes in downstream courses and complete this feedback loop. During that time a bunch of other things may have changed, making it hard to pinpoint what caused things to improve or regress.

How much programming are students doing?

How much programming are students doing?

The numbers above measure student success or failure. But it’s equally important to measure effort. So we’ll close with something near and dear to my heart—the amount of programming that students are actually doing in CS1.

This is particularly important for introductory computer science, which is one of a small set of college classes that teaches a skill.⁽³⁾ When learning to program, practice makes perfect.

Now, obviously there are limits to which you can increase the amount of programming students are doing. At some point the course starts being far too difficult, and students get discouraged.

But I think that many CS1 courses are not even close to that point of diminishing returns. My experience has been that students are highly-motivated to learn computer science and programming, and relish opportunities to improve. We’ve put a lot of effort recently into both building and using new systems allowing us to rapidly author new programming problems for them to solve, to keep up with their desire for more practice.

A straightforward way to quantify this is to just count the number of lines of code that students write each semester. It can also make sense to break that down further by the environment or task that they are working on: unproctored versus proctored, small programming tasks versus larger projects.

I’ve written previously about quantity versus quality in CS1 and the early programming curriculum. But as a recap:

Starting in Fall 2018 we began requiring students to complete a daily homework problem in CS1. We also included several small programming problems on the weekly quizzes given in our computer-based testing facility. In Fall 2018 students completed 108 small programming problems in both unproctored and unproctored environments, writing approximately 14,000 non-commenting lines of code per student, or around 133 lines per student per day over a 15-week semester. By Spring 2020 that number had risen a bit to around 19,000 lines-per-student.

In summary—I guess I’ve written another long bullet-point-delimited essay. I promise that the next thing I post will be in story form, rather than list form.

Overall, even four years later I continue to find trying to teach CS1 effectively an enormous and fulfilling challenge. The design space is huge and was expanded further by the pandemic. There are all kinds of tradeoffs to make and, due to autograding, CS1 courses can explore various types of course assessment design that simply aren’t available to other courses: For example, assigning over 75 daily homework problems without placing any burden on human graders. There’s also huge technology gaps and opportunities in this space to create better and more engaging content and materials.