August 16, 2012
What is “quality” in higher education? Isn’t that the same question as “what makes one institution better than another?”
For most institutions of higher education, “better” would largely or exclusively mean “more likely to engender learning in students.” For those institutions that also have a research mission, “better” would also mean “more successful at creating or discovering new knowledge.” For the present, though, let’s stick with the student learning mission, since (a) that is the mission of most higher education institutions most of the time, and (b) for many institutions that have research as a second mission, the rationale is that a more research-active faculty will be more successful at promoting student learning.
The question of “what is quality” is on my mind today because of a recent, interesting research paper by Sarah Cohodes, and Joshua Goodman, both at Harvard’s Kennedy School, with the wonky title “First Degree Earns: The Impact of College Quality on College Completion Rates.” The Chronicle’s Eric Hoover provides an overview here.
Cohodes and Goodman want to see whether students are more likely to complete their degrees if they go to a “higher quality” college than if they for some reason choose a “lower quality one.” They find interesting data in Massachusetts that arose from a state policy that gave some students tuition waivers if they enrolled at in-state public institutions rather than making another choice, often out of state. “For the marginal student,” they find, “enrolling at an in-state public college lowered the probability of graduating on time by more than 40%. The low completion rates of scholarship users imply the [state] program had little impact on the in-state production of college degrees.”
This is an interesting and perhaps important finding. But I’m troubled at the way they deploy the word “quality.” I’ll let them explain how they define the term:
Specifically, we construct “college quality” from our student level data as the first component from a principal component analysis of each college’s SAT math 75th percentile, admissions rate and student-faculty ratio. We think that the first two capture some element of student quality while the last is a proxy for available resources.
Translated from wonkese into English, they are saying that “quality” means “high selectivity” and “high (faculty) resources.” Those institutions are taken to be higher quality in their study that can (a) exercise more selectivity in admitting students, measured by how small is the percentage of students they admit and how high is the percentage of students they enroll with high math SAT scores, and (b) have higher faculty-student ratios.
But is that “quality?” Do higher education institutions with higher selectivity and higher faculty-student ratios engender more learning? Perhaps. But Cohodes and Goodman do not demonstrate that or point to any research that does. They just posit this definition of quality.
And the awkward thing is, we don’t really know. Why don’t Cohodes and Goodman use data on learning outcomes to define quality, if that’s what “quality” means in higher education? That would be straightforward, wouldn’t it? They don’t because they can’t. Many institutions don’t have systematic data on learning outcomes. And some that do won’t make any of it publicly available. To frost the cake, those institutions that are (a) highly selective and (b) resource rich (in terms of faculty-student ratio, for example) are much less likely to make publicly available any data on learning outcomes. They are less likely, for example, to participate in national indicators of learning outcomes (CLA, NSSE), and less likely to release the data if they do.
So Cohodes and Goodman do what higher education researchers have been doing for some time. In the absence of data on learning outcomes, they posit that “quality” means high selectivity plus high faculty-student ratios. It is the best they can do. But they don’t even pause to note that this is a substitution that has no empirical support.
I believe we should refuse to use the word “quality” in higher education in this way. I’d like to see the empirical confirmation that, in general, higher faculty-student ratios correlate with more student learning. They may, but I’d like to see it.
And I’m very troubled with the student selectivity side of the substitution. It says, in effect, an institution is better (higher quality) that can deny service to more students and that can cherry-pick from among the students seeking admission the ones that have already shown themselves most adept at learning. I want to reserve the word “quality” in higher education for those institutions that are successful at promoting learning among ALL students.