Academically Adrift: Doing assessment at small colleges

Academically Adrift has reignited the debate about the amount of value that attending college adds to a student's learning.  Its conclusions--that many students do not learn much during college, and that that fact is due to the low requirements for the amount of student work are both sad and unsurprising.

The book's reliance on data from the CLA has received much less comment than its conclusions.  This is due, in part, to the fact that over time the CLA has become a non-controversial assessment tool.  But it is due also to the fact that few people outside higher ed know how the CLA works on a particular campus.  For our campus, though, the CLA, regardless of what is says about our students' performance (and the news is sometime quite good), is always of questionable value.

The CLA purports to measure the "value-add" of a college by giving students a real-world critical thinking and writing challenge, and then measuring how students perform on that test in comparison to how the CLA (using a sophisticated algorithm) predicts they should have done.  If senior students perform better than predicted (both by the performance of freshmen and by their own aptitude scores), that improved performance is the "value-add."  Schools receive a report from CLA that breaks down student performance by field of study, and that compares the college's value-add to that of other institutions.  Institutions are free to do with the data what they want.  In my experience what they most want to do is see how they stack up against their peer and aspirant institutions on the "value-add" measure.

Westminster has used the CLA for six years. Each year's report is met by the same reactions.  First, whether our value-add is high or low, we always wonder what that measure is due to.  Because the CLA reports data only at major and campus level, and because it is a cross-sectional study, it is never possible to be sure what causes student performance gains or losses.  Is there a particular class that made our philosophy students a year ago stand out so much?  Is there a particular gap in the liberal education curriculum that lead our students to perform poorly on the "make an argument" portion of the test? Did some change in advising or curriculum have an effect?  Do we just happen to have an outstanding bunch of freshmen whose scores make our senior's performance look bad? How would we know?

Second, and most importantly, our results are always compromised by small sample size.One year we were able to get the entire graduating class of philosophy majors to take the test.  They performed very well.  But there were six of them, so the statistical significance of the findings is in question.  Some years we are only able to get 14 nursing majors to take the test, and so we have a huge standard deviation.  Every year there are outliers who either make our scores look great or make them look awful.

The message here isn't that the CLA is a bad tool, or that Academically Adrift's conclusions are dubious.  It is, instead, that small colleges and universities are poorly served by assessment tools that sample, aggregate data, and make comparisons at the campus level.  The numbers in our sample groups will always be too small to be trustworthy, and the unit of measure (the entire college experience) too large to do something about.

Instead of these sort of abstract measures, small colleges and universities need to become much more serious about tracking and influencing individual student performance over time.  There is no reason why a campus could not administer a measure of student learning each year to each student and track that particular student's performance. Then, if a student performs poorly on critical thinking, the student's advisor could recommend a particular course, or a particular shift in study habits, to respond.

The assessment mantra of small colleges should be something like this: Disaggregate, don't aggregate.  Do longitudinal studies, not cross-sectional ones.  And most importantly, assess the learning of students as real living human beings, not as part of an abstraction of how the entire institution is doing.

Such steps will make it harder for small colleges to play in the rating/ranking game that measures like CLA and NSSE allow.  But it will allow small colleges and universities to be able to link assessment and learning in the lives of individual students--the thing we say we do right now.

online, blended, and face-to-face courses

This report: http://www.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf is worth reading. The press coverage (http://www.insidehighered.com/news/2009/06/29/online) focuses on the finding that where there is a difference, online classes outdo face-to-face classes.

I've been reading the report, though, and there is a lot more in it worth paying attention to. Here are a few that stand out to me:
1. that blended courses outdo face-to-face only and online (where there is a difference)
2. that the key variable might be time, not the medium of the course--that is, that time on task is the key factor in learning (consistent with all sorts of other studies)
3. that online and blended courses excel in their ability to foster reflection and community (characteristics that are sometimes associated with face-to-face courses)

I'm also intrigued by the report's typology of online courses: expository, active, and interactive (see the table on p. 25) Seems like a smart way to think about all sorts of learning.

See the summary of findings and conclusions on pp. xvi-xvii.

Thoughts?

Expectations Gaps in HS and College

Civic Enterprises has just released a report on the dropout problem in high school. One of the key findings of On the Front Lines of Schools is that while many teachers and principals doubt that all students can take college-prep courses and graduate HS, 2/3 of dropouts say that they would have stuck with school if there had been higher expectations for their performance. Front Lines' authors call this the "expectation gap."

I wonder what that gap might mean for thinking and practice in higher ed. When students aren't retained or fail to graduate college, we rarely attribute it to low expectations. More commonly we imagine that college drop outs are caused by too-high expectations at the college (or financial and other personal challenges).

Would colleges retain more students if expectations (and support) were higher?

Learning from other high cost enterprises

Atul Gawande's piece, "The Cost Conundrum" in the New Yorker is startling. He describes McAllen Texas, one of the poorest, least healthy, and costliest (at least for health care) places in America. The cost isn't due to its distance from major cities, or its unique health challenges, but instead to four other things:

1. McAllen physicians seem to be particularly entrepreneurial, seeking many ways to make money--through owning buildings, clinics, and stores instead of simply maintaining a practice.
2. McAllen health care professionals are also particularly non-communicative, meaning that they are unaware of the high costs of health care in their town, unaware of what other physicians are doing, and unaware of the health benefits (or lack thereof) of particular medical actions.
3. As a result of (1) and (2) above, McAllen physicians recommend far more tests and procedures than their counterparts elsewhere.
4. As a result of (3), health care costs are unusually high in McAllen.

Physicians in healthier, less expensive places avoid (1) and (2) above, and therefore avoid (3) and (4). Instead they have created ways to share information about cost and treatment, and devote more time to patients as individuals, rather than patients.

The article got me thinking about cost in higher education. Education is one of the only other fields where costs have risen with a pace and force comparable to those in health care. And education, at least in many areas, follows the practices of McAllen physicians. We are entrepreneurial, non-communicative and non-cooperative (especially with colleagues from other institutions), and therefore our goal becomes to recruit and place students in programs (treatments) rather than meeting them where they are. (In addition, their funding streams are coming to look more similar every day, with the federal government paying out large portions of the cost of health care and education.)

Higher education lacks one thing that health care has--a clear measure of health. One of the things that physicians in low-cost high quality towns are certain about is how to help people manage their health. Educators lack this. We don't agree on what educational sickness looks like (is it the absence of a degree? a job? certain facts? certain habits of the mind?), and we don't agree on what educational health looks like.

In the absence of clear measures, can we learn from the story of McAllen? Can we focus on students as individuals with particular needs and desires? Can we shape education to meet those specific needs? Can we cooperate on outcomes and treatments so that we increase educational health and reduce its costs? And can we help students, parents, faculty, donors, legislators, and regulators think this through?