Thursday, January 7, 2010

Checklists and transparency

In an excellent comment on my first checklist post, Bryce Bunting wrote:

On another note, I could see educators voicing arguments similar to those of physicians with regard to why they don't need checklists (e.g. "teaching is an art," "I don't want to be restricted by a checklist," etc.).

Shortly after reading Bryce's comment I got a copy of an article on health care reform from my boss. (Thanks, Cid.) The piece, Making Health Care Better profiles Intermountain Health Care's efforts to improve health care and cut costs (go, Utah!)

IHC uses checklists (or protocols like them) but the improvement of health care system-wide comes not just from the checklists but from using and sharing data from them. IHC's computer system recommends certain treatments to doctors and nurses (the preferred treatments are developed by a team of doctors, nurses, and administrators from IHC). They are free to use those treatments or choose others. Those choices and the outcomes go back into the system. So, over time, IHC has developed a set of preferred practices, based on research and outcomes, and the evidence from those practices is visible.

Contrast this with what happens in education. Teachers prepare for class, using a combination of the things they have done in the past, innovations they prefer, ideas from colleagues, etc. My sense is that most teachers (and nearly all professors) spend a substantial amount of time in preparation, as well we should.

Then they hold class. There, they likely do most of the things they planned to do, leaving some out because of time constraints or because the class goes in a different direction. At some point students are supposed to show evidence that they learned. The professors evaluate them. Class ends. Students get final grades and go on to the next course.

The major difference between excellent health care and excellent college classes, then, isn't the preparation. It probably isn't the presence or absences of checklists either (though they cannot hurt, it seems). Instead it is transparency.

The transparency problem has two components. First, with the exception of student teachers and faculty undergoing review, no other qualified teacher ever sees what happens in a classroom. Second, no one ever sees the outcomes of what happens in a particular class. As a result, systemic improvement generally depends on training rather than results.

Why is the classroom so opaque? Tradition has something to do with it--universities still carry with them a model of master-disciple born in the middle ages, and that model suggests that the key relationship is between teacher and student, not between student and educational system. Academic freedom, at least as we have come to see it, influences as well.

But I think the main obstacles are not cultural, but systemic. After all, faculty favor collaboration with colleagues on nearly every other task (committees, research, writing, etc.) and most good teachers turn to colleagues for guidance all the time. The "fear" of having someone else watch and comment on your work is probably not too deep, particularly if that someone is a fellow faculty member.

But the system makes it impossible in three ways: first, almost no system of education believes in standard practices (a "meta-checklist" if you will). Teaching chemistry differs from teaching history; teaching first-year students isn't like teaching grad students. But why is this? Those few institutions with a campus-wide commitment to particular teaching approaches seem to have positive effects. Second, we schedule faculty work so nearly all professors are teaching simultaneously. Why don't colleagues visit your classes? Because they are in their own. Third, we don't know what outcomes would look like. What would be the result of a well-taught class in history? How would it differ from a similar course in, say, music?

I don't have good solutions to these transparency problems. But I can think of a simple first step. As part of campus faculty development efforts and accounted for in faculty workload, mandate (or incentivize) faculty members to enroll in one class taught by a colleague each year. A few benefits would be almost immediate. First, faculty development would be campus-wide rather than focused on the willing few. Second, faculty would learn from each other. Third, they would see the experience of students first-hand. And fourth, colleagues would be in a position to talk with each other about the best ways to do our common work.

Tuesday, January 5, 2010

Education checklists?

This morning NPR interviewed the surgeon Atul Gawande about his new book The Checklist Manifesto. In it he argues that surgeons do much better work when their surgeries are guided by checklists. As the surgery goes along, surgeons, nurses, and techs make sure they have covered all best practices by completing a checklist. Gawande reports that surgical errors declined by 35% when surgeons used a checklist. The checklists are based on those used by pilots and developed by Boeing.

I have often advised my students to create checklists, though more as a time management practice than as something that would make their performance better. But the Gawande interview left me wondering if colleges ought to use checklists much more frequently.

Take, for example, advising, mentoring, and other student support activities. Campuses recommend them for all students, but often the best students (or at least the most compliant) are those who take advantage, while at risk students don't. Do schools use mandatory checklists to ensure that all students take advantage of high impact experiences? (George Kuh and the NSSE people found that students of color and first-generation students are less likely than other students to participate in high impact activities like learning communities that are proven to improve student engagement.)

Would retention go up 35%?

Or consider faculty members. What if they went into every class with a checklist of best practices. Write outline on the board...Take roll...Elicit questions about previous class meeting...Uncover pre-existing understanding among students...Switch pedagogies every 15 minutes...Give students assignments to be completed in class...Check for understanding...

Would learning improve 35%?

If not, why? Is a failure to learn not the result of mistakes and overlooked opportunities? Are some students not retained because they simply "fall through the cracks"? Is education more complicated than surgery?

Sunday, January 3, 2010

What do we make of the shakeout in higher ed open content?

First, a disclaimer. I am a neophyte in the opencourseware/open content world. But I've been following it for the past year, and am involved in Westminster's efforts to figure out how to play in that world. And if you follow this blog (all three of you :-)) then you will know it is central to some of my thinking about cost and quality in higher ed.

Having said that, it seems to me that over the past couple of months there have been some major changes in the higher ed open content landscape. First, Utah State University got rid of its opencourseware project, the second largest in the US. Second, MIT released a report on the costs of hosting its OCW project, with some ideas about how to sustain it in the future without the major institutional and grant funding that made it possible in headier economic times. Third, the Mellon Foundation consolidated one of its major grant programs that had funded a lot of open content work and its two grant officers left the foundation.

What do these three things imply?
1. They do not imply that open content or opencourseware are going out the window. There are still lots of efforts out there, and lots of successes to be noticed.

2. They do imply that individual institutions of higher education will struggle to provide open content infrastructure necessary for the "revolution" in learning that some people expect. (Check out this recent issue of the journal IRRODL for some of the revolution language.)

3. They suggest that the same economic downturn that makes open content attractive as an alternative to expensive higher education makes it difficult to provide that open content.

4. If (3) is correct, then future open content infrastructure will be more likely to be provided by for-profit companies (i.e. youtube.edu; ITunesU, etc.) or the federal government than by individual institutions of higher ed.

5. If individual institutions want to stay at the front, it seems that they ought to specialize. The future seems not to be MIT providing an entire university worth of courses online for free, but instead institutions leading in areas of their speciality. MIT in engineering, Utah State in agriculture, who (???) in liberal arts.

Wednesday, December 23, 2009

What do we lose when we overlook the stories of students?

It is not a wise thing to make meaning out of coincidence. I'll do it here.

My daughter, about whose struggles I have written off and on, got home from her first semester at college the same day I turned in grades for my freshman seminar (about which I have written as well). My daughter's first semester grades were quite good, and after a difficult semester--for her emotions, her connections with others, and her sense of self--she got home with a bit of hope.

The day she got home I gave 3 failing grades in my seminar. They were for students who never attended or completed work, and who did not respond to my efforts (or those of others) to reach out to them. Those students, and the rest in my class, though, are registered for next semester. And so, they may stay at the college (I would bet against it). I can't know what their visits home were like.

My daughter and the students in my seminar have been, for the time being, retained. That fact will be recorded in the retention data we keep, and over which we puzzle. And that fact will be publicized, especially when the retention numbers are high. (The President of my daughter's college, in a letter at the beginning of his annual report, noted that 96.6% of undergraduates returned in the last year for which they have statistics.)

But having experienced my daughter's first semester so intensely, and having some insight into the experiences of my students, I can't help but feel the loss when those experiences are folded into "retention data."

Perhaps I have been too much influenced by my training as a historian, or my fondness for the writing of Wendell Berry and its reminder to tell local stories. And as I have been thinking about this post, a few lines from Zbigniew Herbert's "Mr. Cogito on the Need for Precision" keep coming to mind.

Herbert was a Polish poet, and witness to many of the atrocities of totalitarianism. Mr Cogito is his alter ego. Near the end of a long poem on the violence of the 20th century, Mr. Cogito reminds us:

[...] in these matters
Accuracy is essential
We must not be wrong
Even by a single one

We are despite everything
The guardians of our brothers

Ignorance about those who have disappeared

Undermines the reality of the world.

(Commentary here and here.)

I know that staying in college (or not) is nothing like disappearing in Stalin's gulag. But being invisible to people who, for whatever reason, are supposed to be connected to you, is a failure. And if the stories of those students who stay, and those who leave, are lost, then in important ways our systems allow "ignorance about those who have disappeared." If Herbert is right, this is not just a small problem. It is a way of undermining the reality of the world, turning it from concrete to something abstract.

So at the end, here, let me put in a plug for gathering and using stories of retention (or of not-retention). The fact that 78% or 96.6% of our students come back tells us only a little. What will tell us more is if we know our students well enough to say for each one that he or she has stayed (or left) for good reason. And it isn't enough, probably, for student affairs staff, or a professor here or there, to be able to speak to the actual experience of actual students. Their stories need to be told, and remembered, by fellow students, and by administrators, and leaders.


Colleges imagine themselves as communities. We aim to build places where relationships lead to learning. If we are to be communities of learners, we must remember that we are, "despite everything the guardians of our brothers."


Friday, December 18, 2009

Learning about cost and quality from health care and agriculture

Atul Gawande has a great piece in the 14 December 09 New Yorker. "Testing, Testing" contains hints at two ideas that would help higher ed move ahead on the cost/quality issue.

First, he reminds us that for health care reform to succeed government has to mandate that nearly every American get health insurance. Or, put another way--the best way to ensure access to health care is to require that people get access to health care. Mandated access is important because it is only by expanding the market that the pool of the insured is large enough to make insurance a viable business.

Access to higher education is the sometimes forgotten key to cost and quality in higher ed as well. Millions of new potential university students aren't attending college. And millions more who have dipped their toes into higher ed are no longer pursuing it. Why? Cost is a big factor, along with cultural challenges, and the indifference of much of HE to their plight. So, if we want movement on cost, perhaps we need to move on requiring access. (I know the objections--not every job requires higher ed, many, many people can't afford it, it extends the power of the state well-beyond its current already-impressive reach, etc.)

And who knows--millions more adults, required to get higher education, might be a powerful force for reform of higher education. (Right now students and potential students have a very limited role in pushing reform--they act largely by choosing another campus, dropping our, or never joining up.)

Gawande then explains why access should be a requirement, but there should be no standardized approach to insurance and treatment. To make this point he turns to another mandatory practice--eating. In the late 19th and early 20th centuries food was expensive and access was limited. Farming was outdated, soils depleted, public health poor, life expectancy relatively low, farm work brutal.

Government did not mandate a particular approach to growing food, though. Instead it catalyzed changes through extension agents. One particularly influential agent, Seaman Knapp, traveled to Terrell Texas to try to get farmers to try out some new techniques. Not every farmer adopted them at first. Instead, community leaders (most of them farmers) encouraged one farmer, Walter Porter, to try the new techniques. Terrell's leaders provided support for Porter. In some ways they helped reduce his exposure to risk (of embarrassment, failure, financial ruin) while he experimented. When his changes made a difference, most of the farmers in the area adopted them. Food costs began to decline, access to food improved, as did quality and public health. Based on this success the US government created an army of extension agents, dedicated to doing what Walter Porter did.

Now the history of agriculture since then hasn't been all good. Small farms have died, natural growing techniques have disappeared, agribusiness rules the day.

But in that story there is an important suggestion for higher ed. Experiments on cost are risky when gone alone. But a consortium of schools can help spread risk. The consortium agrees that a particular approach looks promising. One school agrees to try it out. The other schools indemnify the experimental school. If things go poorly the consortium helps fix the problem ( by providing money, support etc.) But if they go well the consortium shares in the benefit and they all adopt the changes. The consortium comes to look something like a mutual aid society--a group of people or organizations committed to the well-being of their members by sharing risk and success.

Today there are plenty of consortia of colleges and university. But few of them collaborate in this way. Instead they gather for meetings, swap stories, produce papers, but when the chips are down members of the consortium are on their own. Perhaps it is time to look to the past when these cooperative organizations were the basis of our social safety net and civic innovation. Couldn't hurt.

or if the historical allusion doesn't work, consider a modern one--venture capital. Imagine 20 college presidents, each serious about cost/quality. They each pitch in $100K to a fund. They fund experiments, and get equity in the results. Something works, all benefit, something fails, no one faces a total crisis.

Thursday, December 17, 2009

Can all students have an honors-like experience?

Last week a group of us--faculty, staff, administrators--got together for a conversation about vision in higher ed. Most of us lead co-curricular programs. The premise of the conversation was that in a time where people feel constrained, anxious, etc. about the future of higher education it makes sense to think about what a better world would look like. So we talked about these questions: What would you do if you suddenly had many more resources to run your program with? What if you had freedom to lead your program where you want it to go. Where would you lead?

Though we all said it in different ways the group came to one agreement--if we had freedom and/or resources, we would spend more time working directly with students, and that work would look a lot like an honors program.

I should explain--most of the group has little interest in teaching honors classes--at Westminster small seminars team-taught by our best faculty for students with the highest academic profile. But there are components of the honors program, or athletics, or music students, that most of us enjoy and believe lead to good learning. Here they are:
  1. students strongly committed to the program--once students are admitted to honors they rarely leave. Instead that commitment becomes part of their identity
  2. long-term commitments and connections--honors students participate in two years of seminars and then complete an honors thesis. In other words, they commit to each other and the program for the duration of their time at the college. And the honors faculty return year after year, so the connections between faculty and students endure as well.
  3. freedom to give the program distinctiveness--honors students complete their liberal education requirements in the program. But their LE looks substantially different from that of other students
Not every student can be an honors student. (After all, honors is a way of drawing distinctions between some students and others.) And honors is an expensive program per-student. Small team-taught classes are expensive. So are scholarships for students. But the conversation got me thinking about what a school built around an honors-like experience would look like. Here are some thoughts;
  • the campus would be broken into smaller and longer-lasting groups--every student in a cohort of 20, say, for their entire experience at the college.
  • individual faculty/staff would connect with a group the entire time they are on campus.
  • the faculty/staff group leader would have a substantial amount of say on who is admitted to the group and what the group's program looks like.
  • groups might overlap, and they would share administrative support, but there would be no common experience for all students, except the experience of being part of an on-going group of fellow-learners.
  • within the groups, students would have heightened leadership roles, shaping the activities of the group, its educational experience, and its standards.
  • in short, the educational experience would be decentralized.
Is this the sort of thing you would like to see happen on your campus? Could it? What would be the result for students? For faculty and staff? For the institution? If not, why does it work for honors students but not the rest of the student body?

Wednesday, December 16, 2009

Learning Through E-Portfolios

This semester my class has been piloting a new e-portfolio project at Westminster. The overall goal of the initiative is to have all undergraduates create e-portfolios showing that they have achieved the college-wide learning goals. The portfolio would include artifacts and reflections from the student's entire time at the college, and from the co-curriculum as well as the curriculum.

During the semester we have been testing the e-portfolio software (Foliotek) as well as seeing if first-semester students care about things as abstract as our learning goals, can use rubrics, can reflect on ways that their artifacts are evidence of learning, etc. And we have been investigating whether this can happen in a freshman seminar.

I cannot speak for what the students have learned, but I've learned quite a bit about learning goals, definitions, rubrics, and e-portfolios. Here are a few of the highlights:

1. Come around to the learning goals--my students are cool to abstraction, so when confronted with a learning goal like "global consciousness, social responsibility, and ethical awareness" they cannot get much traction. But start a discussion about how outlawing DDT preserved the environment in the US while heightening the impact of malaria in the developing world and they connect right away to the goal. Present them with a definition and the class becomes as interesting as a dictionary. Ask them to create a definition based on the DDT conversation and all of the sudden definitions are more interesting. Of course this is true, you will say. But a lot of the work on learning goals/outcomes, whatever its aim, comes across as abstraction.

2. When using a rubric, have students score the work of strangers first--one of our goals was to see if the rubrics for our learning goals made sense to students. So I asked them to use our rubrics to evaluate their work. As is almost universally the case, the students overestimated the quality of their work. Consistently. Through the semester. But when some students used the rubrics to analyze the work of professional journalists writing about science, the rubrics gave them a framework to point out the gaps in professional writing. The lesson? A rubric is a frame that makes sense of the world, not a window into your own soul.

3. Creativity is a good way into reflection--I had students write reflections about their work in critical thinking, ethical awareness, and communication. Their reflections tended not to show much self-awareness. But when I asked them to reflect on their creativity, the reflections were much more powerful. In retrospect, this is understandable. We understand creativity as being personal and internal--something emerging from who we are. Communication is external--a tactic rather than a trait. By writing about their acts of creativity students were much closer to who they think they are, good and ill.

4. A small barrier is all it takes--with something new like an e-portfolio, a small barrier was all it took to discourage students from participating. In the foliotek system a student can begin to create her e-portfolio only upon receiving a user name and password from foliotek. (Foliotek sends them out once we request the creation of an account in a particular student's name.) For students who deleted the email, or skipped it, or who never read their campus email account, the small obstacle of having to hunt down the email was very likely to keep them from getting started on the portfolio or doing good work in it. I know, I know--causation vs. correlation. But when I do it again I am going to make sure I lower the barriers as much as possible for students, because if you don't start you go nowhere.