Academically Adrift has reignited the debate about the amount of value that attending college adds to a student's learning. Its conclusions--that many students do not learn much during college, and that that fact is due to the low requirements for the amount of student work are both sad and unsurprising.
The book's reliance on data from the CLA has received much less comment than its conclusions. This is due, in part, to the fact that over time the CLA has become a non-controversial assessment tool. But it is due also to the fact that few people outside higher ed know how the CLA works on a particular campus. For our campus, though, the CLA, regardless of what is says about our students' performance (and the news is sometime quite good), is always of questionable value.
The CLA purports to measure the "value-add" of a college by giving students a real-world critical thinking and writing challenge, and then measuring how students perform on that test in comparison to how the CLA (using a sophisticated algorithm) predicts they should have done. If senior students perform better than predicted (both by the performance of freshmen and by their own aptitude scores), that improved performance is the "value-add." Schools receive a report from CLA that breaks down student performance by field of study, and that compares the college's value-add to that of other institutions. Institutions are free to do with the data what they want. In my experience what they most want to do is see how they stack up against their peer and aspirant institutions on the "value-add" measure.
Westminster has used the CLA for six years. Each year's report is met by the same reactions. First, whether our value-add is high or low, we always wonder what that measure is due to. Because the CLA reports data only at major and campus level, and because it is a cross-sectional study, it is never possible to be sure what causes student performance gains or losses. Is there a particular class that made our philosophy students a year ago stand out so much? Is there a particular gap in the liberal education curriculum that lead our students to perform poorly on the "make an argument" portion of the test? Did some change in advising or curriculum have an effect? Do we just happen to have an outstanding bunch of freshmen whose scores make our senior's performance look bad? How would we know?
Second, and most importantly, our results are always compromised by small sample size.One year we were able to get the entire graduating class of philosophy majors to take the test. They performed very well. But there were six of them, so the statistical significance of the findings is in question. Some years we are only able to get 14 nursing majors to take the test, and so we have a huge standard deviation. Every year there are outliers who either make our scores look great or make them look awful.
The message here isn't that the CLA is a bad tool, or that Academically Adrift's conclusions are dubious. It is, instead, that small colleges and universities are poorly served by assessment tools that sample, aggregate data, and make comparisons at the campus level. The numbers in our sample groups will always be too small to be trustworthy, and the unit of measure (the entire college experience) too large to do something about.
Instead of these sort of abstract measures, small colleges and universities need to become much more serious about tracking and influencing individual student performance over time. There is no reason why a campus could not administer a measure of student learning each year to each student and track that particular student's performance. Then, if a student performs poorly on critical thinking, the student's advisor could recommend a particular course, or a particular shift in study habits, to respond.
The assessment mantra of small colleges should be something like this: Disaggregate, don't aggregate. Do longitudinal studies, not cross-sectional ones. And most importantly, assess the learning of students as real living human beings, not as part of an abstraction of how the entire institution is doing.
Such steps will make it harder for small colleges to play in the rating/ranking game that measures like CLA and NSSE allow. But it will allow small colleges and universities to be able to link assessment and learning in the lives of individual students--the thing we say we do right now.
Saturday, February 26, 2011
Tuesday, February 22, 2011
Portfolios as a tool to respond to the big questions about private higher education
It is not clear what higher education will look like in a decade. Or, to be more precise, it is clear that higher education will be even harder to describe in 10 years than it is today. There will be more varieties of schools, more ways to get degrees, more degrees available, more disagreement about higher education's values, and more debate about the value of higher education. Some well-regarded schools will be shuttered; schools that you've never heard of will surge to prominence.
But while the higher education environment of the future is hard to describe, the questions that will form that environment are clear. Campuses will distinguish themselves by how they respond to these questions, and by the tools they choose to craft their responses. By choosing to use portfolios to support, track, and assess student achievement of Westminster's college-wide learning goals, we are selecting certain responses to these questions. in turn, those responses can help distinguish a Westminster experience from those at other colleges and universities. A portfolio system, then, is nothing unique. But the way we use it at Westminster will allow us to bolster our claims of uniqueness while strengthening those portions of the learning experience that we do best.
Here are the big questions:
Which students will an institution choose to serve? There is a disjunction between the major trends in college-going and the sorts of students that many private colleges and universities hope to recruit. More and more college students will be first-generation students, or students of color, or from low-income homes, or returning to school after a career. Many will need remediation, and many will face added difficulties staying in school. At the same time, most private colleges and universities will compete for two other classes of students--those that can pay a large portion of private school tuitions, and those whose prior educational achievement adds to the prestige of the institution. These students may be more likely to be retained, but their number is relatively small.
Adopting a portfolio for all undergraduate students allows Westminster to aim for a broad range of students, because the assumption of portfolios is that they help students demonstrate their growth over time. Doing so allows us to shift the conversation away from demographic characteristics and towards the fit between students and the learning environment of the college.
How will those students learn? Prognosticators assume that this is a closed question--that in the future more learning will take place via technology. This is undoubtedly true, but hardly meaningful, since educational technology has always tracked with technological innovation in society. The bigger question is whether students will learn only in one way, and only in the classroom; or whether their learning (or the learning we count) will be various and take place everywhere.
By investing heavily in the learning environment--in undergraduate research, and civic engagement, and environmental programs, and global learning, and student life--Westminster is wagering that learning will be varied and constant. And by requiring a portfolio for all undergraduate students, the college is arguing that students need to be aware of and responsible for the varieties of their learning.
How will they demonstrate their learning? Colleges and universities have a long list of ways for students to demonstrate their learning--tests, papers, projects, presentations, lab reports, reflections, performances, etc. Students, at the same time, often say that while they learned a lot in class, the most significant learning in their lives took place outside the classroom. This is undoubtedly true, and undoubtedly difficult to demonstrate.
A portfolio system attached to the college-wide learning goals makes a bold claim--that learning inside and outside the classroom can, with the right measures in place--be demonstrated in the same way. Artifacts provide evidence for learning whether they come from a class or a club or an act of civic engagement. And reflections require students to make connections between the artifact and the learning outcomes, showing how learning outside of class is equivalent to that from within.
What role will faculty and staff play in that learning? While the phrases "sage on the stage" and "guide on the side" refer to two main varieties of in-class faculty behavior, it is the case at Westminster that faculty play a much more complicated role in student learning. A faculty member, through the entire course of her interaction with a student, is a recruiter, an advisor, a mentor, a teacher, an antagonist, a colleague, and an evaluator. Increasingly staff members play a similar range of roles.
The temptation in higher education is to specialization--to decoupling these roles in order to be more efficient. A portfolio system, though, helps to link those roles, since it asks students to make connections across their experiences and asks faculty to facilitate those connections rather than focus entirely on classroom interactions.
How will campuses innovate? Some campuses innovate relatively little. Others focus largely on finding efficiencies in their systems, but do relatively little innovation in the academic setting. Still others locate innovation largely in the creation of new academic programs. And many of the most innovative campuses innovate in silos, so that individual innovations do not add up to something in common.
Portfolios should strengthen the innovation culture at Westminster in two ways. First, they help students draw together their experiences with innovative programs--so that, for example, they see how participation in the Westminster Scholars program, their concern for environmental sustainability, and their majors go together. Second, they will help the campus see where innovation needs to take place. If the quality of work in, say, the "leadership, collaboration, and teamwork" learning goal is relatively small, it signals that the college needs to strengthen its work there. Or if portfolios show us that learning is particularly robust in "global consciousness, social responsibility, and ethical awareness" then we know that we have an advantage in that area.
How will campuses demonstrate their value to stakeholders? This is perhaps the biggest question for the institution as a whole. Parents, students, funders, legislators, employers, accreditors, and campus members are all asking whether college is worth the cost. A portfolio system allows a deep response to this question. It helps students identify how their college experience as a whole was valuable. It demonstrates to employers what graduates can do, not just what they know. It shows legislators, funders, and friends what the results of a Westminster experience are, and it portrays concretely the college's ability to live up to its strategic plan and mission. In this way, then, portfolios allow a new richness in the way schools sum up their work, for they give us evidence, in a common format and on common themes, of the power of a Westminster experience.
But while the higher education environment of the future is hard to describe, the questions that will form that environment are clear. Campuses will distinguish themselves by how they respond to these questions, and by the tools they choose to craft their responses. By choosing to use portfolios to support, track, and assess student achievement of Westminster's college-wide learning goals, we are selecting certain responses to these questions. in turn, those responses can help distinguish a Westminster experience from those at other colleges and universities. A portfolio system, then, is nothing unique. But the way we use it at Westminster will allow us to bolster our claims of uniqueness while strengthening those portions of the learning experience that we do best.
Here are the big questions:
Which students will an institution choose to serve? There is a disjunction between the major trends in college-going and the sorts of students that many private colleges and universities hope to recruit. More and more college students will be first-generation students, or students of color, or from low-income homes, or returning to school after a career. Many will need remediation, and many will face added difficulties staying in school. At the same time, most private colleges and universities will compete for two other classes of students--those that can pay a large portion of private school tuitions, and those whose prior educational achievement adds to the prestige of the institution. These students may be more likely to be retained, but their number is relatively small.
Adopting a portfolio for all undergraduate students allows Westminster to aim for a broad range of students, because the assumption of portfolios is that they help students demonstrate their growth over time. Doing so allows us to shift the conversation away from demographic characteristics and towards the fit between students and the learning environment of the college.
How will those students learn? Prognosticators assume that this is a closed question--that in the future more learning will take place via technology. This is undoubtedly true, but hardly meaningful, since educational technology has always tracked with technological innovation in society. The bigger question is whether students will learn only in one way, and only in the classroom; or whether their learning (or the learning we count) will be various and take place everywhere.
By investing heavily in the learning environment--in undergraduate research, and civic engagement, and environmental programs, and global learning, and student life--Westminster is wagering that learning will be varied and constant. And by requiring a portfolio for all undergraduate students, the college is arguing that students need to be aware of and responsible for the varieties of their learning.
How will they demonstrate their learning? Colleges and universities have a long list of ways for students to demonstrate their learning--tests, papers, projects, presentations, lab reports, reflections, performances, etc. Students, at the same time, often say that while they learned a lot in class, the most significant learning in their lives took place outside the classroom. This is undoubtedly true, and undoubtedly difficult to demonstrate.
A portfolio system attached to the college-wide learning goals makes a bold claim--that learning inside and outside the classroom can, with the right measures in place--be demonstrated in the same way. Artifacts provide evidence for learning whether they come from a class or a club or an act of civic engagement. And reflections require students to make connections between the artifact and the learning outcomes, showing how learning outside of class is equivalent to that from within.
What role will faculty and staff play in that learning? While the phrases "sage on the stage" and "guide on the side" refer to two main varieties of in-class faculty behavior, it is the case at Westminster that faculty play a much more complicated role in student learning. A faculty member, through the entire course of her interaction with a student, is a recruiter, an advisor, a mentor, a teacher, an antagonist, a colleague, and an evaluator. Increasingly staff members play a similar range of roles.
The temptation in higher education is to specialization--to decoupling these roles in order to be more efficient. A portfolio system, though, helps to link those roles, since it asks students to make connections across their experiences and asks faculty to facilitate those connections rather than focus entirely on classroom interactions.
How will campuses innovate? Some campuses innovate relatively little. Others focus largely on finding efficiencies in their systems, but do relatively little innovation in the academic setting. Still others locate innovation largely in the creation of new academic programs. And many of the most innovative campuses innovate in silos, so that individual innovations do not add up to something in common.
Portfolios should strengthen the innovation culture at Westminster in two ways. First, they help students draw together their experiences with innovative programs--so that, for example, they see how participation in the Westminster Scholars program, their concern for environmental sustainability, and their majors go together. Second, they will help the campus see where innovation needs to take place. If the quality of work in, say, the "leadership, collaboration, and teamwork" learning goal is relatively small, it signals that the college needs to strengthen its work there. Or if portfolios show us that learning is particularly robust in "global consciousness, social responsibility, and ethical awareness" then we know that we have an advantage in that area.
How will campuses demonstrate their value to stakeholders? This is perhaps the biggest question for the institution as a whole. Parents, students, funders, legislators, employers, accreditors, and campus members are all asking whether college is worth the cost. A portfolio system allows a deep response to this question. It helps students identify how their college experience as a whole was valuable. It demonstrates to employers what graduates can do, not just what they know. It shows legislators, funders, and friends what the results of a Westminster experience are, and it portrays concretely the college's ability to live up to its strategic plan and mission. In this way, then, portfolios allow a new richness in the way schools sum up their work, for they give us evidence, in a common format and on common themes, of the power of a Westminster experience.
Friday, February 18, 2011
Public places, contemplation, and E. B. White
My friend Diane VanderPol, the Director of the Giovale Library at Westminster, wisely pointed out that libraries have bucked the decline in public spaces. In response to my last post she wrote:
I hold out hope for one public something: libraries! In the economic downturn, public libraries have seen increases in numbers of visitors, check outs and program attendance. During the very eras that saw a decline in the private fraternal orders, civic groups, etc, public libraries- as municipally and regionally supported entities full of paid professionals- thrived. Public libraries= the quintessential third places and the last hope for public something!
Diane's is an important insight because it highlights an exception to the rule of public decline. But it is even more interesting because it hints at an important but under-appreciated role of public spaces--they are a venue for communal contemplation.
Thinking of public places as refuges goes against most of the literature on those locations, which focuses on their ability to spark deliberation and engagement between citizens. Think for example, of Robert Putnan's Bowling Alone which uses individual action in public spaces as a metaphor for the decline of community life. Or consider Christopher Lasch's The Revolt of the Elites, which laments the decline in public dialogue that has accompanied the decline in third places.
Recall for a minute, though, those types of public spaces that have flourished in the past 20 years--libraries, spas, yoga studios, coffee houses, and trail systems. They all provide people the opportunity to be alone in public. When I run on the trails near my house I often see other runners. We always greet each other but never stop to talk. When my wife and I stop at Starbucks we are surrounded by other couples who engage in quiet conversation, and by individuals alone with newspapers or iPods. Spas fill up with people getting massages or manicures in close physical proximity, but they may never speak to each other. Yoga classes consist of rooms full of people enacting what has been for millenia a contemplative practice. And when I have had enough with my office and the pressures flowing through the computer and phone, I get up, walk across campus, and enjoy the peace of being quietly surrounded by dozens of other people who pay me no mind.
Recall for a minute, though, those types of public spaces that have flourished in the past 20 years--libraries, spas, yoga studios, coffee houses, and trail systems. They all provide people the opportunity to be alone in public. When I run on the trails near my house I often see other runners. We always greet each other but never stop to talk. When my wife and I stop at Starbucks we are surrounded by other couples who engage in quiet conversation, and by individuals alone with newspapers or iPods. Spas fill up with people getting massages or manicures in close physical proximity, but they may never speak to each other. Yoga classes consist of rooms full of people enacting what has been for millenia a contemplative practice. And when I have had enough with my office and the pressures flowing through the computer and phone, I get up, walk across campus, and enjoy the peace of being quietly surrounded by dozens of other people who pay me no mind.
I have no great wisdom about why this is the case. But it is true that those places that once provided contemplation--churches on one hand and homes on the other--have lost their ability to do so. Home is either a riot of engagement--getting kids to lessons, rushing to make a meal, cleaning house, watching TV while online--or a place of genuine alone-ness. Churches (at least in the Protestant traditions) have moved away from quiet, solitude, and thoughtfulness to electrified music, non-stop talking, and enforced interactions. So perhaps we must get into public now to find the conditions we need to be alone.
Is there any hope for public anything?
Both at the federal and the state levels, the same questions I raised about the future of public higher education can be asked about the future of the entire public sphere. With the need for significant budget cuts and the anti-government views of most leading elected Republicans, it is not at all certain that the organizations we see as stalwarts of public life--schools, cultural organizations, arts organizations, social service providers--will continue to be vigorous. And if they do not, it is an open question whether our common lives will diminish with them.
There is a history here, one that we ought to attend to. As recently as the 1960s government played a relatively small role in supporting third places and non-governmental organizations. The public sphere was populated with some non-profits, but even more with a wide range of fraternal organizations, public houses, civic organizations, parks, etc. that received almost no federal funding and very limited state funding.
Most of those organizations have been in decline in the past 40 years. And they have been replaced by a new breed of non-profits, distinguished both by their focus on a single issue, and by their receipt, in one way or another, of tax funds. Those same non-profits have benefited from a more generous class of donors, some of whose gifts are driven by tax breaks. The receipt of tax funds, the specialization in particular issues, and the increased need to cultivate donors has led to a professionalized non-profit sector, one that employs a large number of people, and that relies on continually growing budgets to move its work forward.
The economic downturn and now, the thirst to cut government budgets, thus hurt the public sphere in three ways. They undermine government funding for non-profits even while donors are less able to give, they endanger the jobs of a large number of professionals, and because the public sphere and the non-profit sector have become almost synonymous, they threaten the infrastructure of public life.
Of course many non-profits will survive, but those that do will rely ever more heavily either on a dedicated tax funding stream (as, for example, providers of drug treatment who contract with governments), or upon the largesse of a small group of rich donors.
This is not entirely a bad thing in the long run. I expect that entrepreneurs will step in and find ways to run arts and culture organizations on a fee-for-service, not a donation basis. The internet will replace the work of many employees in the non-profit sector (it is noteworthy, for example, that civil society organizations have played almost no role in the recent rebellions on Tunisia and Egypt. Their organizational roles have been taken over by twitter and by ad-hoc on-the-ground decision-making). And small-scale volunteer work will return to churches and evanescent neighborhood organizations. But between now and the long run, the public sphere is in decline.
There is a history here, one that we ought to attend to. As recently as the 1960s government played a relatively small role in supporting third places and non-governmental organizations. The public sphere was populated with some non-profits, but even more with a wide range of fraternal organizations, public houses, civic organizations, parks, etc. that received almost no federal funding and very limited state funding.
Most of those organizations have been in decline in the past 40 years. And they have been replaced by a new breed of non-profits, distinguished both by their focus on a single issue, and by their receipt, in one way or another, of tax funds. Those same non-profits have benefited from a more generous class of donors, some of whose gifts are driven by tax breaks. The receipt of tax funds, the specialization in particular issues, and the increased need to cultivate donors has led to a professionalized non-profit sector, one that employs a large number of people, and that relies on continually growing budgets to move its work forward.
The economic downturn and now, the thirst to cut government budgets, thus hurt the public sphere in three ways. They undermine government funding for non-profits even while donors are less able to give, they endanger the jobs of a large number of professionals, and because the public sphere and the non-profit sector have become almost synonymous, they threaten the infrastructure of public life.
Of course many non-profits will survive, but those that do will rely ever more heavily either on a dedicated tax funding stream (as, for example, providers of drug treatment who contract with governments), or upon the largesse of a small group of rich donors.
This is not entirely a bad thing in the long run. I expect that entrepreneurs will step in and find ways to run arts and culture organizations on a fee-for-service, not a donation basis. The internet will replace the work of many employees in the non-profit sector (it is noteworthy, for example, that civil society organizations have played almost no role in the recent rebellions on Tunisia and Egypt. Their organizational roles have been taken over by twitter and by ad-hoc on-the-ground decision-making). And small-scale volunteer work will return to churches and evanescent neighborhood organizations. But between now and the long run, the public sphere is in decline.
Sunday, February 13, 2011
Is there any hope for public higher education?
I work at a private, non-profit college. I have many good friends, though, who work in the state system of higher education in Utah. And I know many other people working in public higher ed around the US. Most of my friends in private higher ed say they would not move to a public college or university. And most of my friends in public higher ed feel a certain measure of despair about the future.
Why is this? Because states like Nevada and Utah face a series of tensions--between limited capacity and increased demand, between budget cuts and increased costs, between a desire to respond to the market and the desire to maintain academic tradition, between higher ed leaders wanting autonomy and legislators wanting control, between traditional models of education and shifting student approaches to learning, between a public mission (serve the common good, strengthen civic skills, etc.) and a private mission (get good jobs for students), between declining quality and increased desire for prestige--none of which is easily resolved.
(I should say that private higher ed faces its own challenges, most of which rotate around cost, access, and quality.)
One can hardly imagine a "solution" to this set of problems (though there are plenty of attempts at it)--some single thing that if implemented would provide better, more responsive, less costly, more accessible education to a greater number of students.
But the fact that there is no solution points at least to a set of changes in the tone, organization, and goals of higher ed. We would be well-served by a greater humility, a willingness to be honest with the public about the challenges facing higher education. We would be well-served by legislators and leaders letting go of control, rather than increasing it.
There is increasing evidence that complex problems require small, distributed responses and clear feedback on the effect of the response. This is the insight of markets, ecosystems, and democracies. It should be the insight of educational leaders as well.
Why is this? Because states like Nevada and Utah face a series of tensions--between limited capacity and increased demand, between budget cuts and increased costs, between a desire to respond to the market and the desire to maintain academic tradition, between higher ed leaders wanting autonomy and legislators wanting control, between traditional models of education and shifting student approaches to learning, between a public mission (serve the common good, strengthen civic skills, etc.) and a private mission (get good jobs for students), between declining quality and increased desire for prestige--none of which is easily resolved.
(I should say that private higher ed faces its own challenges, most of which rotate around cost, access, and quality.)
One can hardly imagine a "solution" to this set of problems (though there are plenty of attempts at it)--some single thing that if implemented would provide better, more responsive, less costly, more accessible education to a greater number of students.
But the fact that there is no solution points at least to a set of changes in the tone, organization, and goals of higher ed. We would be well-served by a greater humility, a willingness to be honest with the public about the challenges facing higher education. We would be well-served by legislators and leaders letting go of control, rather than increasing it.
There is increasing evidence that complex problems require small, distributed responses and clear feedback on the effect of the response. This is the insight of markets, ecosystems, and democracies. It should be the insight of educational leaders as well.
Subscribe to:
Posts (Atom)
