NEAL CONAN, HOST:
This is TALK OF THE NATION. I'm Neal Conan in Washington. With the high cost of college education, many people wonder if we can find reliable ways to measure the value of that sheepskin and what students actually learn. In 2004, a number of colleges and universities started to administer what's called the Collegiate Learning Assessment, a standardized test of open-ended questions and situations to evaluate the grasp of critical thinking, analytical reasoning and written communication skills.
Critics denounce the CLA as inaccurate and unfair. Advocates argue it's a good beginning and charge that colleges resist any test that may come up with embarrassing results. We want to hear from those of you in academia. Professors, students, grads, what's the best way to measure the value of your college education? 800-989-8255 is the phone number. Email us, talk@npr.org. You can also join the conversation on our website at npr.org. Just click on TALK OF THE NATION.
Later in the program, Dexter Filkins of The New Yorker with questions about the political culture in America's crucial Middle East ally, Turkey. But first, holding colleges accountable. As mentioned, many institutions administer the College Learning Assessment to a sample of freshmen and seniors every spring and fall. The CLA was created by the Council for Aid In Education. Jeffrey Steedle is the measurement scientist for the council. He joins us from our bureau in New York. Nice to have you with us today.
JEFFREY STEEDLE: Thanks, Neal, it's great to be here.
CONAN: And the CLA made news recently when results of some institutions showed little difference between freshmen and seniors, which suggested to some that four years of college education did not make a lot of difference.
STEEDLE: That's true, Neal, though I think it's important to recognize that the generalizability of those results are limited by what the CLA measures. The CLA is often set up by its critics as a straw man in the sense that they say the CLA is supposed to measure all aspects, all important outcomes of higher education, when in fact the CLA is really focused on a small slice of the academic outcomes of higher education, these critical thinking and written communication skills.
CONAN: And so obviously it's not testing for organic chemistry or mathematics.
STEEDLE: Exactly. These are the sorts of skills that most universities uphold as being very important. They state this in their general learning outcome statements, and these are goals that they set for all of their students across all academic domains.
CONAN: So how are schools using it?
STEEDLE: Schools are using the test really as a way to help them improve teaching and learning. This isn't the accountability systems that we see in K-12 education, where it's being imposed upon them by the federal government. There are no high stakes involved with these test results.
What we have are schools who are electing to administer this test, and they're free to share their results publicly if they so choose. But otherwise, they're kept private, and they use these results to get a sense of whether or not their students are gaining as much as similar students at other institutions on these important skills like critical thinking and written communication.
CONAN: And there, as mentioned, have been a couple of, I guess, embarrassing results.
STEEDLE: You could consider them embarrassing. I suppose they've - some schools are trying to explain them away, as we've seen. But really the focus here for the CLA is on taking that result and making some changes in their educational programs, their general ed programs, that could potentially raise those scores.
CONAN: And there are some of your advocates who say hooray, finally a way to assess whether this college or university is better than that one.
STEEDLE: Well, we at the Council for Aid Education, we don't directly support the use of these scores for ranking institutions. When a school gets these scores, they do get a sense of where they sit relative to other institutions that admit students of similar entry and academic abilities, but they don't know who those other institutions are. They don't know that they're better than this state college or that university.
CONAN: But they may know they're 73rd on the list, which might suggest room for improvement.
STEEDLE: They may know their percentile rank, and right, that gives them a sense of their relative standing, and it may be a call to arms, as it is for many of the schools who administer the CLA, to make some changes to, for example, increase the requirements for writing for their first-year students or introduce courses that focus specifically on critical thinking.
CONAN: There's a long list of colleges that administer the College Learning Assessment, but many have also decided against using that test. Dan Berrett, is senior reporter for the Chronicle of Higher Education. He's been covering this story. He's kind enough to join us today here in Studio 3A. Nice to have you with us.
DAN BERRETT: Thank you, Neal.
CONAN: And there are advocates who say, yes, finally a good way to begin an objective measurement, and other people are saying this is completely ridiculous.
BERRETT: Well, I think there are people who support the idea of this, and they say, as was alluded to before, we have to do this or else the federal government will do it instead.
CONAN: For us, yeah.
BERRETT: For us instead. So I think a lot of people have seized on the CLA as a great way to do what they call an external referent, something that you can compare like colleges or even unlike colleges to each other. But there are a number of institutions that aren't terribly fond of it. Prestigious universities, in particular the Ivy Leagues, don't tend to rush toward this, and a skeptic may say, well, it's because their brand is...
CONAN: Is pretty good, and they don't want to besmirch it.
BERRETT: Right, and - but they might launch the counter-argument, which there may be some merit to, that their students already enter college with well-developed critical thinking skills. And what this does is it's a value-added measure. So it takes where you are freshman year, where you are senior year, and sometimes those are two different groups of students, which is another criticism, and says, you know, how much did you grow.
And if you're already coming in near the top of the chart there, how much can you go up?
CONAN: Is it fair to say some schools are concerned that the results will somehow end up being leaked to U.S. News & World Report?
BERRETT: Yeah, or us, or, you know, or the media, absolutely. And as nuanced as we try to be, journalists like numbers, and we like rankings, and it's a very tempting thing to take a simple number that quantifies one thing and try to make this stand in for - that's - for something much larger.
CONAN: Jeffrey Steedle, do you understand the critics' concerns that this number might be misused?
STEEDLE: I do absolutely, and I would want to address Dan's comments about the possible - what we would call a ceiling effect, where a Harvard or - we saw recently a University of Texas might suggest that, well, the reason we don't see large gains is that our students are already coming in toward the top.
But indeed in our data, we just don't find evidence consistent with the fact that there's a ceiling effect. We see a normal distribution of the universities, even those - and among those selective university, the average gains at those schools are very similar to the average gains at less selective universities.
And if it was the case that there was a ceiling effect, we wouldn't see that. We would see smaller average gains at those more selective schools than we do at the less selective schools.
CONAN: So what do you conclude from that?
STEEDLE: We conclude that it's not fair to explain away your scores by claiming that there's a ceiling effect when in fact there's no statistical evidence to support that.
CONAN: And is this, Dan Berrett of the Chronicle of Higher Education, is this, as some critics fear, you say we should do it or the federal government - the federal government provides a lot of money for higher education. They have every - just as much right to do it for higher education as they do for K-12.
BERRETT: Sure, sure...
CONAN: And is the thin end of the No Child Left Behind wedge?
BERRETT: Well, that's certainly the fear. And I think there's probably something to that. And to the CAE's credit, and to the CLA's credit, they have been very explicit at saying we don't want to do that. And it's also worth pointing out that the CLA is one of the number of tests that are used.
And there's a National Survey of Student Engagement, which tries to get at the culture and the things that are in place on campus that help students learn. And I think folks at the CLA would say that a number of measurements would really work well together.
CONAN: And the fact is there no accepted measurement, and there's great debate over any way to say how well this college prepares its students for life in the real world, as they like to say.
STEEDLE: Well, absolutely, and we just did a project called the College Completion Project, looking at a very straightforward metric, graduation rates, and even that, which would seem to be really simple on its face, is far from it.
CONAN: There are those who say, well, take the number of people who gets jobs or how much they make, five or 10 or 15 years after graduation. That might be a good metric.
STEEDLE: It might be, but it also is dependent on a lot of things that are beyond the school's control.
CONAN: We want to hear from those of you who have a stake in academia. Professors, students, graduates, how do you value your college education? 800-989-8255. Email us, talk@npr.org. And let's start with Joseph, Joseph with us on the line from Seattle.
JOSEPH: Oh, hi. Hey, I want to bring in something to the debate because I have been a teacher, physics and math professor, before I went to med school. Because I had my education from the British system, which was in those days, during the colonial days, and before the first (unintelligible) case, actually I went to Ghana, they had a system that was actually (unintelligible) was set up by the four-year institutions in U.K., which had a syllabus that covered all the Commonwealth countries, including New Zealand, Canada, Australia.
And the standard was excellent. I don't know why we cannot use some of those standards to do a better measurement for our system here.
CONAN: Well, Joseph, that might be a good suggestion, but it seems to me that you value greatly your college education.
JOSEPH: Well, I do, and so is the American system, I do. But the fact that somebody is arguing there are too many schools, and we can't have a standard, how could they do it...
CONAN: You're getting, I think, confused because you're listening to the radio, and you're on it, on a 10-second delay. But Joseph, thanks very much for the phone call, we appreciate it. Syllabus is one thing. Finding a way to measure how well something does, that, Dan Berrett, is very elusive.
BERRETT: It is, and faculty in the past have said, well, we measure it all the time, it's called grading at the end of the semester. Now, I think they have started to move away from that and to acknowledge that because of external pressures, we need to do something that is more standardized. But still, there are a lot of devils in the details.
CONAN: And Jeffrey Steedle, are you going ahead with something that could be more acceptable or a wider range of tests, to have a math test or a science test as well?
STEEDLE: We're not at present developing anything like that for higher education, although I think there's ample opportunities for schools to develop homegrown assessments of the things that they think are very important for math majors or for history or for psychology majors, for example.
CONAN: We're talking about ways to hold colleges accountable and measure how much people have learned in their four years or five years or six years. Professors, students, grads, what's the best way to measure the value of your college education? Give us a call, 800-989-8255. Email us, talk@npr.org. When we come back, we'll also talk with Clarence Page, the syndicated columnist, who's got another idea for a test of another sort, a college GED. Stay with us. This is NPR News.
(SOUNDBITE OF MUSIC)
CONAN: This is TALK OF THE NATION. I'm Neal Conan from NPR News. If you're a student in grade school, middle school or high school, you probably take standardized tests. They measure what you've learned and how schools compare from city to city, state to state.
A growing number of universities now test their students too, with an essay exam designed to measure critical thinking, communication and reasoning skills. We're talking today about the debate over whether that's a good step toward holding colleges accountable or a lousy way to measure the results of four years of a college education.
We want to hear from those of you in academia. Professors, students, grads, what's the best way to measure the value of your college education? Tell us your story, 800-989-8255. Email is talk@npr.org. You can also join the conversation on our website. That's at npr.org. Click on TALK OF THE NATION.
Our guests: Jeffrey Steedle, measurement scientist at the Council for Aid in Education, the organization that created the Collegiate Learning Assessment; and Dan Berrett, the senior reporter at The Chronicle of Higher Education. This email from Josh(ph) in Coralville, Ohio: Even if you give standardized tests to college students, how can you determine the accountability of the results? A lot of kids skip classes and party more than anything - more than learn anything. Who's to determine who's to blame for poor results, the classes and school, or is it the students?
And I'm - Jeffrey Steedle, I'm not sure how you can design anything that can measure that.
STEEDLE: That's fair enough, Neal, and you're right. I don't have that information. We don't collect that information about students' study habits or social habits. But you're aware of this report that came out last year, "Academically Adrift," by Richard Arum and Josipa Roksa. And they did collect some of that information, and they did have a sense that students' study habits and their social habits do - are associated with gains on the Collegiate Learning Assessment.
CONAN: And Dan Berrett, it's impossible to say the culture at any given institution does not contribute to that.
BERRETT: Oh sure, absolutely, and to piggy-back on what Jeffrey said, one of the biggest predictors was how much students had to read and write, that if they were asked to read and write more, they did better, which seems pretty straightforward. But there are very wide ranges and a wide variance also between disciplines and majors, sometimes at the same school.
CONAN: Email from Dave in Tempe, Arizona: I teach at ASU. A couple of years ago, Arizona State University was trying to get all our seniors to take the CLA. And what I heard back from those who did was that they didn't try their best, as we often had to offer extra credit in the class to get them to do it in the first place. What's the best practice for dealing with this student motivation issue? Jeffrey Steedle, is that something you've come across?
STEEDLE: Certainly, and in fact, it's something that I've carried out some research on and published. It is a persistent concern on these low-stakes tests. So these are tests that the students volunteer to take or they are incentivized to take, and whether they do well or not might not have any consequences for those individual students, and that could potentially raise some questions about the validity of those results.
What we see in the research is that that motivation or possible lack thereof is pretty evenly distributed across the schools. And so it has almost an identical effect on the results for all schools. And so the including motivation in the statistical analyses really doesn't have an effect on the relative standing of schools, so it wouldn't have any notable impact on these scores that come back out, at least these norm reference scores.
CONAN: And Dan Berrett, somebody might say, well, the way to solve that problem is make it a high-stakes test, and people will pay attention then.
BERRETT: Right, and then we're going right down the path of NCLB, although it's interesting to point out the Council of Independent Colleges ran a test using CLA where about 47 member colleges participated. And one of the strange things they found was that students wanted to know what their scores were so that they could use it as a credential.
CONAN: Hmm. Well, then they're going to have a higher stake on it, aren't they?
BERRETT: Yeah.
CONAN: Yeah. Victoria writes us from Perrysville - Perrysburg, excuse me, Ohio: When I was in college, I was a tuba performance major. I can't imagine a standardized test that would be able to adequately evaluate just how good a tuba player I learned to be. What would they do, have me play excerpts of "Ride of Valkyries" for an expert panel? Yes, they probably would.
(SOUNDBITE OF LAUGHTER)
CONAN: Let's bring another idea into the conversation and perhaps another idea for testing. In a column last week, Chicago Tribune columnist Clarence Page argued in favor of a different kind of college test, a GED for people who don't have a college degree. He's with us here in Studio 3A. Clarence, always a pleasure to have you on the program.
CLARENCE PAGE: Glad to be back, Neal.
CONAN: And there are tests that could declare someone competent or collegiately competent?
PAGE: Well, you know, let's not too ask too much of the tests, but let's also not ask too little. This discussion began with the fact that these tests are designed specifically to measure critical thinking and communication skills, regardless of whether you're an engineering major, a journalism major or a tuba major. You're going to need critical thinking.
You know, why do you go to college? That's what we're really talking about here. And that test is designed to look at what's the added value of going to four years of college or two years of college.
CONAN: Considering that you're going to be paying quite a bit for it.
PAGE: Yeah, and that's why this question about the ceiling effect is an important one. I for one, Neal, as you may remember, decided back in high school I wanted to be a journalist. My little town in Middletown, Ohio, luckily I had a teacher, Mrs. Mary Kindle(ph), an English teacher, who loved journalism, pushed that school to have a student paper, came out once every two weeks. Very few high schools have that.
And she was the one that first put the bug in my ear to be a journalist, and you know, you can blame her, at least half, if you don't like any of my columns now. But I learned so much in high school - everything I know about journalism I learned in high school, really, before I even went to Ohio University, an excellent journalism school.
And there, while studying economics, I met Richard Vedder(ph), who comes back into this story because then, as Richard Vedder, he's a blogger for the Chronicle of Higher Education. He's also the head of the Center for College Affordability. And he has spent years looking at this question of what value does college add?
College costs have been rising at a much faster pace than other assets in our economy, and they have the least explanation for it, except to say we're colleges, we're universities, we are valuable, and no question about that. But are there alternatives?
And nowadays, with the college costs rising the way they are, amidst all the debates, with the fact that education beyond high school is more essential than ever, a topic in itself worth another show...
CONAN: Well, one of the things we keep hearing in this aftermath of the recession, whatever it is, a slow recovery, is that the one defining factor that can identify somebody who's going to do better in this economy than somebody else, is a college degree.
PAGE: Well, something beyond high school. It can be two years of college. It can be a technical school or something. I got started on this because Rick Santorum raised the issue, calling Barack Obama a snob for pushing college education, even though Santorum's got three degrees, you know.
But the fact is that it is true: Everybody should get more than high school. So getting back to this question, Richard Vedder raised the issue, and he's not the only one, about why not have a college GED. How many people do we know who never finished college but are very bright people, very hard-working and just never got that credential, whether, you know, for personal economic reasons or for whatever.
But if you're an employer, and you're looking for people who have got the kind of a hard-work attitude, the kind of ability to learn, the enthusiasm you're looking for, but they just haven't got the sheepskin, maybe you'd like to hire them, and you could start them at a lower starting salary, say, than like a four-year Harvard degree-carrying person or whatever.
But in any case, why not have a college GED? I threw this idea out there, Neal, I got very little negative mail in response. I got two emails from people saying...
CONAN: So unusual for your columns, yes.
PAGE: So unusual, yes. You know, I always wonder what I've done wrong if I don't get much - I got a lot of emails from people saying, hey, that's me you're describing. You know, I've got a wife and three kids. I couldn't finish such-and-such university, but I'd be - you know, I'd be happy to take any test you want to give me in order to show you I'm qualified...
CONAN: Let's ask our testing professional here. Jeffrey Steedle, measurement scientist at the Council for Aid in Education, are such tests feasible? Are such tests - do they exist?
STEEDLE: I'd have to see the test to judge it myself, but Clarence, I really - I wonder what sorts of skills, what would be the construct being measured by such tests?
PAGE: Well, there's actually - in fact the Educational Testing Service, which does the SAT with the College Board, has been working on this. They recently entered a deal with Straighter Line, that's one online education company. There are many others out there. MIT has been looking into the possibility of providing a competency test online.
Other schools have been looking into this. Stanford is talking about it. We'd be talking about the full range. The very questions that you're asking are the questions that they're asking. How do you test it?
CONAN: Yeah, go ahead.
STEEDLE: Oh, I was just going to say, Clarence, actually Straighter Line has a deal with the Council for Aid to Education as well. They'd like to be using some of our tests to provide some certification for their students.
PAGE: Right, so this is not a new idea, and it's been percolating out there. And the question is, I've gotten mail from people who took a similar test in the Air Force, to test their college competency, and they said they got their career started thanks to this Air Force test because they never finished college. Because they could wave this certificate, they had some documentation as to what their knowledge and abilities are.
And so this is something that - it's not a question of whether it should be done. It seems to me it's a question of how do you do it. And those questions can be answered. I did get, as I say, a couple of emails from people saying, well, you need to go to college in order to learn critical thinking and communication skills, right? Well, what do we talk about here today? The fact that kids are coming out of colleges, high percentages without great critical thinking.
CONAN: ...increased.
PAGE: Let's say - yeah, yeah. No measurable increase in their - or very little measurable increase. I got a great letter from a retired University of Chicago professor, who I just talked to again today before the show. And she was relating to me a couple of students she had at University of Chicago, no less. We're talking about seniors in biological psychology who complained to her when she handed them a take-home assignment for two weeks: an article of a medical journal.
Evaluate this article, the sourcing, the research methods, the logic. Do not bring me a summarization in all caps. Do not just bring me a summary of it. Let me know what you think about the sources and the researching, the argument, et cetera. She had one student who was very angry, indignant with her, who was a, you know, a good straight-A, straight-B student, who had gone through high school and college with a very good performance, but who was not a very good writer and was complaining that the teacher wanted her to think, said, nobody ever required me to think before.
(SOUNDBITE OF LAUGHTER)
PAGE: She put it in a nutshell. Yeah. You know, are we asking our students to really think or just answer multiple choice questions?
CONAN: Let's get a caller in. This is Jeffrey(ph), Jeffrey with us from Spring Branch in Texas.
JEFFREY: Hello, Neal.
CONAN: Hi. Go ahead, please.
JEFFREY: I'm a 21-year-old college student. I lost my business a while ago, and so I'm back in school. And I would love to take some tests so I could get my bachelor's in history and be a teacher.
CONAN: And be a teacher. So, to your knowledge, does such a test exist?
JEFFREY: I don't believe there are any tests. I just heard you guys talking about what kind of test could it be? You know, there's final exams that all college students take every time for their class. I'd be happy to just take a whole bunch of those one weekend.
CONAN: So, cram for a couple of weeks and...
JEFFREY: I could use some kind of GED type of college equivalency test.
CONAN: Let me just ask you, Dan Berrett of The Chronicle for Higher Education: Is there a demand for this? Is - we've seen so many new kinds of educational alternatives between, obviously, community colleges and online colleges. Is there demand for this?
BERRETT: Oh, there's certainly demand. You mentioned two big ones. And also, Clarence talked about a reference to MITX, which is another one. There are a lot of folks who would really love to speed things up. There's also three-year degrees, this notion that you should try to fit it all in as quickly as possible.
CONAN: Skip that summer break.
BERRETT: Well, yeah.
CONAN: Yeah. We're talking with Dan Berrett of The Chronicle of Higher Education, senior reporter there. Jeffrey Steedle is with us as well, measurement scientist at the Council for Aid in Education, the organization that created the Collegiate Learning Assessment. Also with us, Clarence Page, syndicated columnist for the Chicago Tribune. You're listening to TALK OF THE NATION from NPR News.
And here's some emails about the CLA: As a student, writes Gabriela(ph) in Arizona, I don't believe standardized tests like the CLA are a clear reflection of the progress students make in college. The CLA often has very broad topics that don't truly measure the specific content of classes within the major or specialization of students. I think that's been addressed before. I've taken the CLA, she wrote, and I find it's just a pain to schedule and take. Now, ASU offers an incentive of $10 to take the exam, and a $100 incentive for those who excel. That might be a good way to increase participation, but not the best way to measure the achievement rates of students across the university and its numerous disciplines.
And this from Kathleen(ph): My university took part in the CLA for several years because my course was dominated by first-time freshmen. It was volunteered to take part. I was puzzled by what I observed. Some students had quite a number of questions to answer and took almost 40 minutes to complete. Most students had far fewer questions and took 10 to 15 minutes. How could have anything of value come from something that took so little time?
Jeffrey Steedle?
STEEDLE: Actually, students who sit for the CLA, they do one of two types of tasks. Some students - about half of students will do what's called a performance task, and this is what the CLA is probably best known for. It's where the students are presented with some sort of real-world scenario and they're given a library of relevant documents, and they're asked to analyze and synthesize the information they see in these documents. And they will answer a series of questions that lead them through some analysis and also get them to make a decision to solve that problem, to propose a course of action and to potentially suggest additional research that would address some unanswered questions.
Other students who sit for the CLA do what we call an analytic writing task, which has them address a prompt, which is probably more familiar, where they're given a single-sentence prompt and asked to respond to it, to agree or disagree with it and to address counterarguments. And those students are also then asked a shorter task, where they're presented with an argument that someone else has created - maybe about six to eight sentences long and they're asked specifically to critique that argument and to identify the problems with that argument.
And so that would explain why some students are seeing more questions than others. As for the amount of time students spend on it, they're allotted 90 minutes, but I can't stop them from leaving it blank, for example, or not answering the question and leaving quickly.
CONAN: All right. Here's an email from Michael(ph) in Palm Beach: I was listening in my car on the way to the office, so I won't hear you mention this if you do. That said, I hold a doctorate in higher education. After 21-plus years of formal education, I value my higher education because, one, earning power. The U.S. Department of Labor publishes stats that consistently show one's earning power goes up with one's level of formal education. Two, quality of life. To quote the president of Brown education some years back: Education changes lives. It sure has mine.
And, Clarence, that was a big part of your argument. We need to find ways to help those in the lowest economic quarter get higher degrees...
PAGE: Yes.
CONAN: ...and get those better jobs - and better quality of life.
PAGE: Yeah. How do we bridge that gap? There are so many I know - Professor Vedder, again, among others have done studies that - or just statistics that show how many of the jobs that used to call for a high school diploma are now being occupied by college degree holders. Society and economics, in all reality, has devalued a high school diploma. It is now the first step, really, toward that sheepskin or credential that's going to really get you into the work world. This is why I don't mind paying students $10 to take the test or $100 to get a higher score because I think that's part of the question of why you're in college in the first place. A lot of people are there just simply for their pure intellectual enhancement. But most of them are there to increase their income.
CONAN: Well, this email from Ken in Sturgis, Michigan exemplifies that. I'm a slow learner. Started college in 1970, got my bachelor's in 1998. But the day I finished my last course and notified my employer, I got immediate $3,000-per-year raise. The value of the degree is much more, of course, but that quantified the monetary value.
(SOUNDBITE OF LAUGHTER)
CONAN: So there's one answer anyway.
PAGE: There you go.
CONAN: Gentlemen, thank you very much for your time. Clarence Page of the Chicago Tribune, Dan Barrett of The Chronicle of Higher Education were with us in Studio 3A. Jeffrey Steedle of the Council for Aid in Education joined us from our bureau in New York. Transcript provided by NPR, Copyright NPR.