With a well-designed three-hour exam, it’s not about remembering a whole bunch of stuff. The basic elements that comprise a three hour exam can be remembered by the average student (and tested in half an hour). The time is used to allow the student to demonstrate a capacity to use the materials learned.
This capacity can be measured adequately in three hours. Adding more time – and the need to accumulate more resources – does not add to the evaluation of this ability. A longer exam typically involved having the person replicating the capacities they were demonstrating in a three hour exam.
A three hour exam takes place at a known and set time, for a specific interval, during school hours. A take-home exam, by contrast, extends into evenings and mornings. When I was studying, I would work as well, in order to support myself. A take-home exam would make me face the choice of skipping work or doing poorly on the exam.
My mentor at MIT, the late Gian-Carlo Rota, assigned grueling take-home problem sets (with open problems for extra credit!) but nonetheless used timed exams. Those exams were deliberately easy–they were intended as basic competency tests for the material. I see merit to his approach: there is basic material that you need to have handy in order to do advanced work. Much to his credit, he did not use timed exams to test people’s creativity under time pressure.
ps. Cross-posting this at John Cook’s blog, since I’m not sure where the discussion will evolve.
Jonsays:
“When I was studying, I would work as well, in order to support myself. A take-home exam would make me face the choice of skipping work or doing poorly on the exam.”
And this seems perfectly fine, deal with it. There is no need to bring the class down to the level of the lowest common denominator.
Parandsays:
A take home exam is equivalent to a very heavily weighted home work assignment with a tightly defined time commitment. I’m not a big fan. If you’re looking for a way to measure the real world ability of students, why not do away with exams and base their grade on assignments throughout the course? You could align assignments such that they’re all building towards a unified goal (eg. build a compiler from scratch). That way it’s not a single assignment deciding the student’s fate.
The 3 hour (or 1 hour for that matter) exam is a different means of measuring a student’s ability. You could argue it should be given a different weight, perhaps a reduced weight, but doing away with it does take away a tool from your measuring toolbox. I’d keep both and play around with their relative weights.
Tomsays:
I think 8h (that’s what “24h” should really mean) is no more realistic in simulating real-world problem solving than 3h. But it’s a huge drain on student time, and it creates inequalities among students who may have, by chance, different obligations and time commitments during exam period. Furthermore, the rules governing university exams simply don’t allow it here.
I view exams as random samples of the total knowledge a student has accumulated, and the total amount of work a student has performed, during the course. One way I incorporate testing the ability to do more complex work into shorter exams is to ask questions building on the results of homework problems.
Ragib Hasansays:
In the age of Google, can you really expect students not to cheat, and just google for the answers?
Sure, you can try setting new, interesting problems. But in many scenarios, problems can’t be new, and a clever student can find solutions via google. Of course, if web street smarts is the name of the course, that will be ok, but that’s often not the case.
Also, there ARE some scenarios/topics where you need a proctored, timed exam. My wife is a doctor, and all her licensing exams are fixed-time, on-site exams. To answer most of the questions, doctors have to know a lot of things … i.e. memorize them. If it were a take home exam, a non-doctor with a good webbrowser could answer many of the questions. Same applies to law scenarios.
As for CS, you can have a take home exam given you can design interesting, thought provoking problems, but how are you going to prevent cheating? You can either have a common problem set for all, or design individual problems. If the problemset is common, then all the students need to do is to form a study group to solve them as a group. Making individual problems is completely impractical and non-scalable. You can’t just claim student honor code will prevent cheating … and you can’t catch every cheat by running plagiarism-detecting software. If the questions are common to all, … only 1 student needs to solve them — the others can just show their talent in paraphrasing/reorganizing the answer.
Interesting. I have taken lots of exams like you. I think the best “exams” are those which are open to public srutiny, like posting on blogs, forums, developing wikis, working on research, teaching… that is more rewarding and could contribute to the “world” of digital sphere. The best problems are not simulated ones, but real life ones. Those problems could challenge our wisdom, our capacity to respond to the complexity of life and business and personal issues or problems.
So, getting an A+ from a 3 hr exam will only show that one is smart, but getting an A+ attitude towards life and “exams” as mentioned above will show that one is wise. Would I do another 3 hour exam or the “life exam”? Or would you like your students to take up the challenge of life exam? Like to learn…
Thanks for your interesting insights.
Should be public scrutiny. I just wonder why I always made such typo mistakes. Sorry.
Jon Dronsays:
@Ragib Hasen…
Who would you trust more:
a) a doctor who intelligently used every means at his or her disposal, including Google, Wikipedia and asking/arguing with other doctors, to find out what is wrong with you and how to treat it or
b) a doctor who had successfully memorised (say, to pick a figure) 90% of the conditions and treatments presented on a multiple choice quiz but who refused to use any other technologies or ask other people to back up his or her diagnosis?
Inauthentic assessments are unfair, stupid, lazy, systemically and virulently pathogenic to learning, and fail to tell us the most important things about what people know.
Of course the students should use Google, and Wikipedia, and Wolfram Alpha, and books, and papers, and calculators and computers and pens and paper and other students and experts and the family dog and whatever resources they can muster (as long as they keep it legal and cite fairly). It’s just plain common sense. How can it be cheating for them to do exactly what you or I would do when faced with the kind of problem that can be solved that way?
What is *much* stranger, and requires some serious justification, is the notion of arbitrarily taking away all those things. If you are taking away the tools of the trade, why not take away a pianist’s piano? (this is not as absurdly incredible or unbelievable as it sounds – I have the dubious pleasure of having to mark exams on computer programming written on paper, by students having no access to a computer!). To force us to work in a way that is antagonistic to learning, bears little resemblance to anything we will do in the future, and encourages us to think and act in ways that are harmful to ourselves and others is a cruel and unusual approach. Let’s stop it now.
Thanks for pointing out Jon Dron’s excellent article. Here’s a response I wrote.
I’d rather have the three-hour exam.
With a well-designed three-hour exam, it’s not about remembering a whole bunch of stuff. The basic elements that comprise a three hour exam can be remembered by the average student (and tested in half an hour). The time is used to allow the student to demonstrate a capacity to use the materials learned.
This capacity can be measured adequately in three hours. Adding more time – and the need to accumulate more resources – does not add to the evaluation of this ability. A longer exam typically involved having the person replicating the capacities they were demonstrating in a three hour exam.
A three hour exam takes place at a known and set time, for a specific interval, during school hours. A take-home exam, by contrast, extends into evenings and mornings. When I was studying, I would work as well, in order to support myself. A take-home exam would make me face the choice of skipping work or doing poorly on the exam.
My mentor at MIT, the late Gian-Carlo Rota, assigned grueling take-home problem sets (with open problems for extra credit!) but nonetheless used timed exams. Those exams were deliberately easy–they were intended as basic competency tests for the material. I see merit to his approach: there is basic material that you need to have handy in order to do advanced work. Much to his credit, he did not use timed exams to test people’s creativity under time pressure.
ps. Cross-posting this at John Cook’s blog, since I’m not sure where the discussion will evolve.
“When I was studying, I would work as well, in order to support myself. A take-home exam would make me face the choice of skipping work or doing poorly on the exam.”
And this seems perfectly fine, deal with it. There is no need to bring the class down to the level of the lowest common denominator.
A take home exam is equivalent to a very heavily weighted home work assignment with a tightly defined time commitment. I’m not a big fan. If you’re looking for a way to measure the real world ability of students, why not do away with exams and base their grade on assignments throughout the course? You could align assignments such that they’re all building towards a unified goal (eg. build a compiler from scratch). That way it’s not a single assignment deciding the student’s fate.
The 3 hour (or 1 hour for that matter) exam is a different means of measuring a student’s ability. You could argue it should be given a different weight, perhaps a reduced weight, but doing away with it does take away a tool from your measuring toolbox. I’d keep both and play around with their relative weights.
I think 8h (that’s what “24h” should really mean) is no more realistic in simulating real-world problem solving than 3h. But it’s a huge drain on student time, and it creates inequalities among students who may have, by chance, different obligations and time commitments during exam period. Furthermore, the rules governing university exams simply don’t allow it here.
I view exams as random samples of the total knowledge a student has accumulated, and the total amount of work a student has performed, during the course. One way I incorporate testing the ability to do more complex work into shorter exams is to ask questions building on the results of homework problems.
In the age of Google, can you really expect students not to cheat, and just google for the answers?
Sure, you can try setting new, interesting problems. But in many scenarios, problems can’t be new, and a clever student can find solutions via google. Of course, if web street smarts is the name of the course, that will be ok, but that’s often not the case.
Also, there ARE some scenarios/topics where you need a proctored, timed exam. My wife is a doctor, and all her licensing exams are fixed-time, on-site exams. To answer most of the questions, doctors have to know a lot of things … i.e. memorize them. If it were a take home exam, a non-doctor with a good webbrowser could answer many of the questions. Same applies to law scenarios.
As for CS, you can have a take home exam given you can design interesting, thought provoking problems, but how are you going to prevent cheating? You can either have a common problem set for all, or design individual problems. If the problemset is common, then all the students need to do is to form a study group to solve them as a group. Making individual problems is completely impractical and non-scalable. You can’t just claim student honor code will prevent cheating … and you can’t catch every cheat by running plagiarism-detecting software. If the questions are common to all, … only 1 student needs to solve them — the others can just show their talent in paraphrasing/reorganizing the answer.
Interesting. I have taken lots of exams like you. I think the best “exams” are those which are open to public srutiny, like posting on blogs, forums, developing wikis, working on research, teaching… that is more rewarding and could contribute to the “world” of digital sphere. The best problems are not simulated ones, but real life ones. Those problems could challenge our wisdom, our capacity to respond to the complexity of life and business and personal issues or problems.
So, getting an A+ from a 3 hr exam will only show that one is smart, but getting an A+ attitude towards life and “exams” as mentioned above will show that one is wise. Would I do another 3 hour exam or the “life exam”? Or would you like your students to take up the challenge of life exam? Like to learn…
Thanks for your interesting insights.
Should be public scrutiny. I just wonder why I always made such typo mistakes. Sorry.
@Ragib Hasen…
Who would you trust more:
a) a doctor who intelligently used every means at his or her disposal, including Google, Wikipedia and asking/arguing with other doctors, to find out what is wrong with you and how to treat it or
b) a doctor who had successfully memorised (say, to pick a figure) 90% of the conditions and treatments presented on a multiple choice quiz but who refused to use any other technologies or ask other people to back up his or her diagnosis?
Inauthentic assessments are unfair, stupid, lazy, systemically and virulently pathogenic to learning, and fail to tell us the most important things about what people know.
Of course the students should use Google, and Wikipedia, and Wolfram Alpha, and books, and papers, and calculators and computers and pens and paper and other students and experts and the family dog and whatever resources they can muster (as long as they keep it legal and cite fairly). It’s just plain common sense. How can it be cheating for them to do exactly what you or I would do when faced with the kind of problem that can be solved that way?
What is *much* stranger, and requires some serious justification, is the notion of arbitrarily taking away all those things. If you are taking away the tools of the trade, why not take away a pianist’s piano? (this is not as absurdly incredible or unbelievable as it sounds – I have the dubious pleasure of having to mark exams on computer programming written on paper, by students having no access to a computer!). To force us to work in a way that is antagonistic to learning, bears little resemblance to anything we will do in the future, and encourages us to think and act in ways that are harmful to ourselves and others is a cruel and unusual approach. Let’s stop it now.