I agree with your sentiments that research needs a coal-face but I think your conclusion is too strong.
I would argue that pure theory *is* an acceptable methodology for the unifying and exposition of work that has already been tested against the coal-face. This is essentially mathematics’ role for most of the physical sciences.
The (valid) examples you give of research gone awry are for when the generalisations and connections made by pure theory are not brought back to the coal-face for further scrutiny. That is, when the mathematics for the objects of study are confused for the objects of study themselves.
I agree with what you said. I work in Computer Vision and I come up with more problems than my guide just because I code, I experiment and I interact. I have been telling to my guide continuously that you cant be a good researcher unless you interact with the system. And whatever cool looking theories that you have in your mind are useless because they might not work[infact his theories never work in practice]. I would like to add to your post that solving only problems is not research. It is probably 20% part of it. Main part of research is actually coming up with a problem, formulating it, interacting with the system and breaking the system that you have built, or to think a problem from a completely different perspective.
I have done the toughest of course assignments in one night. But when I wanted to come up with a problem statement it took twenty days at masters level. And the first solution which I come up with forced me to look at the problem statement again.
It’s like if one of the part either the problem or the solution (I’ve seen people coming with the solution first and then the problem) is well formulated then things are easy. In theory I have generally seen that problems are very well formulated or it can be formulated very well with small amount of effort *in most of the cases*. Both theory and system go hand in hand.
JeffEsays:
You’re not describing “pure” theory; you’re describing BAD theory.
Jeremysays:
You do describe bad theory, when I think there are results of pure theory which are important in practice: I will give a few examples.
1) “Given a sorted array of n distinct elements, a search algorithm using only comparisons cannot perform less than log n operations in the worst case.”
This is pure theory because it was not implemented, nor was it confronted to reality by implementing all possible search algorithms and checking one by one if they can do better than log n in the worst case.
Yet it is a result useful in practice, saying that if your data is so large that log n is too much time, you need either to use other operations than comparisons, or use a finer analysis than the worst case complexity, trying to take advantage of the specificities of the search queries of your application.
2) A new algorithmic technique which theoretical advantage over previous solutions is valid only for instances larger than the number of atoms in the universe is not worth implementing, yet it is worth designing and publishing because it might pave the way for a more practical solution based on similar principles (the same argument hold for quantum computing).
3) Ultimately, solving a problem which does not and can not have any practical application is like solving a puzzle: the tools and techniques that you develop for solving this problem might help you (or others) to solve more practical (and often more complex) problems, for which it would have been harder to develop the technique. (Think about the Hanoi Tower Problem as a useless problem, yet the simplest puzzle possible to illustrate recursion and the power of constant space algorithms).
Nobody will contradict you on that theory which agrees with practice is the best. Yet it would be difficult to prove that a theoretical result has no use, direct or indirect, present or future.
Michael Mitzenmachersays:
I agree with Jeff. I’m not quite sure what your notion of pure theory is, though I’d agree that there’s bad theory out there.
Let’s take an extreme example of “pure theory” in algorithm design — the quantum algorithm for factoring. I don’t think Peter Shor planned on implementing it, but it sure is a nice result.
On the other hand, I do agree I’ve seen Algorithm Design work where people designed a “better” algorithm — where often “better” in context had a non-practical but mathematically useful definition, such as a better competitive ratio — that I didn’t think was particularly interesting. I classify that as “bad theory”, as Jeff says, not pure theory.
Vsays:
Sorry – Pure theory *is* acceptable research methodology.
I do not see the need to distinguish between mathematics and algorithm design. It is a perfectly fine way to do research to do theoretical work on algorithms, without ever trying to implement one. I would be glad to leave implementation to others who are more enthusiastic about laboring towards experimental insight.
HAsays:
Note: I do not have as much experience as you all have. I’m just about to complete my masters. You can even ignore my comment(s).
Every researcher has a long term research goal. Every researcher would want whatever they do is eventually used in practice or by other researchers. And eventually society should benefit either aesthetically or by living a life in a better way than they lived ever.
Apart from the effort in developing good theories effort should be made to make sure that the society benefits from the theory. If I develop algorithms and theories on quantum computing, then I must make every possible effort so that at the end of my life I get to see a quantum computer running my algorithms. If I’m not fit enough to make a quantum computer then I must (apart from developing theories and algorithms) help the physics and electrical researcher to get good funding (need not be for myself), I must encourage people to take PhDs in this area (either under the physics guy, other researchers who work in theory and myself) and I must encourage people to open start-ups on small/large applications on whatever the outcome has been till now. And simultaneously I should learn from them or from the system they developed on new kind of algorithms required and provide deep insights into the algorithms developed upto now. Both theory and system go hand in hand.
Definition of theory is different in different subjects. In physics, machine learning, computer vision, economics etc. *a good theory* is one which can stand the test of time. Empirical evaluation along with mathematical justification wherever possible are the only ways to put your theory forward. It also mean that you can develop new theories by interacting with the system over a period of time.
There is no use of the theory you build when you are not making any effort to make people use it ten years after you proposed or proved it. Chances are very rare that other people will put it to use. Author of the blog post is actually right, people do not make efforts.
@Jeremy,
The examples which you have provided are small parts of long term research goals, where the purpose is to help other researchers to stop wasting their time (by providing a lower bound) or by disseminating the information (tower of hanoi example). As I wrote earlier in this comment long term research goal is to finally help society or other researchers.
It’s not that you cant develop good theories without ever interacting with the system. But interacting with the system and the people using that system does make a lot of impact.
Ooh, what a great troll of a post! I am not, nor have I ever been, a pure theorist, even though I once flirted with the idea of devoting my life to complexity theory. But I think your criticism is better directed against researchers with bad taste in problems than in researchers who pursue “pure” theory.
As Jeremy pointed out, the n log n lower bound on sorting by comparisons is a purely theoretical results with very practical implications. Impossibility / intractability results often have implications for how we approach problems in optimization, cryptography, databases, systems, and even politics. And often theory results build on other theory results, which means that we shouldn’t be too quick to dismiss a theory contribution just because it has no immediate practical application.
But that doesn’t mean that all theory is worthwhile. Algorithms and complexity theory are littered with useless results whose only practical application was to help their authors get tenure. And “theory” results in less mathematical areas can be particularly irrelevant.
I do feel that practical application is a great measure for the value of a research contribution, at least in computer science. But I think we should take a long view and give researchers credit for a) long-term influence and b) pursuing the problems that seemed promising at the time.
Dansays:
Sure there is some bad theory out there but the hate doesn’t seem to end there. So in addition to the excellent comments above i just want to add this: There is nothing wrong with working on pure theory problems, without any concerns about having practical impact, to satisfy your need of problem solving. If you feel like what you are doing has to have value in practice to justify your existence, that’s your problem.
Yeah, continuum from pure theory to pure experiment. Here’s a quote from Phil Johnson-Laird. (Read the bit about programs as theory: he’s talking about research in the cognitive sciences.)
“Experiments provide information about correspondence with the facts, but they exert a dangerous pull in the direction of empirical pedantry, where the only things that count are facts, no matter how limited their purview. Computer programs provide information about the coherence of a set of assumptions, but they exert a dangerous pull in the direction of systematic delusion, where all that counts is internal consistency, no matter how remote from reality. Give up one approach and you turn into a Gradgrind, the teacher in Dicken’s novel Hard Times, whose only concern is with the facts; give up the other and you become an architect for the Flat Earth Society.”
So maybe pure experiment isn’t a good idea either?
Daniel Reisersays:
I would add theoretical architects to your list of those whose work can’t hold up to the harsh light of reality. Architects like Tom Wiscombe (http://www.emergentarchitecture.com/) do beautiful, complex renderings of projects that would be difficult or impossible to build.
There are countless others, and architecture students eat up their work. The problem is that most students are going to work in a corporate firm for a couple of years, drawing hospital bathroom plans in CAD, before they get to design anything.
I wish more theoretical architects would put their ideas to the test instead of working in a digital simulacra and hoping that construction technology will eventually catch up with their genius ideas . Real writers publish, real artists make, and real architects build.
i believe that practical is more essential than theory because theory only teaches about how to0 do but practical thaches to use it so only by reading is nothing
I agree with your sentiments that research needs a coal-face but I think your conclusion is too strong.
I would argue that pure theory *is* an acceptable methodology for the unifying and exposition of work that has already been tested against the coal-face. This is essentially mathematics’ role for most of the physical sciences.
The (valid) examples you give of research gone awry are for when the generalisations and connections made by pure theory are not brought back to the coal-face for further scrutiny. That is, when the mathematics for the objects of study are confused for the objects of study themselves.
“Theory guides; Experiment decides.”
[I.M. Kolthoff]
Very true HA.
I agree with what you said. I work in Computer Vision and I come up with more problems than my guide just because I code, I experiment and I interact. I have been telling to my guide continuously that you cant be a good researcher unless you interact with the system. And whatever cool looking theories that you have in your mind are useless because they might not work[infact his theories never work in practice]. I would like to add to your post that solving only problems is not research. It is probably 20% part of it. Main part of research is actually coming up with a problem, formulating it, interacting with the system and breaking the system that you have built, or to think a problem from a completely different perspective.
I have done the toughest of course assignments in one night. But when I wanted to come up with a problem statement it took twenty days at masters level. And the first solution which I come up with forced me to look at the problem statement again.
It’s like if one of the part either the problem or the solution (I’ve seen people coming with the solution first and then the problem) is well formulated then things are easy. In theory I have generally seen that problems are very well formulated or it can be formulated very well with small amount of effort *in most of the cases*. Both theory and system go hand in hand.
You’re not describing “pure” theory; you’re describing BAD theory.
You do describe bad theory, when I think there are results of pure theory which are important in practice: I will give a few examples.
1) “Given a sorted array of n distinct elements, a search algorithm using only comparisons cannot perform less than log n operations in the worst case.”
This is pure theory because it was not implemented, nor was it confronted to reality by implementing all possible search algorithms and checking one by one if they can do better than log n in the worst case.
Yet it is a result useful in practice, saying that if your data is so large that log n is too much time, you need either to use other operations than comparisons, or use a finer analysis than the worst case complexity, trying to take advantage of the specificities of the search queries of your application.
2) A new algorithmic technique which theoretical advantage over previous solutions is valid only for instances larger than the number of atoms in the universe is not worth implementing, yet it is worth designing and publishing because it might pave the way for a more practical solution based on similar principles (the same argument hold for quantum computing).
3) Ultimately, solving a problem which does not and can not have any practical application is like solving a puzzle: the tools and techniques that you develop for solving this problem might help you (or others) to solve more practical (and often more complex) problems, for which it would have been harder to develop the technique. (Think about the Hanoi Tower Problem as a useless problem, yet the simplest puzzle possible to illustrate recursion and the power of constant space algorithms).
Nobody will contradict you on that theory which agrees with practice is the best. Yet it would be difficult to prove that a theoretical result has no use, direct or indirect, present or future.
I agree with Jeff. I’m not quite sure what your notion of pure theory is, though I’d agree that there’s bad theory out there.
Let’s take an extreme example of “pure theory” in algorithm design — the quantum algorithm for factoring. I don’t think Peter Shor planned on implementing it, but it sure is a nice result.
On the other hand, I do agree I’ve seen Algorithm Design work where people designed a “better” algorithm — where often “better” in context had a non-practical but mathematically useful definition, such as a better competitive ratio — that I didn’t think was particularly interesting. I classify that as “bad theory”, as Jeff says, not pure theory.
Sorry – Pure theory *is* acceptable research methodology.
I do not see the need to distinguish between mathematics and algorithm design. It is a perfectly fine way to do research to do theoretical work on algorithms, without ever trying to implement one. I would be glad to leave implementation to others who are more enthusiastic about laboring towards experimental insight.
Note: I do not have as much experience as you all have. I’m just about to complete my masters. You can even ignore my comment(s).
Every researcher has a long term research goal. Every researcher would want whatever they do is eventually used in practice or by other researchers. And eventually society should benefit either aesthetically or by living a life in a better way than they lived ever.
Apart from the effort in developing good theories effort should be made to make sure that the society benefits from the theory. If I develop algorithms and theories on quantum computing, then I must make every possible effort so that at the end of my life I get to see a quantum computer running my algorithms. If I’m not fit enough to make a quantum computer then I must (apart from developing theories and algorithms) help the physics and electrical researcher to get good funding (need not be for myself), I must encourage people to take PhDs in this area (either under the physics guy, other researchers who work in theory and myself) and I must encourage people to open start-ups on small/large applications on whatever the outcome has been till now. And simultaneously I should learn from them or from the system they developed on new kind of algorithms required and provide deep insights into the algorithms developed upto now. Both theory and system go hand in hand.
Definition of theory is different in different subjects. In physics, machine learning, computer vision, economics etc. *a good theory* is one which can stand the test of time. Empirical evaluation along with mathematical justification wherever possible are the only ways to put your theory forward. It also mean that you can develop new theories by interacting with the system over a period of time.
There is no use of the theory you build when you are not making any effort to make people use it ten years after you proposed or proved it. Chances are very rare that other people will put it to use. Author of the blog post is actually right, people do not make efforts.
@Jeremy,
The examples which you have provided are small parts of long term research goals, where the purpose is to help other researchers to stop wasting their time (by providing a lower bound) or by disseminating the information (tower of hanoi example). As I wrote earlier in this comment long term research goal is to finally help society or other researchers.
It’s not that you cant develop good theories without ever interacting with the system. But interacting with the system and the people using that system does make a lot of impact.
Ooh, what a great troll of a post! I am not, nor have I ever been, a pure theorist, even though I once flirted with the idea of devoting my life to complexity theory. But I think your criticism is better directed against researchers with bad taste in problems than in researchers who pursue “pure” theory.
As Jeremy pointed out, the n log n lower bound on sorting by comparisons is a purely theoretical results with very practical implications. Impossibility / intractability results often have implications for how we approach problems in optimization, cryptography, databases, systems, and even politics. And often theory results build on other theory results, which means that we shouldn’t be too quick to dismiss a theory contribution just because it has no immediate practical application.
But that doesn’t mean that all theory is worthwhile. Algorithms and complexity theory are littered with useless results whose only practical application was to help their authors get tenure. And “theory” results in less mathematical areas can be particularly irrelevant.
I do feel that practical application is a great measure for the value of a research contribution, at least in computer science. But I think we should take a long view and give researchers credit for a) long-term influence and b) pursuing the problems that seemed promising at the time.
Sure there is some bad theory out there but the hate doesn’t seem to end there. So in addition to the excellent comments above i just want to add this: There is nothing wrong with working on pure theory problems, without any concerns about having practical impact, to satisfy your need of problem solving. If you feel like what you are doing has to have value in practice to justify your existence, that’s your problem.
Yeah, continuum from pure theory to pure experiment. Here’s a quote from Phil Johnson-Laird. (Read the bit about programs as theory: he’s talking about research in the cognitive sciences.)
“Experiments provide information about correspondence with the facts, but they exert a dangerous pull in the direction of empirical pedantry, where the only things that count are facts, no matter how limited their purview. Computer programs provide information about the coherence of a set of assumptions, but they exert a dangerous pull in the direction of systematic delusion, where all that counts is internal consistency, no matter how remote from reality. Give up one approach and you turn into a Gradgrind, the teacher in Dicken’s novel Hard Times, whose only concern is with the facts; give up the other and you become an architect for the Flat Earth Society.”
So maybe pure experiment isn’t a good idea either?
I would add theoretical architects to your list of those whose work can’t hold up to the harsh light of reality. Architects like Tom Wiscombe (http://www.emergentarchitecture.com/) do beautiful, complex renderings of projects that would be difficult or impossible to build.
There are countless others, and architecture students eat up their work. The problem is that most students are going to work in a corporate firm for a couple of years, drawing hospital bathroom plans in CAD, before they get to design anything.
I wish more theoretical architects would put their ideas to the test instead of working in a digital simulacra and hoping that construction technology will eventually catch up with their genius ideas . Real writers publish, real artists make, and real architects build.
I have commented on this
http://suifaijohnmak.wordpress.com/2009/02/13/is-applied-research-wasteful-in-academia/
i believe that practical is more essential than theory because theory only teaches about how to0 do but practical thaches to use it so only by reading is nothing