The single most important thing I learnt when programming commercially was the importance of tests.
I cannot imagine implementing a numerical algorithm now without first defining some simple tests for expected input/output behaviour.
As I said in my comment on your previous post, writing tests also have the added bonus of making you focus on the important code first.
If you’re interested, the *second* most important thing I learnt while programming commercially was the proper use of version control systems (CVS, then SVN, now git) and using them to define processes for code maturation (e.g., having “sandpit” and “production” branches and tagging releases).
I’ve found both unit testing and version control very useful in my hobby and research programming.
David Eppsteinsays:
For more complicated algorithmic problems, I’ve also heard anecdotal evidence of good results from “property testing”: your algorithm should return a short proof that the result is actually what it says it is, and then check that this proof is valid.
Example: suppose you want an algorithm that determines whether a graph is bipartite, and returns the answer as a Boolean. Rather than just doing that, implement a (not much more complicated) algorithm that, when the graph is bipartite, returns a two-coloring, and when it is not bipartite, returns an odd cycle, and then add to the implementation some simple code for checking whether the result returned by this algorithm really is a two-coloring or an odd cycle.
Unit testing can ensure that your code runs correctly on the test cases you’ve thought to try. Property testing can ensure that your code runs correctly on the cases your users actually run it on. And when done well, it doesn’t much hurt the efficiency of the code because the testing part should be much faster than the original algorithm.
So true. I used to write all of my research code in the Cowboy coding manner, and the only “debugging” tool I used was printf. While that type of code is easy to write, I also spent hundreds of hours scratching my head over why the code broke.
But as I spent my last summer at Google, I was forced to go through unit testing. While it took me a while to get used to it, my code never broke after I added a change. I found “unit testing” to be sort of like a religion in the industry … and I wish more academics emphasized on this for research code.
There are instances I can think of where unit testing is appropriate, but “property testing” less so.
1) Any form of indexing… “find all elements having property X”. Ok, you can check cheaply that all your elements do have the required property, but checking that you have all of them all the time is going to make your fast algorithm useless.
2) Any form of approximation algorithm… suppose you wanted to compute some measure within epsilon of the true value… computing the true value each time would defeat the purpose of the approximation.
3) Any form of optimization problem. Suppose your algorithm finds a low-cost instance, how do you check that it is the best possible solution? In many real-life cases, it may just be totally impractical (or impossible) to check numerically that you have the best solution.
Many practical problems are such that constants do matter. Being twice as fast is a big deal. Even being 40% faster is a big deal. So checking your solution at runtime is very costly.
However, unit testing does not impact the performance of your software negatively.
That is not to say that property testing is not a cool idea. But is it as practical as unit testing?
Your point leads to an important corollary: You will write a lot more code than just the algorithm in order to properly implement the algorithm.
This may contribute to poor delivery time estimates, even for research projects implemented by a single person.
David Eppsteinsays:
Not as practical as unit testing, maybe, and as you say it doesn’t work well for all problems. I was just throwing it out as a counterexample to your “all more sophisticated strategies are usually not worth the cost†claim.
Right. Of course, but I think that even if you do property testing, you should also do unit testing. I could elaborate, but I think that static, deterministic tests are very good for your sanity.
Now. If only I would do unit testing for all software I publish. 😉
“”””You should always do unit testing for any kind of code that is supposed to have lasting value.””””
See the second part? 😉
David Eppsteinsays:
Re: even if you do property testing, you should also do unit testing: I agree. It’s better to find out about the bugs yourself while you’re working on the code than to let your users find them for you later.
jeremysays:
I see a tension between insight #1 (keep it simple, because you’re going to throw the code away in 4 days) and insight #2 (develop unit tests for everything).
Given that I also do a lot of throwaway code, just to get a sense about whether certain ideas are worth pursuing or not, should I also be writing unit tests for that code? It seems like it would take me longer to write the code than I would actually use it. So why should I write unit tests for all my throwaway sed and awk code? My disposable matlab functions? My one-off unix pipe structures?
Kevembuanggasays:
There are instances I can think of where unit testing is appropriate, but “property testing†less so.
I only agree with the following point 3 because for 1 and 2 you don’t seem to make the distinction between test runs and production runs.
(Huh? in the roman numeral spam protection the example matches the actual test case: I + II + IX= XII !!! )
Kevembuanggasays:
It’s better to find out about the bugs yourself while you’re working on the code than to let your users find them for you later.
You’ll never be employed by Microsoft!
Or have you and been kicked off?
jeremysays:
See the second part?
Ah, yes! Silly me.
But then I guess the next interesting question is: As a researcher, at what point do you make the decision to turn throwaway code into real, unit-tested code? And what is your migration path? Do you take your scripts and integrate them into the larger project? Or do you rewrite the ideas in a more “stable” language? Or does it always depend on the situation?
1) Build code for a project
2) Publish the source code with unit testing (only the part of the source code used in the production of papers)
3) Publish the paper(s)
What I actually do is a hybrid where I sometimes omit the unit testing. I have no excuse except that I sometimes run out of energy or time.
I can’t agree with this insight more.
The single most important thing I learnt when programming commercially was the importance of tests.
I cannot imagine implementing a numerical algorithm now without first defining some simple tests for expected input/output behaviour.
As I said in my comment on your previous post, writing tests also have the added bonus of making you focus on the important code first.
If you’re interested, the *second* most important thing I learnt while programming commercially was the proper use of version control systems (CVS, then SVN, now git) and using them to define processes for code maturation (e.g., having “sandpit” and “production” branches and tagging releases).
I’ve found both unit testing and version control very useful in my hobby and research programming.
For more complicated algorithmic problems, I’ve also heard anecdotal evidence of good results from “property testing”: your algorithm should return a short proof that the result is actually what it says it is, and then check that this proof is valid.
Example: suppose you want an algorithm that determines whether a graph is bipartite, and returns the answer as a Boolean. Rather than just doing that, implement a (not much more complicated) algorithm that, when the graph is bipartite, returns a two-coloring, and when it is not bipartite, returns an odd cycle, and then add to the implementation some simple code for checking whether the result returned by this algorithm really is a two-coloring or an odd cycle.
Unit testing can ensure that your code runs correctly on the test cases you’ve thought to try. Property testing can ensure that your code runs correctly on the cases your users actually run it on. And when done well, it doesn’t much hurt the efficiency of the code because the testing part should be much faster than the original algorithm.
So true. I used to write all of my research code in the Cowboy coding manner, and the only “debugging” tool I used was printf. While that type of code is easy to write, I also spent hundreds of hours scratching my head over why the code broke.
But as I spent my last summer at Google, I was forced to go through unit testing. While it took me a while to get used to it, my code never broke after I added a change. I found “unit testing” to be sort of like a religion in the industry … and I wish more academics emphasized on this for research code.
David:
There are instances I can think of where unit testing is appropriate, but “property testing” less so.
1) Any form of indexing… “find all elements having property X”. Ok, you can check cheaply that all your elements do have the required property, but checking that you have all of them all the time is going to make your fast algorithm useless.
2) Any form of approximation algorithm… suppose you wanted to compute some measure within epsilon of the true value… computing the true value each time would defeat the purpose of the approximation.
3) Any form of optimization problem. Suppose your algorithm finds a low-cost instance, how do you check that it is the best possible solution? In many real-life cases, it may just be totally impractical (or impossible) to check numerically that you have the best solution.
Many practical problems are such that constants do matter. Being twice as fast is a big deal. Even being 40% faster is a big deal. So checking your solution at runtime is very costly.
However, unit testing does not impact the performance of your software negatively.
That is not to say that property testing is not a cool idea. But is it as practical as unit testing?
Your point leads to an important corollary: You will write a lot more code than just the algorithm in order to properly implement the algorithm.
This may contribute to poor delivery time estimates, even for research projects implemented by a single person.
Not as practical as unit testing, maybe, and as you say it doesn’t work well for all problems. I was just throwing it out as a counterexample to your “all more sophisticated strategies are usually not worth the cost†claim.
Right. Of course, but I think that even if you do property testing, you should also do unit testing. I could elaborate, but I think that static, deterministic tests are very good for your sanity.
Now. If only I would do unit testing for all software I publish. 😉
I wrote:
“”””You should always do unit testing for any kind of code that is supposed to have lasting value.””””
See the second part? 😉
Re: even if you do property testing, you should also do unit testing: I agree. It’s better to find out about the bugs yourself while you’re working on the code than to let your users find them for you later.
I see a tension between insight #1 (keep it simple, because you’re going to throw the code away in 4 days) and insight #2 (develop unit tests for everything).
Given that I also do a lot of throwaway code, just to get a sense about whether certain ideas are worth pursuing or not, should I also be writing unit tests for that code? It seems like it would take me longer to write the code than I would actually use it. So why should I write unit tests for all my throwaway sed and awk code? My disposable matlab functions? My one-off unix pipe structures?
There are instances I can think of where unit testing is appropriate, but “property testing†less so.
I only agree with the following point 3 because for 1 and 2 you don’t seem to make the distinction between test runs and production runs.
(Huh? in the roman numeral spam protection the example matches the actual test case: I + II + IX= XII !!! )
It’s better to find out about the bugs yourself while you’re working on the code than to let your users find them for you later.
You’ll never be employed by Microsoft!
Or have you and been kicked off?
See the second part?
Ah, yes! Silly me.
But then I guess the next interesting question is: As a researcher, at what point do you make the decision to turn throwaway code into real, unit-tested code? And what is your migration path? Do you take your scripts and integrate them into the larger project? Or do you rewrite the ideas in a more “stable” language? Or does it always depend on the situation?
What I should do is this:
1) Build code for a project
2) Publish the source code with unit testing (only the part of the source code used in the production of papers)
3) Publish the paper(s)
What I actually do is a hybrid where I sometimes omit the unit testing. I have no excuse except that I sometimes run out of energy or time.