That’s interesting. I chose Sebastiano Vigna’s last PRNG based mostly on that it was the last thing published on the internet and that the claims made by the author were bold and generally reasonable. I never bothered to run the benchmarks myself and verify the claims. I just took them for granted. I guess the peope who chose PCG in their projects just did it when it was the last cool thing published. There is a room for accident in software engineering.
What you did is reasonable. Certainly, Vigna has been clear on how he recommends against PCG and he is a good researcher and programmer. Of course, we should remain open to the possibility that the people who went with PCG are correct in their choice. I refer you to John Cook who has been doing comparisons.
Marc Reynoldssays:
For what it’s worth I’ve run ~500 testu01 crush battery runs on the main PCG variants and xorshift128+ and xorshiro with no questionable p-values, so I feel OK using any of these with sample size per problem <= 10^5 which is way more than I ever care about. John Cook mostly talks about DIEHARDER and the NIST suites, which I personally don't have much confidence in (too easy to pass). Of course to make me feel like a jerk he's just put up a post using PractRand (which I know nothing about) on all of these…so it goes: https://www.johndcook.com/blog/2017/08/14/testing-rngs-with-practrand/
I think you’re misunderstanding the peer-review-and-publishing process. Ending up published is supposed to be at the end of a peer-review process. Though many “scientific journals” now exist where no such thing happens, as has been demonstrated, e.g. through fake papers, time and again. Not ending up published doesn’t mean no peer review happened. That happens, formally at each submission, presumably, and informally any time the author receives comments from other readers of the paper.
So what happens at “peer review”? Well, in this case it appears the paper didn’t convey enough of that “one of us”-smell for some reviewers to assent to publishing. Because the prose was too accessible, maybe? (Which is a silly proposition since science is useless if inaccessible, so its prevalence is a problem.) That doesn’t mean the work is any good, just that it is accessible.
And that is perhaps the core problem with any scientific work these days. You still have to read the paper and make up your own mind. The number of people who’ve just copied the ideas into (at least seemingly) working code says nothing about how good the presented method is, either; all it does is give some indication of popularity. But that is a different thing.
I think we are in agreement but I’d like you to consider the possibility that I am less naive about the process than you suggest.
Robert Zehsays:
As a someone who builds computer systems, I’m far more likely to use something like PCG — something with clear and easy to read public documentation and a public code repository — than I am to use something from a paper. If the paper is only available in a paywalled journal I’m unlikely to grab it.
My experience has been that the closer in spirit an academic project is to an open source project, the more likely it will be to have working code I can use and contribute to.
I know that academic papers are intended to describe novel ideas, not working code, but I’ve found that unimplementable ideas are plentiful. Novel ideas that also good engineering solutions are much rarer.
.Tomsays:
That’s an important point. O’Neill made it easy to use the works.
Hrobjartur Thorsteinssonsays:
Melissa seems to be right that PRNG are largely verified benchmarks, hopefully well designed and possibly research published statistical tests.
I imagine all these tests that all these professors have devised and applied will eventually fault the PCG in some way as they fault other generators. She is also very right to advertise that many of the worst generators are still defaults in our modern compilers and programming languages.
The current default Golang math/rand.go is a very scary choice, because nobody seems to have been able to track down the creator of that algorithm, nor any publication describing it.
PCG will not disappear just because the professor has quirks when it comes to academia and publishing — loads of quirky ones out there. As you indicate also, no evidence seems to have surface to reject PCG, are academics not trying hard enough to fault the algorithm?
Someone expert in testing randomness may want to write a statistical test to fault PCG soon because it seems it may become a prime choice for Go 2.0 and future versions of C++. After all, most software developers, compiler developers and hardware developers love benchmarks, and the benchmarks that exist say this one is better. Problem is maybe that we have a selection effect here, someone just hasn’t yet written a benchmark that specifically faults PCG?
That’s interesting. I chose Sebastiano Vigna’s last PRNG based mostly on that it was the last thing published on the internet and that the claims made by the author were bold and generally reasonable. I never bothered to run the benchmarks myself and verify the claims. I just took them for granted. I guess the peope who chose PCG in their projects just did it when it was the last cool thing published. There is a room for accident in software engineering.
What you did is reasonable. Certainly, Vigna has been clear on how he recommends against PCG and he is a good researcher and programmer. Of course, we should remain open to the possibility that the people who went with PCG are correct in their choice. I refer you to John Cook who has been doing comparisons.
For what it’s worth I’ve run ~500 testu01 crush battery runs on the main PCG variants and xorshift128+ and xorshiro with no questionable p-values, so I feel OK using any of these with sample size per problem <= 10^5 which is way more than I ever care about. John Cook mostly talks about DIEHARDER and the NIST suites, which I personally don't have much confidence in (too easy to pass). Of course to make me feel like a jerk he's just put up a post using PractRand (which I know nothing about) on all of these…so it goes:
https://www.johndcook.com/blog/2017/08/14/testing-rngs-with-practrand/
I’d like to clarify that I did not mean to suggest in any way that Vigna’s RNG were not good choices.
You also might be interested in Xoroshiro:
http://xoroshiro.di.unimi.it
I think you’re misunderstanding the peer-review-and-publishing process. Ending up published is supposed to be at the end of a peer-review process. Though many “scientific journals” now exist where no such thing happens, as has been demonstrated, e.g. through fake papers, time and again. Not ending up published doesn’t mean no peer review happened. That happens, formally at each submission, presumably, and informally any time the author receives comments from other readers of the paper.
So what happens at “peer review”? Well, in this case it appears the paper didn’t convey enough of that “one of us”-smell for some reviewers to assent to publishing. Because the prose was too accessible, maybe? (Which is a silly proposition since science is useless if inaccessible, so its prevalence is a problem.) That doesn’t mean the work is any good, just that it is accessible.
And that is perhaps the core problem with any scientific work these days. You still have to read the paper and make up your own mind. The number of people who’ve just copied the ideas into (at least seemingly) working code says nothing about how good the presented method is, either; all it does is give some indication of popularity. But that is a different thing.
I think we are in agreement but I’d like you to consider the possibility that I am less naive about the process than you suggest.
As a someone who builds computer systems, I’m far more likely to use something like PCG — something with clear and easy to read public documentation and a public code repository — than I am to use something from a paper. If the paper is only available in a paywalled journal I’m unlikely to grab it.
My experience has been that the closer in spirit an academic project is to an open source project, the more likely it will be to have working code I can use and contribute to.
I know that academic papers are intended to describe novel ideas, not working code, but I’ve found that unimplementable ideas are plentiful. Novel ideas that also good engineering solutions are much rarer.
That’s an important point. O’Neill made it easy to use the works.
Melissa seems to be right that PRNG are largely verified benchmarks, hopefully well designed and possibly research published statistical tests.
I imagine all these tests that all these professors have devised and applied will eventually fault the PCG in some way as they fault other generators. She is also very right to advertise that many of the worst generators are still defaults in our modern compilers and programming languages.
The current default Golang math/rand.go is a very scary choice, because nobody seems to have been able to track down the creator of that algorithm, nor any publication describing it.
PCG will not disappear just because the professor has quirks when it comes to academia and publishing — loads of quirky ones out there. As you indicate also, no evidence seems to have surface to reject PCG, are academics not trying hard enough to fault the algorithm?
Someone expert in testing randomness may want to write a statistical test to fault PCG soon because it seems it may become a prime choice for Go 2.0 and future versions of C++. After all, most software developers, compiler developers and hardware developers love benchmarks, and the benchmarks that exist say this one is better. Problem is maybe that we have a selection effect here, someone just hasn’t yet written a benchmark that specifically faults PCG?
Some criticisms of PCG:
https://groups.google.com/forum/m/#!topic/prng/jJVEldv3sH0