Daniel Lemire's blog

, 4 min read

Pick one: determinism or fairness

Computers changed our life drastically in the last few decades. Correspondingly, I view the world in terms of algorithms. When I think of how the government works, for example, I see over-engineered heuristics. Bureaucracies are akin to spaghetti code.

Can you tell me how my tax dollars are spent? Surely, you can come up with histograms showing how the money is distributed. But beyond such a rough analysis, nobody knows exactly how money flow within government.

I believe this complexity is beneficial. People need to feel that the system is fair. We achieve the appearance of fairness through obscurity. If I knew that my wealthy neighbor, the guy with a larger house and a larger car, makes all this money through government contracts because he has a friend at the right place, I would get upset. But right now, who knows what is happening?

Effectively, to people like me, government appears like a randomness generator. I don’t quite know how much money the government is charging me, and I don’t know what I am getting back in funding and services. I trust that most people are similarly in the dark. I therefore feel that it is somewhat fair… because it is illegible.

But this works mostly because the data is not available. Of course, if I had access to all government data, it would be too much to take in on my own… but I could analyze it with a computer. Decades of work in business intelligence has taught us a few tricks. We can represent very complex systems in a way that lets the user navigate the information efficiently. Couple this capacity with a social approach, where many citizens could collaborate during the analysis, and we can effectively break the secrets of government. Soon, government could become legible. The illusion of fairness might be broken.

How likely is it that we will have such open governments? It is difficult to tell. In many countries like Canada, we had arrived at a compromise where the press can invoke the law to access some documents. More recently, at least in Canada, even the press is routinely frustrated by its attempts to open up the government. And Wikileaks might have unpredictable effects on governments: some governments will surely tighten security.

Yet I don’t believe governments will be able to hide forever: the long-term trend is that we have access to more data, not less. But maybe there is a saner alternative to obscurity?

One could object that governments could tend toward an ideal of deterministic fairness. Alas this is doomed. For example, Arrow’s theorem states that no deterministic voting is fair. It is one of several mathematical results showing that determinism and fairness do not mix. You get one or the other. In Demarchy and probabilistic algorithms, I promoted that idea that elections themselves should rely on randomness. But I also think that the inner workings of government should embrace randomness:

  1. To ensure that they get the lowest prices, governments ask for vendors to bid on jobs or orders. Current bidding systems are easily broken by insiders. If you know how others are bidding, you can easily outbid them. To compensate, governments complicate the process to the point where hardly anyone wants to participate anymore. But there are many random selection algorithms that are much harder to game. By bidding lower, you only increase the probability that you will selected, you can never be sure to get the jobs. Effectively, it can be fairer by being harder to game.
  2. Hiring for public jobs is often a big problem. It is tempting for a civil servant to favor his friends and his family when recruiting. To get around this problem, governments create complicated rules and lengthy processes. But what about selecting randomly among the qualified candidates? This would be fair and I bet that the selected candidates would be just as good as the ones governments hire right now.
  3. In my town, we have some services, such as a community garden, that are provided on a first-come, first served basis. Hardly anyone knows of these services, and that is good because the local government could not provide them to everyone. So these programs remain obscure. It seems like a random allocation would be a lot fairer.

Conclusion: Governments often insist on determinist methods. These are often unfair. We can better tend toward an ideal of fairness by embracing randomness. It seems counter-intuitive at first, but it is certainly preferable to obscurity.