Anonymous benchmark markets

It’s 1982. RSI decides to focus on its main product, Oracle Database; they rename themselves as Oracle. Meanwhile, at the University of Wisconsin, Dina Bitton, David DeWitt, and Carolyn Turbyfill create a database benchmarking framework. Oracle does not fare well.

Larry Ellison tries to have DeWitt fired. When that doesn’t work, he bans Oracle from hiring Wisconsin grads and Oracle adds a term to their EULA forbidding the publication of benchmarks. Soon afterwards, every major commercial database vendor (other than IBM) adds a license clause that makes benchmarking their database illegal, without special permission.

Today, Oracle hires from Wisconsin, but the DeWitt Clause has spread from databases to all software, from compilers to cloud offerings1. It’s impossible to make an informed decision about which cloud doodad to use because it’s illegal to benchmark them. Meanwhile, Bitcoin users have created anonymous markets for assassinations – users can put money into a pot that gets paid out to the assassin who kills a particular target.

How about anonymous markets for benchmarks? People who want to know what kind of performance a database offers under a certain workload puts money into a pot that gets paid out to whoever runs the benchmark.

  1. There are a handful of vendors that disallow not only publishing benchmarks, but even running them to see how well the competition does. There’s a list of companies whose cloud products I’m told I shouldn’t use to avoid even the appearance of impropriety because I work in an office with people who work on cloud related infrastructure. [return]