IMO, when discussion shifts to the squabbling over semantics, it has degenerated. The free market question is whether the market can force producers to provide consumers with good quality information
> The requirement for consumers having accurate information is a government regulation. In the United States, it's enforced by the Consumer Protection Agency. It isn't a free market requirement
Yet, when Oracle prohibits publishing benchmarks you can find benchmarks about MySQL, Mongodb etc on the web
The thing is, a skilled DBA, given two databases and told which one should be the winner, can easily construct a benchmark that seems perfectly plausible, but favours one over the other. The MongoDB guys did it very blatantly e.g. by comparing Postgres writing to disk with them writing to memory, but that's because they don't know anything about databases and lacked the skill to do it subtly, e.g. by finding pathological edge cases in query optimizers. That's what the commercial vendors are most worried about.
I would never rely on TPC for any benchmarking. Just read their history, founders and membership structure carefully and you'll get an idea what type of `neutral` benchmarking they do )
> The requirement for consumers having accurate information is a government regulation. In the United States, it's enforced by the Consumer Protection Agency. It isn't a free market requirement
Yet, when Oracle prohibits publishing benchmarks you can find benchmarks about MySQL, Mongodb etc on the web