Today we release the next chapter in the Quantify Me series: Metacritic. Having done an analysis of the CSDb first (CSDb: Quantify Me) in July 2010 and then Mobygames (Mobygames: Quantify Me) at the end of December 2012, the attention has turned to the games listed at Metacritic. This one turned out to big a big analysis, collecting data almost non-stop from the Metacritic website between the 25th of August 2013 and the 10th of September 2013. Then cleaning of the database and actual analyses took another lengthy period of time, as this needed to be done late in the evenings, as much as possible. Finally the first chapter of this endeavor is here. The conclusion of this 30 page paper (which you can get here as PDF) is as follows:
The data at Metacritic leaves much to be desired and seems to be heavily biased. When critic scores do not comply with user scores in the majority of cases, which has been shown in this paper, and the selection of critics is seemingly under no formal control, the validity and accuracy of the data is low. Caution is necessary when using Metacritic information to guide your own decision to buy or not buy a game. Until there is more transparency on how this process takes place at Metacritic, more transparency on the flow of funding from which sources and the observed biases are removed, the database is of limited use for the end-users.