5.3.1 Meta-analyses

Whereas primary analysis conducts research on the original data, and secondary analysis researches the same data sample with new techniques in order to find an answer, a meta-analysis analyses prior analyses of the same subject (Glass 1976: 3). The data points in a meta-analysis are the results from individual studies, instead of the characteristics of an individual object (Wolf 1986: 11). After the introduction of the meta-analysis by Glass, it has been used increasingly, especially in behavioural and social sciences, but also in other disciplines (Hunter and Schmidt 1990: 35-42). A meta-analysis is especially popular in research areas where individual studies fail to detect a significant statistical relationship between two variables. Due to statistical techniques within meta-analyses, confidence intervals become smaller and non-zero relationships can be detected more easily, which increases the statistical power (Cohn and Becker 2003: 243). These benefits and the statistical or quantitative character of a meta-analysis with techniques to account for errors in research, make people more comfortable with conclusions derived from a meta-analysis than from narrative literature reviews (García-Meca and Sánchez-Ballesta 2009: 595).

Besides the advantages of meta-analyses over literature reviews and the high statistical power, there are also several disadvantages involved in the use of this technique. The availability bias or the file drawer problem is a point of concern in meta-analytic research. It does not focus on problems within the data sample, but on problems with studies that are not included in the meta-analysis. As studies are more likely to be published when they have found significant results, studies without these significant results might have ended up in the file drawer. The significant relationships are more readily available than non-significant relationships, which might lead to an overestimation of the reported relationship (Hunter and Schmidt 1990: 506-507; Utts 1996: 427). A second problem is the mixing of apples and oranges. This phenomenon entails the problem that dependent and independent variables are different between studies, as they are constructed in another way or measured in another way (Hunter and Schmidt 1990: 516-517). The results that are aggregated in a meta-analysis are not based on the same sort of variables and this might cause deviations for the final inference. Utts also addresses this problem, by stating that variables might have the same name, but that subtle differences might lead to differences that will ultimately bias the results of the meta-analysis (1996: 427).