Impact of findings from grey literature on the outcomes of systematic reviews on interventions to prevent obesity among children: a systematic review

Review question/objective Public health interventions are often implemented directly by government and so have an observational, rather than experimental, evidence base that may not be captured in traditional academic published sources. There is, therefore, a need to determine whether this grey literature is being actively sought when evaluating the effectiveness of public health interventions through systematic literature review; and also whether the inclusion of this grey literature has an impact on the conclusions of systematic literature reviews (SRs) evaluating public health interventions. This is particularly important as the findings of systematic literature reviews are increasingly used to inform public health policy decisions. Where possible, the sources used to obtain the grey literature in the included SRs will be categorised. As the prevention of childhood obesity is a major emerging public health issue in developed countries, this has been chosen as the topic for this methodological systematic review. The review question is: What is the impact of findings from grey literature on the outcomes of systematic reviews on interventions to prevent obesity among children? Primary Review Objective is to evaluate the impact of findings from grey literature on the results of systematic reviews on prevention of childhood obesity. Secondary Review Objective is to determine the ratio of black to grey literature (definitions are provided in Background) included in the eligible systematic reviews; to determine the study design of the grey literature included in SRs. Background By 2025 obesity rates are predicted to rise in Australia by 65% despite the myriad substantial efforts of a multiplicity of interventions and strategies from the public health sector, particularly those at the community or small unit level.1 Childhood obesity often persists into adulthood, and these strongly established links to adult obesity along with all the attendant risks and consequences make childhood a natural starting point for a closer examination of prevention literature.2,5-6 It also provides a logical focus for early intervention and therefore an abundance of literature exists in the field. Recent literature increasingly supports the argument that high level policy, regulation and legislation is required to prevent childhood obesity, and that whole-of-population, whole-of -environment approaches need to be developed and implemented.7-8 However while policy: “a set of plans to establish and achieve the desired performance goals of a group or organisation”9 abounds, and often confounds10 successes remain generally elusive. To be genuinely evidence-informed, policy needs to access systematic reviews which provide summary conclusions derived from their critical assessments and syntheses of the relevant evidence base. To be comprehensive and representative this base should include, or at least consider, “grey literature and unpublished studies”.11There are two types of literature which could be included in systematic reviews and are known colloquially as black and grey. Black literature is that which is published in academic, scholarly journals. Grey literature, which is usually defined as “Information produced on all levels of government, academia, business and industry in electronic and print formats not controlled by commercial publishing i.e. where publishing is not the primary activity of the producing body”12, takes many different forms13 but is essentially documents that have not been formally published, and have commonly not been peer-reviewed.14 Research institutes for example, produce working papers, technical and consensus reports, issues papers and policy briefs which have valuable content to impart and constitute grey literature. In addition the Joanna Briggs Institute, the Cochrane Collaboration and the Campbell Collaboration stipulate that, in addition to black literature, searches for grey and unpublished literature must be conducted by their systematic reviewers in order to avoid publication bias.15 While systematic reviews sum up the best available research on a specific question by “synthesizing the results of several studies”16 decision makers are now often faced with numerous systematic reviews on the same health issue. The next logical and appropriate step is an overview of systematic reviews, to synthesise and summarise the findings17 and if the overview is itself a systematic review, then the end result should provide stronger and even more comprehensive evidence. A recent dialogue of Australian public health professionals and dietitians emphasised the advocacy approach to obesity prevention, settling on the tripartite strategy of “political commitment, multi-sectoral support and community engagement.”18 It is this complexity of approach that may benefit from an examination of the grey literature (GL) supporting these policies. Simkhada's view19 that GL is needed to fill the information gap as “an essential part of the evidence base for practice in complex interventions, which may have multi-stakeholders, have multi-variables, have a lack of predictability and robust data and require a broad literature approach”, is held by many others20-23. Currently the Australian government is investing in research into the access and preservation of GL in light of its relevance to the development of health policy24. Similarly in 2006 the U.S. National Library of Medicine, aware that policymakers, more than any other research group, found GL especially relevant for context, and for its ability to reflect and map “how debate changes over time on a particular topic”25 commissioned AcademyHealth26 to conduct a research project into the scope and status of this literature. Often the strength of GL lies in the way it can reflect the lay voice, the voice from the field, public opinion: what people think. How can policy succeed when individuals' beliefs and behaviours are at odds with the underlying principles and truths of achieving healthy weight?27-28 Baum and others suggest that we need to look below the surface, focusing on the underlying social determinants of heath rather than “immediate and visible causes” and that the role of public policy should be to shape the social environment so that it is more conducive to good health.29-30 As Australian obesity prevalence continues to rise (in fact no country has managed to reverse obesity trends)31-32, it is reasonable to suggest that policy associated with obesity prevention is at best flawed or limited and possibly under- or ill-informed. As Badger et al argue “it is irresponsible to interfere in the lives of other people on the basis of theories unsupported by reliable empirical evidence”11. Community engagement is a challenging but well-supported aspect of behavioural change in public health. Community receptiveness needs to be gauged and substantiated in creative ways so that it can feed into the information that underpins our policies and supports implementations.33-34. To achieve a greater degree of success in regard to childhood obesity prevention, there is a need to tap into evidence from the widest variety of sources, so as to reflect the complexities of the issue and perceptions of end-users.35 Grey literature is where community attitudes, beliefs, values and opinions might be reasonably expected to be found, and also where pilot programs and case studies assessing the feasibility of public health interventions will have been reported. The degree to which this literature has been accessed and incorporated into the evidence that informs policy may have an impact on the ultimate success or sustained “take-up” of an intervention. To determine the extent to which systematic reviews in the field of child obesity prevention identify and include grey literature in their collated evidence-bases, a comprehensive search of the literature will be undertaken. The resultant systematic reviews will be retrieved and examined for inclusion. Those that do not match the selection criteria will be excluded. The ratio of black to grey literature in the included systematic reviews will be calculated and the proportion of systematic reviews that are grey will be briefly discussed. A comparison of results from grey versus black systematic reviews, and of those systematic reviews with high versus low ratios of included grey literature, will be undertaken to determine whether grey literature changes the findings or interpretation of the outcomes that are assessed. Inclusion criteria Types of studies Systematic reviews of interventions to prevent obesity among children, where there is either meta-analysis or narrative summary or tabular presentation of results. Types of reviews will include those where the primary literature consists of experimental or observational literature. Traditional literature reviews, or narrative reviews not based upon an a-priori protocol and rigorous methodology will be excluded. Qualitative synthesis or mixed methods reviews will also be included. Types of participants The reviews of interest will only include studies on children aged two to 18 years without (at baseline) a diagnosis of obesity, or eating disorders, or co-morbid conditions that pre-dispose to obesity. Types of intervention(s) Public health interventions aimed at obesity prevention that may be applied at the population, community (including schools) or primary care level. Types of outcomes Primary - Obesity prevention will be measured according to body mass index (weight/height2) as calculated against a suitable growth reference e.g. relevant age and .sex growth classification standards, with classification into underweight, healthy weight, overweight, and obese categories.36-37 Outcomes will be grouped according to the type of prevention program implemented. Secondary - Ratio of black to grey eligible systematic reviews; ratio of black to grey studies included in eligible systematic reviews; study designs of grey literature included in eligible systematic reviews; sources used in eligible systematic reviews to obtain grey literature Search strategy The search strategy aims to find both published and unpublished systematic reviews. A three-step search strategy will be utilised in this review. An initial limited search of MEDLINE, EMBASE, CINAHL and the Cochrane Library will be undertaken to “scope” the literature followed by an analysis of the text words contained in the title and abstract, and of the index terms used to describe article. A second search using all identified keywords and index terms will then be undertaken across all included databases. Thirdly, the reference list of all identified reviews, reports and articles will be searched for additional systematic reviews. Systematic reviews published in English will be considered for inclusion in this review, with no date limitation. The databases to be searched include: ARIF (Aggressive Research Intelligence Facility): Reviews database & Methodology database; Biological Abstracts; Campbell Library; CINAHL; Cochrane Library; EMBASE; ERIC; Google Scholar; INFORMIT: Education, Health and Social Science databases; Medline/PubMed; PAIS; Proquest; PsycInfo; Scopus; Sociological Abstracts; TRIP (Turning Research into Practice) database. The search for unpublished systematic reviews will include: DIPEx - Database of Individual Patient Experiences; EPPI Centre (particularly Obesity and Sedentary Behaviour Database); Google Advanced Search; Google Books; GreyNet; Mednar; Scirus; Social Care Online (SCIE); TROVE (for theses, books); WHOLIS. Major international institutional repositories will also be searched. Journals which will be hand searched include Childhood Obesity, International Journal of Pediatric Obesity, Obesity and Obesity Reviews. Initial search keywords to be used will be: (Obesity OR obese) AND (Child or paediatric or pediatric or adolescent or youth or infant) AND (prevent* or intervent*) AND (systematic review or meta-analys* or meta analys*). Where a thesaurus of descriptors or index terms exists, or the ability to “explode” headings, or limit to years of publication and/or document type, full advantage will be taken of these database-specific features. Searches will comprise of a combination of keywords and other available options designed to maximize the chances of retrieval of eligible studies. An example of a completed reproducible search strategy will be appended to the systematic review report. (Appendix I) All study titles and abstracts will be imported into reference management software, duplicates removed, and each title/abstract reviewed according to the inclusion criteria. The full text paper of each potentially relevant study will then be retrieved and again reviewed for eligibility according to the inclusion criteria. Reasons for exclusion will be documented. In cases where study eligibility is uncertain, a consensus decision will be reached by the co-authors. A final database of included systematic reviews will be produced. A PRISMA flowchart38 will be constructed so that the article selection process is transparent and can be replicated. Assessment of methodological quality Systematic reviews selected for retrieval will be assessed by two independent reviewers for methodological validity using the Joanna Briggs Institute (JBI) critical appraisal checklist for systematic reviews prior to inclusion in the review (Appendix II). Any disagreements that arise between the reviewers will be resolved through discussion, or with a third reviewer. Data collection Data will be extracted using the JBI standardised extraction tool for systematic reviews (Appendix III). The data extracted will include specific details about the phenomenon of interest, populations, study methods and outcomes of significance to the review question and specific objectives. Data synthesis The unit of interest is the pooled body mass index results from the eligible systematic reviews (whether presented as a mean difference in BMI or difference in proportions of underweight, healthy, overweight or obese), grouped according to type of obesity prevention program. As the eligible systematic reviews are likely to include studies presented in other eligible systematic reviews in the evidence base (ie duplicated data), it would not be appropriate to meta-analyse the pooled body mass index results for the same interventions reported in these systematic reviews as it would result in double-counting. As the objective of this systematic review is to determine the impact of grey literature on review results, for each type of childhood obesity prevention program an assessment of the concordance of pooled results between grey and black systematic reviews will be undertaken. ie overlapping confidence intervals of the pooled results, and evaluating the agreement of findings/discrepancy in findings. It will also be determined whether there are patterns or trends in pooled results according to the increasing levels/ratios of grey literature in the included systematic reviews. Potentially, sensitivity analysis will be undertaken and publication bias39 will be examined, both as tests of the impact of Grey Literature if there are a sufficient number of studies A narrative meta-synthesis (i.e. meta-synthesis and narrative summary) of the results for each pre-specified outcome will be presented within subheadings appropriate to each type of obesity prevention program identified and assessed in the included systematic reviews. In the event that any systematic reviews compare results from grey and black literature, these will be presented narratively and potentially meta-analysed if the data are homogenous and not duplicated between systematic reviews. Conflicts of interest No conflicts of interest Acknowledgements This review will form part of a submission for the Masters of Clinical Science and therefore a secondary reviewer Lisa Heuch (LH) will only be used for critical appraisal. LH is a Masters of Clinical Science Candidate, The Joanna Briggs Institute, The University of Adelaide.