Finding and appraising evidence: Using Evidence for Impact

Print document | Download PDF

March 2016

The Social Policy Evaluation and Research Unit’s (Superu’s) purpose is to increase the use of evidence by people across the social sector so that they can make better decisions – about funding, policies or services – to improve the lives of New Zealanders, New Zealand communities, families and whānau.

The Using Evidence for Impact project takes a big-picture approach and aims to inspire all those working in the New Zealand social sector to use evidence in decision-making. 

The objectives behind the programme are to drive:

  • greater accessibility to evidence
  • greater transparency of evidence
  • capability development and good practice in using evidence.

Expectations for evidence-based policy and practice are rising in the social sector, but good evidence can be difficult to find. Policymakers, funders, and practitioners need to be able to assess how trustworthy evidence is, and how relevant it is to their situation, so as to inform their decisions about appropriate actions. 

01_ Critical appraisal of research and evaluation evidence 

back-to-top |

1.1. Introduction to critical appraisal

What is critical appraisal? 

Critical appraisal is the process of systematically examining research or evaluation evidence to judge its trustworthiness, value and relevance in a particular context. Critical appraisal of an individual study involves scrutinising its design and methods, and judging whether its findings are of sufficient quality and have sufficient relevance to justify using it to influence your decision. In the case of a body of evidence, critical appraisal also involves judging whether the number of studies and the consistency across studies is sufficient for decision-making. 

Context matters in determining whether the evidence is good enough to influence decision-making. For example, a decision to fund a large, high profile initiative that affects many people and that could cause harm if it does not work as intended, should usually be underpinned by a large body of high quality and relevant evidence. But smaller initiatives with less risk of harm may require less certainty, and if existing evidence is weak, future evaluation could be planned so as to generate better evidence. 

Recommended approach to critical appraisal 

This guide recommends adopting the approach taken by the UK Department for International Development (DFID) (link in section 1.2). In this approach, the strength of a body of evidence is determined by study quality and relevance, the number of studies, and consistency across studies. 

This differs from approaches that use a hierarchy of methods to assign quality rankings. These hierarchies tend to assign the highest ranking to well conducted randomised controlled trials, followed by quasi-experimental designs, and then non-experimental designs (see section 1.4 for a description of these designs). Qualitative methods are often not included. Some of the resources listed in section 2 use such hierarchies. 

While hierarchies of evidence are useful and appropriate in some circumstances, we recommend adopting the DFID approach when you are critically appraising a body of evidence for use in social sector decision-making. This is because, given the range of purposes for which evidence is used in social policy and programme development, there is no universally applicable hierarchy of methods that is appropriate for answering all of the questions we ask. Different methods are better at answering different questions, and some of the most powerful evidence is produced when a range of methods are used. Consider the following examples of the strengths of methods in addressing different issues. 

  • A randomised controlled trial is useful for demonstrating causal linkages with a high degree of confidence. This can answer the question: did the programme lead to the outcomes that were measured, or were other factors responsible? A randomised controlled trial will not often enhance our understanding of why the programme worked or didn’t, what aspects worked well and for whom, and what worked less well. 
  • A quasi-experimental design is also useful for demonstrating causal linkages, but it may do so with less certainty than a randomised controlled trial. But it can be feasible where a randomised controlled trial cannot be used due to ethical or practical considerations. It can sometimes allow for larger sample sizes, improving the study’s ability to detect effects and allowing investigation of whether a programme has different effects on different people.

  • Qualitative designs can be useful for understanding perspectives and behaviours, and can provide insights into the factors that influence whether a programme works or not for certain people. This can help you to understand whether the programme may be successfully implemented in your context, and what factors you need to address to support successful implementation in your context. 

Overview of critical appraisal resources 

Sections 1.2 to 1.6 describe five types of critical appraisal guidance. Section 1.2 covers the DFID guidance plus two other resources that address aspects of critical appraisal that are not specific to particular research or evaluation methods. 

While the resources in section 1.2 can be applied to various research and evaluation methods, in some situations you will need more information on how to assess the quality of a study that uses a particular method. For example, the validity of a qualitative evaluation depends (in part) on how authentically the findings reflect participant experience, and validity can be promoted using methods such as participant validation of findings, peer debriefing, attention to negative cases, independent analysis of data by another researcher, and verbatim quotes. 

 On the other hand, the validity of a quantitative evaluation depends on how measurement and sampling was carried out. Sections 1.3 to 1.6 refer to guidance that will help you to understand the specifics of critically appraising systematic reviews, quantitative evaluations, qualitative evaluations, and economic evaluations. You should use these resources when you are unsure about the specific features that you should be looking for when appraising a study of that type. 

back-to-top |

1.2. Critical appraisal guidance that is not specific to a method

The resources in this section provide guidance on critical appraisal that is applicable across a range of research and evaluation methods. 

Resource Link

Department for International Development (2014) How to Note: Assessing the Strength of Evidence

This was developed for Department for International Development (DFID) staff, to help them use evidence in designing and implementing effective international development policies and programmes. Its guidance is relevant to evidence from the social sector as well as international development, and we recommend adopting this approach.

https://www.gov.uk/government/ publications/ how-to-note-assessing- the-strength-of-evidence

*recommended

Department for International Development, World Bank, US Agency for International Development, UN Children’s Fund (2015) Assessing the Strength of Evidence in the Education Sector

This is an adaptation of DFID (2014), for the education sector. It adds examples and case studies relevant to education.

http://reliefweb.int/sites /reliefweb.int/files/resources /BE2_Guidance_Note_ASE_final_ 2015-30-06f_.pdf

*recommended

Ontario Public Health Libraries Association (2014) Critical Appraisal of Research Evidence

Overview of the critical appraisal process for individual research articles. It is public health-focused, but much of the guidance can also be applied to research articles on other social sector topics. This adds to the DFID resource by providing more detail on how to read a journal article, for example it describes where in the article you can find different types of information, and it provides a critical appraisal form that you can fill in for each article.

http://www.ophla.ca/pdf/Critical AppraisalResearchEvidence August2014.pdf

Center for Evidence-Based Management resources and tools

This page links to a number of user-friendly resources that cover the basics of evidence-based practice, and critical appraisal of different types of evidence. The teaching materials may be the most useful and we encourage you to explore these links and see if any are of use to you. The resources on this site are intended for use in management practice, but many can also be applied to critical appraisal of evidence when developing policies and programmes.

http://www.cebma.org/resources-and- tools/

back-to-top |

1.3. Critical appraisal of systematic reviews

A systematic review is a literature review that answers a research or evaluation question by collecting and summarising all the evidence that fits the review’s pre-specified eligibility criteria. Systematic reviews ask questions such as ‘how effective is a particular treatment for a disease?’ or ‘what is the cost-effectiveness of a particular type of intervention?’ Reviewers gather all of the publications that investigate the question and meet eligibility criteria, and then use statistical techniques to combine the results and develop an overarching conclusion. Systematic reviews can be a good source of evidence for what works, as they combine results from multiple settings, and they apply quality criteria to ensure that conclusions are based on strong evidence from well-conducted studies. 

The quality of systematic reviews can vary, and it can sometimes be difficult to get the information that you need from a systematic review to determine how relevant the results are to your setting. The resources below provide guidance on assessing the quality and the local relevance of systemic reviews. 

Resource Link

Health Knowledge online tutorial on critical appraisal of systematic reviews

Interactive online learning module that covers how to appraise a systematic review, provides a checklist, a guided example to work through, a quiz, and links to further resources. While it is healthcare-focused, most of the content can also be applied to critical systematic reviews in other social sectors.

http://www.healthknowledge. org.uk/interactive-learning/fae/ systematic-reviews

AMSTAR Checklist (A Measurement Tool to Assess Systematic Reviews)

Seven questions that can be used to assess the methodological quality of systematic reviews. A systematic review is considered to be well done when all of the items on this checklist have been addressed. Some technical knowledge is needed to apply aspects of this checklist.

http://amstar.ca/Amstar_Checklist.php

Critical Appraisal Skills Programme (CASP) Systematic review checklist.

Widely used checklist containing ten questions that can be used to assess the validity of a systematic review, and to understand its results and the local applicability of those results. Some technical knowledge is needed to apply aspects of this checklist.

CASP tools and checklists:

http://www.casp-uk. net/#!checklists/cb36

 

back-to-top |

1.4. Critical appraisal of quantitative evaluations

Three main types of quantitative evaluation designs used. These are randomised controlled trials (sometimes called experimental designs), quasi-experimental designs, and non-experimental designs. The diagram below describes them.

information

The resources below contain guidance for assessing the strength and relevance of evaluations that use quantitative methods.

Resource Link

US Department of Education (2003) Identifying and implementing educational practices supported by rigorous evidence: a user-friendly guide.

User-friendly guidance on factors to consider when assessing whether a programme is underpinned by strong evidence. It is relatively non-technical and accessible for non-specialists. Focuses on education programmes, but is applicable across the social sector.

http://www2.ed.gov/rschstat/ research/ pubs/rigorousevid/ rigorousevid.pdf

U.K. Treasury (2012) Quality in policy impact evaluation: understanding the effects of policy from other influences (supplementary Magenta Book guidance)

This is a more in-depth report that discusses how well different experimental and quasi-experimental designs are able to separate the effects of the programme from the effects of other factors on outcomes.

https://www.gov.uk/ government/uploads /system/uploads/attachment_data /file/190984/ Magenta _Book_quality _in_policy_impact_ evaluation__QPIE_.pdf

Effective Public Health Practice Project (EPPHP) Quality assessment tool for quantitative studies.

Links to the EPPHP quality assessment tool and accompanying dictionary. The tool is a checklist to assist with the appraisal of studies that use experimental and quasi-experimental designs. The dictionary explains key concepts. While it is healthcare-focused, most of the criteria can also be applied to quantitative studies in other social sectors.

http://www.ephpp.ca/tools.html

Health Knowledge online tutorial on critical appraisal of randomised controlled trials.

Interactive online learning module that covers how to appraise a randomised controlled trial. Provides a checklist, a guided example to work through, a quiz, and links to further resources. While it is healthcarefocused, most of the content can also be applied to randomised controlled trials in other social sectors.

http://www.healthknowledge.org.uk/interactive-learning/fae/randomised-control-trials

Critical Appraisal Skills Programme (CASP) Randomised Controlled Trial checklist.

Eleven questions that can be used to assess the validity of a randomised controlled trial and to understand its results and the local applicability of those results. Some technical knowledge is needed to apply aspects of this checklist.

CASP tools and checklists:

http://www.casp-uk. net/#!checklists/cb36

back-to-top |

1.5. Critical appraisal of qualitative evaluations

Qualitative methods in evaluation seek to gain an in-depth understanding of people’s experiences and perspectives, using methods that are sensitive to the social context. Qualitative methods are used to examine issues such as: the needs of different groups; the context in which the programme operates; factors that contribute to successful programme delivery; how outcomes occur for different groups; and stakeholder perspectives on improvements that could be made. Qualitative data is collected using methods including interviews, focus groups, observation, and document review.

The resources below provide some guidance on assessing the methodological quality of qualitative evaluation studies.

Resource Link

U.K. Treasury (2012) Quality in qualitative evaluation: a framework for assessing research evidence (supplementary Magenta Book guidance).

Provides criteria for appraising the quality of qualitative evaluations. Criteria are organised into four principles and 18 sub-questions. For each question, quality indicators are given as a guide to help with answering the questions.

https://www.gov.uk/government/uploads /system/uploads/attachment_data/file /190986/Magenta_Book_ quality_in_qualitative_evaluation__ QQE_.pdf

Hannes K. Chapter 4: Critical appraisal of qualitative research. In: Noyes J, Booth A, Hannes K, Harden A, Harris J, Lewin S, Lockwood C (editors), Supplementary Guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions. Version 1 (2011).

Guidance for authors of Cochrane reviews on critical appraisal of qualitative studies. While some of the content is specific to the concerns of review authors, there is some widely relevant content, including:

  • using checklists as guides for identifying good practice
  • engaging a researcher familiar with qualitative methods to assist with judgement
  • links to critical appraisal checklists.

http://methods .cochrane.org/qi/ supplemental-handbook-guidance

Click on the link to Chapter 4: Critical Appraisal.

Critical Appraisal Skills Programme (CASP) Qualitative checklist.

Ten questions that can be used to understand the validity of a qualitative study and to understand its results and the local applicability of those results. Commonly used checklist.

CASP tools and checklists:

http://www.casp-uk.net/#!checklists/cb36

Long AF, Godfrey M, Randall T, Brettle AJ and Grant MJ (2002) Evaluation Tool for Qualitative Studies.

Table of questions that can be used to understand the validity, generalisability and policy and practice implications of a qualitative study. It is less widely used than the CASP checklist, but it covers some elements of validity more comprehensively.

http://usir.salford.ac.uk/ 12970/1/ Evaluation_Tool_for_Qualitative_Studies.pdf

 

back-to-top |

1.6. Critical appraisal of economic evaluations

An economic evaluation assesses value for money, cost-effectiveness, or cost-benefit. It compares the programme’s costs with its outcomes, to determine its overall worth. These evaluations often use an experimental or a quasi-experimental design, as well as methods that assign monetary values to the programme’s costs and outcomes. 

There are some very technical aspects to economic evaluations, and the resources below are those that are accessible for readers with little pre-existing knowledge of economic evaluation concepts. They provide guidance on assessing the validity and the local relevance of economic evaluations. 

Resource Link

Health Knowledge online tutorial on critical appraisal of economic evaluations.

Interactive online learning module that covers how to appraise an economic evaluation. Provides a checklist, a guided example to work through, a quiz, and links to further resources. It is healthcare-focused, so includes some health economics-specific concepts: cost utility analysis and QALYs. However, most of the content can also be applied to economic evaluations in other social sectors.

http://www.healthknowledge. org.uk/interactive-learning/fae/ economic-evaluations

Critical appraisal checklist for economic evaluations. Department of General Practice, University of Glasgow.

Adapted from the CASP Economic Evaluation Checklist, this resource contains twelve questions that can be used to assess the validity of an economic evaluation and to understand its results and the local applicability of those results. Includes a glossary of technical terms, but some technical knowledge is needed to apply aspects of the checklist.

http://www.gla.ac.uk/media/ media_64048_en.pdf

back-to-top |

02_ Sources of information on evidence-based programmes

This section lists international websites that compile and assess evidence-informed programmes and practices in the social sector. These sites can be used to:

  • get a quick overview of evidence-informed programmes and practices in a particular area

  • find out what is known about effectiveness of the practices and programmes and what strength of evidence underlies those conslusions

  • find links to further information on the programmes and practices of interest.

However, you may need to do further work to find out the issues that affect a programme’s relevance to, and its appropriateness for, implementation in your context. 

Note that different sites use different standards of evidence for the selection and rating of their programmes and practices, and it is important to understand what standards were used in each case. Note that this list of resources includes public health-related sites, but excludes those that focus exclusively on clinical interventions. 

Website URL Topic/s URL for evidence standard Description

Results First Clearinghouse Database (USA)

www.pewtrusts.org

Child, youth, and family support

Crime and justice

Health

Education

www.pewtrusts.org

One-stop resource on the effectiveness of interventions as rated by eight US clearinghouses: Blueprints for Healthy Youth Development, California Evidence-Based Clearinghouse for Child Welfare, Coalition for Evidence-Based Policy, CrimeSolutions.gov, National Registry of Evidence-based Programs and Practice, Promising Practices Network, What Works Clearinghouse, What Works in Re-entry Clearinghouse.

Washington State Institute for Public Policy (WSIPP) Benefit -Cost Results (USA)

www.wsipp.wa.gov

Child, youth, and family support

Crime and justice

Health

Education

www.wsipp.wa.gov

Cost-benefit results based on systematic reviews of evidence for what works for outcomes of interest to Washington State. Cost-benefit predictions are presented for implementation of those programmes in Washington, USA.

Social Programs that Work (USA)

evidencebased     programs.org

Child, youth, and family support

Crime and justice

Health

Education

Employment

toptierevidence.org

Searchable database of social programmes that have had their effectiveness examined using well-controlled randomised controlled trials.

Nest - What Works for Kids (Australia)

whatworksforkids.org.au

Child, youth, and family support

Health

Evidence Standard under development, and is not yet published.

Searchable database of programmes, practices and tools that aim to improve the health and wellbeing of youth. Presently there are only 12 programmes in the database, but this number is likely to increase. Programmes will be assessed against five main dimensions: evaluation quality; intervention impacts; intervention specificity; implementation readiness; and context relevance, which will incorporate assessment of the relevance to the Australian context, including cultural and population considerations.

The Hub (New Zealand)

thehub.superu.govt.nz

Child, youth, and family support

Education

Health

Crime and justice

None.

Clearinghouse of New Zealand government research and evaluation publications. While it is not limited to evidence-based programmes, and an evidence standard is not applied, it useful as a source of evaluation reports on initiatives that have run in New Zealand

Violence Prevention Evidence Base (USA)

www.preventviolence.info

Health

Crime and justice

Publication in a peer reviewed journal

Brings together abstracts and information from published studies that have measured the effectiveness of interventions to prevent violence. Included studies have been selected using a systematic review of academic literature.

The Campbell Collaboration (international)

www. campbell  collaboration. org

Child, youth, and family support

Crime and justice

Education

Varies across reviews. Each review states its inclusion criteria

International research network that produces systematic reviews of the effects of social interventions in crime and justice, education, international development, and social welfare.

Evidence for Policy and Practice Information and Co-ordinating Centre (UK)

eppi.ioe. ac.uk

Child, youth, and family support

Health

Education

Employment

Community development

Varies across reviews. Each review states its inclusion criteria

Develops and provides a searchable resource of systematic reviews, across a range of social policy areas including education, health, social care, developing economies, sport, and environment.

Communities for Children Facilitating Partners -Evidence-based programme profiles (Australia)

apps.aifs.gov. au

Child, youth, and family support

Community development

aifs.gov.au

Profiles of evidence-based programmes chosen from a range of international databases and clearinghouses, intended to be of use to Australian organisations proposing to implement evidence-based programmes.

Knowledge Circle Practice Profiles (Australia)

apps.aifs.gov.au

Child, youth, and family support

Community development

www2.aifs.gov.au

These are snapshots of programs, or elements of programs, that focus on delivering positive outcomes for Aboriginal and Torres Strait Islander children, families and communities.

What Works Centre for Local Economic Growth (UK)

www.whatworks  growth.org

Employment

Community development

www.whatworks  growth.org

Reviews evidence for the impact of different types of policies interventions on employment. Results are presented by policy area, not individual programmes.

Investing in Children (UK)

investinginchildren.eu

Child, youth, and family support

Uses WSIPP method

www.wsipp.wa.gov

Presents cost-benefit predictions for the UK, for over 100 initiatives in children’s services.

Early Intervention Foundation (UK)

guidebook.eif.org.uk

Child, youth, and family support

guidebook.eif.org.uk

Presents summaries of what works and what does not work for early intervention in tackling the root causes of social problems for children and young people. Currently has an initial set of 50 programmes in the UK, this may increase in future.

The Centre for Excellence and Outcomes in Children and Young People’s Services (UK)

www.c4eo.org.uk

Child, youth, and family support

www.c4eo.org.uk

Examples of local practices that have led to significantly improved outcomes for children, young people and their families.

California Evidence-Based Clearinghouse for Child Welfare (USA)

www.cebc4cw.org

Child, youth, and family support

www.cebc4cw.org

Database of programs that serve children and families involved with the child welfare system.

New Zealand Family Violence Clearinghouse (NZ)

nzfvc.org.nz

Child, youth, and family support

None.

Clearinghouse for publications on family and whānau violence in NZ. Publications are not restricted to those that evaluate interventions, but can be searched using the term “evaluation” in addition to your topic-specific keywords.

Blueprints Project (USA)

www.blueprints programs.com

Child, youth, and family support

www.blueprints programs.com

Database of programmes in youth development that have been reviewed by a panel and determined to meet stringent standards.

Child Trends - What Works for Child and Youth Development (USA)

www.childtrends.org

Child, youth, and family support

www.childtrends.org

Searchable register of over 700 social interventions that have had at least one randomized controlled trial to assess child or youth outcomes related to education, life skills, and social/emotional, mental, physical, behavioral, or reproductive health.

Education Endowment Centre Evidence and Data (UK)

education endowment foundation.org.uk

Education

education endowment  foundation.org.uk

Focuses on educational achievement, with two collections of programmes: the teaching and learning toolkit, which covers educational research on teaching 5-16 year olds, and the early years toolkit, which covers educational research for pre-school children.

Best Evidence Encyclopedia (USA)

www. bestevidence.org

Education

www.bestevidence.org

Information about the strength of the evidence supporting a variety of programs for students in grades K-12.

Best Evidence Encyclopedia (UK)

www. bestevidence.org.uk

Education

www. bestevidence.org.uk

Reviews of research-proven educational programmes for primary and secondary education. UK version of the US Best Evidence Encyclopedia.

Evidence 4 Impact (UK)

www. evidence4impact. org.uk

Education

www. evidence4impact.org. uk/ratings.php

Evidence summaries and ratings for educational programmes available in the UK.

What Works Clearinghouse of the Institute of Education Science (USA)

ies.ed.gov

Education

ies.ed.gov

Collates evidence for the effectiveness of programmes, practices and policies in education.

What Works Clearinghouse of the Institute of Education Science (USA)

ies.ed.gov

Education

ies.ed.gov

Collates evidence for the effectiveness of programmes, practices and policies in education.

PubMed Health (international)

www. ncbi.nlm.nih.gov

Health

Varies across reviews. Each review states its inclusion criteria

Keyword searchable collation of systematic reviews of clinical effectiveness research, including those from the Database of Abstracts of Reviews of Effects, the Cochrane Collaboration and a number of public agencies.

National Registry of Effective Prevention Programs (USA)

www. nrepp.samhsa.gov

Health

Criteria changed in September 2015. Current criteria are at:

www.nrepp.samhsa. gov

No information on criteria used from 2008- 2015

Searchable online registry of substance abuse and mental health interventions.

Guide for Community Preventive Services of the Centers for Disease Control and Prevention (USA)

www.thecommunityguide.org

Health

www.thecommunity  guide.org

Findings from the CDC’s Community Preventive Services Task Force reviews on preventative services for improving health.

Canadian Best Practices Portal (Canada)

cbpp-pcpe.phac-aspc.gc.ca

Health

Best practices:

cbpp-pcpe.phac-aspc.gc.ca

Aboriginal Ways Tried and True

cbpppcpe.phac-aspc.gc.ca

Database of evidence-based practices in disease prevention and health promotion. Practices are only included if they meet the ‘Best Practices’ or ‘Aboriginal Ways Tried and True’ criteria.

CDC Community Health Improvement Navigator (USA)

wwwn.cdc.gov

Health

innovations.ahrq.gov

Searchable database of community health improvement interventions.

Healthy Nashville - Promising Practices (USA)

www.healthynashville.org

Health

www.healthynashville.org

Database of approaches to improving community health and quality of life.

University of Wisconsin Population Health Institute’s What Works for Health database (USA)

whatworks  forhealth.wisc.edu

Health

whatworks  forhealth.wisc.edu

Assessments of policies and programmes that can impact health through changes to individual health behaviours, clinical care, social and economic factors, and the physical environment.

What Works Centre for Crime Reduction (UK)

whatworks.college. police.uk

Crime and justice

whatworks.college.police.uk

Searchable database of practices and interventions to reduce crime, with information on quality, cost, impact, mechanism, context, and implementation issues.

National Institute of Justice - crimesolutions.gov (USA)

www.crimesolutions.gov

Crime and justice

www.crimesolutions.gov

Collation of evidence-based programmes in criminal justice, juvenile justice, and crime victim services.

Office of Juvenile Justice and Delinquency Prevention Model Programs Guide (USA)

www.ojjdp.gov

Crime and justice

uses CrimeSolutions.gov’s criteria: 

www.crimesolutions.gov

Collates what works, what is promising, and what does not work in juvenile justice, delinquency prevention, and child protection and safety.

What Works in Reentry Clearinghouse (USA)

whatworks.csg justicecenter.org

Crime and justice

whatworks.csg  justicecenter.org

Collation of evidence on the effectiveness of re-entry programmes.

 

back-to-top |
    Last update: 1 Apr 2016