Stage 4: Appraising evidence
The quality of evidence is the confidence in the veracity of the information or data, and depends on the source, design and quality of each study or piece of information. In contrast with EBM where randomised controlled trials are ranked highest and observational studies ranked lowest, in rapid risk assessment the evidence may be limited and therefore there may be greater reliance on observational studies, including case reports and specialist expert knowledge. For most infectious disease threats only observational data are available.
Certain factors affect the quality of evidence. Factors that may increase the quality include: the method of generating data and study design (i.e. analytical epidemiology versus descriptive), the strength of association, evidence of dose response, and consistency with other studies/expert opinion. Factors that may decrease the quality include: reporting bias, inconsistency, and conflicting evidence/opinion.
Ideally, a rapid risk assessment should not rely on a single study or piece of evidence. There should be a cautious approach to interpreting information if only one research group reports on an infection or disease association in multiple publications. Poor evidence or information should not be used for the rapid risk assessment unless this is the only data available; in this case any uncertainties should be documented in the information table.
Triangulation is a technique widely used in qualitative research to address internal validity by using more than one data collection method to answer a research question. The body of evidence should be considered as a whole, and the triangulation of evidence should confirm (or refute) internal validity of findings. Triangulation of evidence,including specialist expert knowledge, may be important to reach a consensus. Ensure a minimum of two to threedata sources and agreement between these (i.e. two experts or expert and literature). Sources of evidence and agreement between these (or absence of) should be clearly stated in the information table. Based on consistency, relevance and external validity of the available and relevant information the quality of evidence is graded as: good, satisfactory, or unsatisfactory (definitions and examples are given in Checklist 3).
Checklist 3: Evaluating the quality of evidence (for information tables)
| Quality of evidence = confidence in information; design, quality, and other factors assessed and judged on consistency, relevance, and validity. Grade: good, satisfactory, unsatisfactory | Examples of types of information/evidence |
|---|---|
| Good - Further research is unlikely to change confidence in the information. |
|
| Satisfactory - Further research will likely impact information confidence and may change assessment. |
|
| Unsatisfactory - Further research is likely to impact the confidence of information and is likely to change assessment. |
|
