10 common errors in surveillance evaluations

1. Failure to clarify the objectives of the surveillance system

Description of the error The introduction does not mention the surveillance system's objectives or what they should be.

Rationale to change The objectives of public health surveillance differ for different conditions. They may include exhaustive identification of cases, outbreak detection, or description of risk factors. The evaluation exercise will determine whether the surveillance system meets its objectives. The evaluation lacks perspective/reference/standards if these objectives are unclear. The objectives of the surveillance system must be considered in view of but are different from the objectives of the program that the surveillance system is supposed to feed with data for action. For example, the objective of polio surveillance in the context of eradication is not to eradicate polio but to find 100% of cases.

2. Excessive focus on a small area

Description of the error The evaluation uses a nice statistical sampling strategy (e.g., a cluster sample) but focuses on a small area of the location of the assignment (e.g., a district).

Rationale to change A sample that is nicely representative is not of much use (i.e., poor external validity) if the results can only be generalized to a small area. In addition, a surveillance system works at many levels, from the population to the state. Thus, it is best to encompass the largest scope possible (examine what happens at the state level), even if that is done only qualitatively.

3. Insufficient description of the methods used

Description of the error The methods section contains insufficient information to understand the methods used to (1) describe the system and (2) evaluate it.

Rationale to change The evaluation of the surveillance system is a scientific process that reviews information to propose an analysis followed by recommendations. Thus, the methods used to generate the information (the description of the system and its attributes) must be described.

4. Failure to describe the system

Description of the error The document contains a careful evaluation of the system without a description of the system.

Rationale to change In a scientific process, description precedes analysis and interpretation. Before the system can be examined critically with attributes, it needs to be described in how it operates, level by level, from data collection, transmission, analysis, information feedback, and action.

5. Failure to identify the key attributes of the surveillance system

Description of the error The evaluation reviews all the attributes one by one without any appreciation of those that matter most for the system to reach its objectives.

Rationale to change The attributes that matter most vary according to the system and its objectives. In some cases, sensitivity is a key objective if the objective is case identification (e.g., polio). In some others, timeliness is a key objective if the objective is outbreak detection (e.g., meningitis). Thus, according to the objectives of the surveillance system, the focus of the evaluation will be directed to specific attributes that will be of special importance for the system to meet its objectives.

6. Absence of critical review of the attributes based on the objectives of the surveillance system

Description of the error The evaluation has an academic spin by which attributes are considered unsatisfactory if they are not perfect. There is no critical review of the attribute to determine whether its characteristics constitute an obstacle for the system to achieve its surveillance objective.

Rationale to change Neither surveillance nor surveillance evaluations are academic exercises. Surveillance is not supposed to generate perfect data but information for action. Surveillance evaluation is not supposed to point to every weakness in the surveillance system but to propose practical, feasible recommendations so that surveillance information can be used for action. Hence, rather than asking: 'Is the system 100% sensitive', the question should be: 'Is the system sensitive enough to meet its objectives.'

7. Insufficient documentation of the attributes

Description of the error Insufficient data is presented to support the statements made about the various attributes of the system.

Rationale to change Statements about a system's various attributes need to be backed up with qualitative or quantitative data. There is a trade-off between the time and resources to invest in the precise documentation of an attribute and the incremental value brought by such efforts. When the attribute is particularly relevant to the system consider (e.g., sensitivity to measles), special efforts (e.g., survey) are justified to obtain a high-quality estimate. When the attribute is less relevant, a qualitative appreciation is possible. However, the author must not place excessive confidence in data of poor quality.

8. Confusion between the surveillance system evaluation and program evaluation

Description of the error There is confusion between the evaluation of the surveillance system and the evaluation of the program being fed with data by the surveillance system.

Rationale to change Surveillance (the ongoing collection, transmission and analysis of data for feedback and action) is limited to information management. Program is broader and addresses prevention, control, care etc... Surveillance is evaluated using the classical attributes, while programs are evaluated using a different framework (i.e., input, process and outcomes...) The two activities should not be mixed. Programs with a substantial case search component (e.g., tuberculosis, sleeping sickness, leishmaniasis) may be confusing, as some persons will call case search "surveillance".

9. Confusion between the limitations of the surveillance system and the limitations of its evaluation

Description of the error The limitation section of the discussion discusses the limitations of the surveillance system.

Rationale to change The limitation section of the discussion section of a surveillance evaluation discusses the limitation of the evaluation process itself. The limitation of the surveillance system should be the focus of the main part of the discussion section.

10. Poor recommendations

Description of the error The recommendations are weak. They are (1) too general, (2) not based upon the data or (3) not feasible.

Rationale to change The purpose of evaluating the surveillance system is to propose recommendations for its improvement. These recommendations must come from the evidence generated by the evaluation. They need to be specific and precise. They need to be feasible and focused.

FEM PAGE CONTRIBUTORS 2007

Editor

Naomi Boxall

Original Author

Alain Moren

FEM Contributors

Sabrina Bacci

Lisa Lazareck

hccdejonge

Arnold Bosman

Naomi Boxall