Difference between revisions of "10 common errors in surveillance evaluations"
Bosmana fem (talk | contribs) m |
Bosmana fem (talk | contribs) m (→10. Poor recommendations) |
||
| (4 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
==1. Failure to clarify the objectives of the surveillance system== | ==1. Failure to clarify the objectives of the surveillance system== | ||
| − | ''Description of the error'' | + | '''Description of the error''' |
The introduction does not mention the surveillance system's objectives or what they should be. | The introduction does not mention the surveillance system's objectives or what they should be. | ||
'''Rationale to change''' | '''Rationale to change''' | ||
| − | The objectives of public health surveillance differ for different conditions. They may include exhaustive identification of cases, outbreak detection or description of risk factors. The evaluation exercise will determine whether the surveillance system meets its objectives. | + | The objectives of public health surveillance differ for different conditions. They may include exhaustive identification of cases, outbreak detection, or description of risk factors. The evaluation exercise will determine whether the surveillance system meets its objectives. The evaluation lacks perspective/reference/standards if these objectives are unclear. The objectives of the surveillance system must be considered in view of but are different from the objectives of the program that the surveillance system is supposed to feed with data for action. For example, the objective of polio surveillance in the context of eradication is not to eradicate polio but to find 100% of cases. |
| − | 2. Excessive focus on a small area | + | ==2. Excessive focus on a small area== |
| + | '''Description of the error''' | ||
| + | The evaluation uses a nice statistical sampling strategy (e.g., a cluster sample) but focuses on a small area of the location of the assignment (e.g., a district). | ||
| − | + | '''Rationale to change''' | |
| + | A sample that is nicely representative is not of much use (i.e., poor external validity) if the results can only be generalized to a small area. In addition, a surveillance system works at many levels, from the population to the state. Thus, it is best to encompass the largest scope possible (examine what happens at the state level), even if that is done only qualitatively. | ||
| − | The | + | ==3. Insufficient description of the methods used== |
| + | '''Description of the error''' | ||
| + | The methods section contains insufficient information to understand the methods used to (1) describe the system and (2) evaluate it. | ||
| − | Rationale to change | + | '''Rationale to change''' |
| + | The evaluation of the surveillance system is a scientific process that reviews information to propose an analysis followed by recommendations. Thus, the methods used to generate the information (the description of the system and its attributes) must be described. | ||
| − | + | ==4. Failure to describe the system== | |
| + | '''Description of the error''' | ||
| + | The document contains a careful evaluation of the system without a description of the system. | ||
| − | + | '''Rationale to change''' | |
| + | In a scientific process, description precedes analysis and interpretation. Before the system can be examined critically with attributes, it needs to be described in how it operates, level by level, from data collection, transmission, analysis, information feedback, and action. | ||
| − | Description of the error | + | ==5. Failure to identify the key attributes of the surveillance system== |
| + | '''Description of the error''' | ||
| + | The evaluation reviews all the attributes one by one without any appreciation of those that matter most for the system to reach its objectives. | ||
| − | The | + | '''Rationale to change''' |
| + | The attributes that matter most vary according to the system and its objectives. In some cases, sensitivity is a key objective if the objective is case identification (e.g., polio). In some others, timeliness is a key objective if the objective is outbreak detection (e.g., meningitis). Thus, according to the objectives of the surveillance system, the focus of the evaluation will be directed to specific attributes that will be of special importance for the system to meet its objectives. | ||
| − | + | ==6. Absence of critical review of the attributes based on the objectives of the surveillance system== | |
| + | '''Description of the error''' | ||
| + | The evaluation has an academic spin by which attributes are considered unsatisfactory if they are not perfect. There is no critical review of the attribute to determine whether its characteristics constitute an obstacle for the system to achieve its surveillance objective. | ||
| − | + | '''Rationale to change''' | |
| − | + | Neither surveillance nor surveillance evaluations are academic exercises. Surveillance is not supposed to generate perfect data but information for action. Surveillance evaluation is not supposed to point to every weakness in the surveillance system but to propose practical, feasible recommendations so that surveillance information can be used for action. Hence, rather than asking: 'Is the system 100% sensitive', the question should be: 'Is the system sensitive enough to meet its objectives.' | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | Rationale to change | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | Neither surveillance nor surveillance | ||
| − | |||
| − | |||
| − | |||
| − | |||
| + | == 7. Insufficient documentation of the attributes== | ||
| + | '''Description of the error''' | ||
Insufficient data is presented to support the statements made about the various attributes of the system. | Insufficient data is presented to support the statements made about the various attributes of the system. | ||
| − | Rationale to change | + | '''Rationale to change''' |
| + | Statements about a system's various attributes need to be backed up with qualitative or quantitative data. There is a trade-off between the time and resources to invest in the precise documentation of an attribute and the incremental value brought by such efforts. When the attribute is particularly relevant to the system consider (e.g., sensitivity to measles), special efforts (e.g., survey) are justified to obtain a high-quality estimate. When the attribute is less relevant, a qualitative appreciation is possible. However, the author must not place excessive confidence in data of poor quality. | ||
| − | + | ==8. Confusion between the surveillance system evaluation and program evaluation== | |
| + | '''Description of the error''' | ||
| + | There is confusion between the evaluation of the surveillance system and the evaluation of the program being fed with data by the surveillance system. | ||
| − | + | '''Rationale to change''' | |
| − | + | Surveillance (the ongoing collection, transmission and analysis of data for feedback and action) is limited to information management. Program is broader and addresses prevention, control, care etc... Surveillance is evaluated using the classical attributes, while programs are evaluated using a different framework (i.e., input, process and outcomes...) The two activities should not be mixed. Programs with a substantial case search component (e.g., tuberculosis, sleeping sickness, leishmaniasis) may be confusing, as some persons will call case search "surveillance". | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | Rationale to change | ||
| − | |||
| − | Surveillance (the ongoing collection, transmission and analysis of data for feedback and action) is limited to information management. | ||
| − | |||
| − | |||
| − | |||
| − | |||
| + | ==9. Confusion between the limitations of the surveillance system and the limitations of its evaluation== | ||
| + | '''Description of the error''' | ||
The limitation section of the discussion discusses the limitations of the surveillance system. | The limitation section of the discussion discusses the limitations of the surveillance system. | ||
| − | Rationale to change | + | '''Rationale to change''' |
| − | |||
The limitation section of the discussion section of a surveillance evaluation discusses the limitation of the evaluation process itself. The limitation of the surveillance system should be the focus of the main part of the discussion section. | The limitation section of the discussion section of a surveillance evaluation discusses the limitation of the evaluation process itself. The limitation of the surveillance system should be the focus of the main part of the discussion section. | ||
| − | 10. Poor recommendations | + | ==10. Poor recommendations== |
| + | '''Description of the error''' | ||
| + | The recommendations are weak. They are (1) too general, (2) not based upon the data or (3) not feasible. | ||
| − | + | '''Rationale to change''' | |
| − | + | The purpose of evaluating the surveillance system is to propose recommendations for its improvement. These recommendations must come from the evidence generated by the evaluation. They need to be specific and precise. They need to be feasible and focused. | |
| − | |||
| − | |||
| − | + | <div style="display: inline-block; width: 25%; vertical-align: top; border: 1px solid #000; background-color: #d7effc; padding: 10px; margin: 5px;"> | |
| + | '''FEM PAGE CONTRIBUTORS 2007''' | ||
| + | ;Editor | ||
| + | :Naomi Boxall | ||
| + | ;Original Author | ||
| + | :Alain Moren | ||
| + | ;FEM Contributors | ||
| + | :Sabrina Bacci | ||
| + | :Lisa Lazareck | ||
| + | :hccdejonge | ||
| + | :Arnold Bosman | ||
| + | :Naomi Boxall | ||
| + | </div> | ||
| + | [[Category:Evaluation of Surveillance Systems]] | ||
Latest revision as of 21:13, 10 April 2023
Contents
- 1 1. Failure to clarify the objectives of the surveillance system
- 2 2. Excessive focus on a small area
- 3 3. Insufficient description of the methods used
- 4 4. Failure to describe the system
- 5 5. Failure to identify the key attributes of the surveillance system
- 6 6. Absence of critical review of the attributes based on the objectives of the surveillance system
- 7 7. Insufficient documentation of the attributes
- 8 8. Confusion between the surveillance system evaluation and program evaluation
- 9 9. Confusion between the limitations of the surveillance system and the limitations of its evaluation
- 10 10. Poor recommendations
1. Failure to clarify the objectives of the surveillance system
Description of the error The introduction does not mention the surveillance system's objectives or what they should be.
Rationale to change The objectives of public health surveillance differ for different conditions. They may include exhaustive identification of cases, outbreak detection, or description of risk factors. The evaluation exercise will determine whether the surveillance system meets its objectives. The evaluation lacks perspective/reference/standards if these objectives are unclear. The objectives of the surveillance system must be considered in view of but are different from the objectives of the program that the surveillance system is supposed to feed with data for action. For example, the objective of polio surveillance in the context of eradication is not to eradicate polio but to find 100% of cases.
2. Excessive focus on a small area
Description of the error The evaluation uses a nice statistical sampling strategy (e.g., a cluster sample) but focuses on a small area of the location of the assignment (e.g., a district).
Rationale to change A sample that is nicely representative is not of much use (i.e., poor external validity) if the results can only be generalized to a small area. In addition, a surveillance system works at many levels, from the population to the state. Thus, it is best to encompass the largest scope possible (examine what happens at the state level), even if that is done only qualitatively.
3. Insufficient description of the methods used
Description of the error The methods section contains insufficient information to understand the methods used to (1) describe the system and (2) evaluate it.
Rationale to change The evaluation of the surveillance system is a scientific process that reviews information to propose an analysis followed by recommendations. Thus, the methods used to generate the information (the description of the system and its attributes) must be described.
4. Failure to describe the system
Description of the error The document contains a careful evaluation of the system without a description of the system.
Rationale to change In a scientific process, description precedes analysis and interpretation. Before the system can be examined critically with attributes, it needs to be described in how it operates, level by level, from data collection, transmission, analysis, information feedback, and action.
5. Failure to identify the key attributes of the surveillance system
Description of the error The evaluation reviews all the attributes one by one without any appreciation of those that matter most for the system to reach its objectives.
Rationale to change The attributes that matter most vary according to the system and its objectives. In some cases, sensitivity is a key objective if the objective is case identification (e.g., polio). In some others, timeliness is a key objective if the objective is outbreak detection (e.g., meningitis). Thus, according to the objectives of the surveillance system, the focus of the evaluation will be directed to specific attributes that will be of special importance for the system to meet its objectives.
6. Absence of critical review of the attributes based on the objectives of the surveillance system
Description of the error The evaluation has an academic spin by which attributes are considered unsatisfactory if they are not perfect. There is no critical review of the attribute to determine whether its characteristics constitute an obstacle for the system to achieve its surveillance objective.
Rationale to change Neither surveillance nor surveillance evaluations are academic exercises. Surveillance is not supposed to generate perfect data but information for action. Surveillance evaluation is not supposed to point to every weakness in the surveillance system but to propose practical, feasible recommendations so that surveillance information can be used for action. Hence, rather than asking: 'Is the system 100% sensitive', the question should be: 'Is the system sensitive enough to meet its objectives.'
7. Insufficient documentation of the attributes
Description of the error Insufficient data is presented to support the statements made about the various attributes of the system.
Rationale to change Statements about a system's various attributes need to be backed up with qualitative or quantitative data. There is a trade-off between the time and resources to invest in the precise documentation of an attribute and the incremental value brought by such efforts. When the attribute is particularly relevant to the system consider (e.g., sensitivity to measles), special efforts (e.g., survey) are justified to obtain a high-quality estimate. When the attribute is less relevant, a qualitative appreciation is possible. However, the author must not place excessive confidence in data of poor quality.
8. Confusion between the surveillance system evaluation and program evaluation
Description of the error There is confusion between the evaluation of the surveillance system and the evaluation of the program being fed with data by the surveillance system.
Rationale to change Surveillance (the ongoing collection, transmission and analysis of data for feedback and action) is limited to information management. Program is broader and addresses prevention, control, care etc... Surveillance is evaluated using the classical attributes, while programs are evaluated using a different framework (i.e., input, process and outcomes...) The two activities should not be mixed. Programs with a substantial case search component (e.g., tuberculosis, sleeping sickness, leishmaniasis) may be confusing, as some persons will call case search "surveillance".
9. Confusion between the limitations of the surveillance system and the limitations of its evaluation
Description of the error The limitation section of the discussion discusses the limitations of the surveillance system.
Rationale to change The limitation section of the discussion section of a surveillance evaluation discusses the limitation of the evaluation process itself. The limitation of the surveillance system should be the focus of the main part of the discussion section.
10. Poor recommendations
Description of the error The recommendations are weak. They are (1) too general, (2) not based upon the data or (3) not feasible.
Rationale to change The purpose of evaluating the surveillance system is to propose recommendations for its improvement. These recommendations must come from the evidence generated by the evaluation. They need to be specific and precise. They need to be feasible and focused.
FEM PAGE CONTRIBUTORS 2007
- Editor
- Naomi Boxall
- Original Author
- Alain Moren
- FEM Contributors
- Sabrina Bacci
- Lisa Lazareck
- hccdejonge
- Arnold Bosman
- Naomi Boxall
Root > Assessing the burden of disease and risk assessment > Epidemic intelligence > Surveillance principles > Indicator-based Surveillance > Establishing a Surveillance System > Evaluation of Surveillance Systems
