Evolution of benchmarks

The Phidisa Project is a government-sponsored trial that does not seek to market a particular drug. The primary aims of benchmarking in this setting are to improve data quality and increase subject safety. The benchmarking strategy in this project is designed to establish site goals, identify areas for improvement, highlight achievements, and utilize information to drive process enhancements.

In accordance with Phidisa protocols and regulatory requirements, Phidisa active sites are monitored at regular intervals. During routine monitoring visits, recorded data is reviewed and validated, processes implemented at sites are assessed, and protocol compliance and adherence to regulatory requirements are evaluated. Findings are documented in a monitoring report and submitted to the sponsor through the regulatory department.

Monitoring reports were detailed but lacked a quantitative assessment of site performance. In addition, as the study progressed and sites were added, quantitative assessments were needed to ensure that key protocol activities were being performed according to comparable standards across all sites. Therefore, based on monitoring findings, regulatory requirements, and GCP fundamentals, the benchmark assessment tool was developed. It consists of 12 parameters that can be evaluated and reported quantitatively, and each parameter is measured against a target standard as shown in Table 1.

Current evaluation process

Benchmark assessments are completed during each monitoring visit, tabulated, reported to the clinical site team during the exit meeting, and commented on in the final monitoring visit report. Benchmarks are assessed as follows:

• Informed consent process. The number of new informed consent forms (ICFs) completed correctly (according to protocol, Phidisa SOPs, and GCP requirements) is compared to the total number of new ICFs obtained since the last monitoring visit.
• Unreported protocol violations. The number of protocol violations properly reported by the site is compared to those found while reviewing the CRFs and source data during the monitoring visit.
• Reportable events. The number of properly reported adverse events (AEs) and serious adverse events (SAEs) compared to those found while reviewing the CRFs and source data during the monitoring visit.
• Study endpoints. The number of properly reported study endpoints (reported on the same CRF page) compared to those found while reviewing the CRFs and source data during the monitoring visit.
• CRF completion and submission. The number of CRFs received by data management (DM) as compared to the number of CRFs expected (according to visit schedules). This is determined by DM.
• Drug accountability. Pharmacy drug accountability records are compared to actual drug counts for 23 antiretrovirals dispensed to participants. Details of discrepancies are given to pharmacy staff and clarifications taken into consideration before the benchmark is finalized.
• Action on critical alert laboratory findings. This assessment is based on documentation of appropriate subject management after the site investigator is informed of a critical laboratory value versus alerts not acted on appropriately.
• Monitoring queries. Queries remaining on site following a monitoring visit are evaluated during the subsequent visit for completion. The number of queries that have been completed is compared to the number of queries generated.
• DM queries generated. The number of errors or queries generated by DM on receipt of CRF pages is compared to the number of CRF pages received for a given period.
• Data management queries addressed. The number of queries generated versus the number of queries answered during a given period—it usually precedes the period covered by DM's query assessments.
• Participant visit attendance. Identifies the number of participants who did not have data collected within the defined visit window period. Monitors check to see if data were collected but not submitted versus data not collected.
• Follow-up of participants who miss scheduled visits. If a study visit was not honored, the source documentation should show that the site staff have implemented adequate measures for follow-up in order to determine the cause for nonattendance and to arrange a subsequent visit date.
• Participants lost to follow-up. The cumulative number of participants lost to follow-up is compared to the number of participants enrolled.

Benefits of objective benchmarks

Project Phidisa is a large, multicenter, multisponsor, multicountry clinical research project. There has been much debate as to whether cumbersome, lengthy monitoring reports documenting monitors' activities truly assist in assuring subject safety, human subject protection, and data integrity. Given that capacity building is one of the priorities for Phidisa, identifying areas for improvement is considered critical to the success of the program.

The benchmark assessment tool was developed to record quantitative evaluations in crucial areas. Each of these areas reflects critical components of GCP compliance, participant safety, and data integrity. Primarily, the tool was used to evaluate performance and target areas of improvement at the largest recruitment sites (greater than 1000 participants in two years). The outcome of the assessment provided tangible results that were used to target corrective action in areas needing improvement not only at the site level but also within the various sections or departments that supported the trial. The assessments were repeated at regular intervals to evaluate the success of corrective action and improvement at the site. Depending on the outcome, further changes were made to ensure the site could attain the standards set by the benchmarks.

Benchmark case study one. Benchmark assessments initiated in July 2005 quantified the backlog in the resolution of monitoring queries. Underlying causes were identified and corrective action implemented. There was a drop in monitoring query resolution at the site in August and September 2005 while corrective actions were being implemented. A directed, concerted effort was required by all site staff, augmented by a team of experts to resolve the issues. One of the most important lessons learned was that early warning signs handled proactively can prevent recurrences of backlogs.

Figure 1. Sequential measurement of site performance related to monitoring queries shows improvement over time.

Benchmark case study two. In the Phidisa protocols, subjects are scheduled to return for follow-up visits at intervals ranging from one month to 12 months depending on the protocol and stage of the protocol. After the initial months of rapid recruitment, the site's focus moved from recruitment to retention of subjects and assessment of nonattendance and reasons for it.

Figure 2. This benchmark assessment provides a valuable tool to determine whether site procedures to follow up with subjects on missed visits are being implemented effectively. If site management is adequate, then other underlying problems such as protocol design could be the cause for poor visit compliance.