The Reliability of Survey Assessments of Characteristics of Medical Clinics

Objective

To assess the reliability of survey measures of organizational characteristics based on reports of single and multiple informants.

Data Source

Survey of 330 informants in 91 medical clinics providing care to HIV‐infected persons under Title III of the Ryan White CARE Act.

Study Design

Cross‐sectional survey.

Data Collection Methods

Surveys of clinicians and medical directors measured the implementation of quality improvement initiatives, priorities assigned to aspects of HIV care, barriers to providing high‐quality HIV care, and quality improvement activities. Reliability of measures was assessed using generalizability coefficients. Components of variance and clinician–director differences were estimated using hierarchical regression models with survey items and informants nested within organizations.

Principal Findings

There is substantial item‐ and informant‐related variability in clinic assessments that results in modest or low clinic‐level reliability for many measures. Directors occasionally gave more optimistic assessments of clinics than did clinicians.

Conclusions

For most measures studied, obtaining adequate reliability requires multiple informants. Using multiple‐item scales or multiple informants can improve the psychometric performance of measures of organizational characteristics. Studies of such characteristics should report the organizational level reliability of the measures used.