Notes on Precision, Accuracy, Confounds, and Effect Modifiers The following slides accompanied a presentation delivered at ASHA's Clinical Practice Research Institute. ... Presentation Slides
Research Design and Methods > Validity and Bias    >   Presentation Slides
Free
Presentation Slides  |   March 01, 2011
Notes on Precision, Accuracy, Confounds, and Effect Modifiers
Author Affiliations & Notes
  • Randall R. Robey
    University of Virginia
  • Originally presented at the Clinical Practice Research Institute (CPRI). Hosted by the American Speech-Language-Hearing Association Research Mentoring Network.
    Originally presented at the Clinical Practice Research Institute (CPRI). Hosted by the American Speech-Language-Hearing Association Research Mentoring Network.×
Article Information
Research Issues, Methods & Evidence-Based Practice / Research Design and Methods / Validity and Bias
Presentation Slides   |   March 01, 2011
Notes on Precision, Accuracy, Confounds, and Effect Modifiers
CREd Library, March 2011, doi:10.1044/cred-vb-ps-cpri006
CREd Library, March 2011, doi:10.1044/cred-vb-ps-cpri006

The following slides accompanied a presentation delivered at ASHA's Clinical Practice Research Institute.

Precision and Accuracy
Adapted from Hulley, et al. (2007) .
Just about every statistic (descriptive and inferential) is an expression of a basic ratio.
In the numerator goes a measurement of the influence of the research hypothesis.
In the denominator goes a measurement of the influence of error.
The ratio is very much like signal:noise.
This series of notes presumes a really good research hypothesis and centers on enhancing signal and reducing noise.
Experimental Precision
Experimental precision is closely related to, among other issues, reliability.
Imprecision is brought about by random error.
Enhancing precision is akin to reducing noise in your system.
Sources of random error variance
  • Observers

  • Instruments

  • Participants

Preventative steps (minimize opportunities for chance at play)
  • Standardize observation protocols Make the protocols as explicit, simple, and straightforward as is possible

  • Standardize interventions (consider a treatment algorithm)

These translate to writing an Operations Manual.
  • Train observers and clinicians to criterion

  • Simplify all instructions for clarity

  • Use pooled observations if possible

  • Assess and report reliability within an experiment

Experimental Accuracy
Experimental accuracy is a function of validity
Inaccuracy is brought about by systematic error -- AKA bias
Enhancing accuracy is brought about by increasing the signal in your system
Sources of systematic error variance
  • Observer bias -- e.g., they become more expert with experience, they have knowledge of the hypothesis

  • Instrument bias -- e.g., ceiling or floor effects, under representation of construct

  • Participant bias -- e.g., selection bias, differential attrition, compensatory rivalry, maturation

These are nothing more than Cook & Campbell's (1979)  various threats to experimental validities.
Preventative steps (minimize opportunities for chance at play)
  • Parsimony in admitting variables into an experiment and calibrate instruments

  • Streamline and standardize observation and intervention protocols

  • Make unobtrusive measurements

  • Take great care forming a protocol for participant selection

  • Take great care forming a protocol for participant allocation

  • Take great care forming a protocol for interim analyses

  • Blind as possible

Graphical Examples

Good precision, good accuracy

Good precision, good accuracy

Good precision, good accuracy

×

Good precision, poor accuracy

Good precision, poor accuracy

Good precision, poor accuracy

×

Poor precision, good accuracy

Poor precision, good accuracy

Poor precision, good accuracy

×

Poor precision, poor accuracy

Poor precision, poor accuracy

Poor precision, poor accuracy

×
Confounds and Effect Modifiers
Adapted from Portney & Watkins (2009) .
That which is screwed up in design cannot be fixed through analysis.

A straightforward challenge in logical conclusion

A straightforward challenge in logical conclusion

A straightforward challenge in logical conclusion

×
Confounders and effect modifiers make rival explanations for the results
AKA a threat to internal validity
Confounders
A confounder is an extraneous or nuisance variable
  • It is associated with the antecedent -- co-occurs to some extent

  • It is associated with the consequent – exacerbates or suppresses the outcome

Confounders are fundamentally the threats to internal validity first described by Cook & Campbell (1979) 
  • Under representation of the population

  • Misrepresentation of the population

  • Allocation Imbalance

  • (Differential) history

  • (Differential) maturation

  • (Differential) testing

  • (Differential) regression

  • Diffusion of treatment

  • Compensatory equalization of control treatment

  • Compensatory rivalry

Effect Modifiers
Effect modifiers are not a nuisance; they are not extraneous
Effect modifiers are variables that moderate the relationship between antecedent and consequent
Remedies
Requires a priori identification of potential confounders that are measured for later uses needed.
  • Unbiased recruiting independent of assignments

  • Random assignment

    Monitoring and breaking protocol

    Representativeness

  • Blinding (as possible)

    • Participant

    • Clinician

    • Analyst

  • Intention-to-treat analysis

    • Completer analysis

    • Non-completion = fail

    • Last observation brought forward

    • Persuade a return for post testing only

  • Matching

  • Narrow inclusion criteria

  • Stratification

  • Analysis of covariance

Resources for Assessing Quality Indicators

Case Reports and Case Series

Jabs, D. A. (2005). Improving the reporting of clinical case series. American Journal of Ophthalmology, 139(5), 900–905 [Article] [PubMed]
Jabs, D. A. (2005). Improving the reporting of clinical case series. American Journal of Ophthalmology, 139(5), 900–905 [Article] [PubMed]×
Vandenbroucke, J. P. (2001). In defense of case reports and case series. Annals of Internal Medicine, 134(4), 330–334 [Article] [PubMed]
Vandenbroucke, J. P. (2001). In defense of case reports and case series. Annals of Internal Medicine, 134(4), 330–334 [Article] [PubMed]×

Cross Sectional Designs

Institute of Social and Preventive Medicine (ISPM). (2007). STROBE Checklist for Cross-Sectional Studies. STROBE Statement (Available at www.strobe-statement.org).
Institute of Social and Preventive Medicine (ISPM). (2007). STROBE Checklist for Cross-Sectional Studies. STROBE Statement (Available at www.strobe-statement.org).×

Case Control Designs

Institute of Social and Preventive Medicine (ISPM). (2007). STROBE Checklist for Case-Control Studies. STROBE Statement (Available at www.strobe-statement.org).
Institute of Social and Preventive Medicine (ISPM). (2007). STROBE Checklist for Case-Control Studies. STROBE Statement (Available at www.strobe-statement.org).×

Cohort Designs

Institute of Social and Preventive Medicine (ISPM). (2007). STROBE Checklist for Cohort Studies. STROBE Statement (Available at www.strobe-statement.org).
Institute of Social and Preventive Medicine (ISPM). (2007). STROBE Checklist for Cohort Studies. STROBE Statement (Available at www.strobe-statement.org).×

Parallel-Groups RCTs

Moher, D., Schulz, K. F. & Altman, D. G. for the CONSORT Group (2001). The CONSORT statement: Revised recommendations for improving the quality of reports of parallel group randomized trials. BMC Medical Research Methodology, 1(1), 2 [Article] [PubMed]
Moher, D., Schulz, K. F. & Altman, D. G. for the CONSORT Group (2001). The CONSORT statement: Revised recommendations for improving the quality of reports of parallel group randomized trials. BMC Medical Research Methodology, 1(1), 2 [Article] [PubMed]×
Consolidated Standards of Reporting Trials (CONSORT). (2010). CONSORT Website. (Available at www.consort-statement.org).
Consolidated Standards of Reporting Trials (CONSORT). (2010). CONSORT Website. (Available at www.consort-statement.org).×

Parallel Groups CTs

Des Jarlais, D. C., Lyles, C. & Crepaz, N. for the TREND Group (2004). Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: The TREND statement. American Journal of Public Health, 94(3), 361–366 [Article] [PubMed]
Des Jarlais, D. C., Lyles, C. & Crepaz, N. for the TREND Group (2004). Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: The TREND statement. American Journal of Public Health, 94(3), 361–366 [Article] [PubMed]×
Centers for Disease Control and Prevention. (2004). Transparent Reporting of Evaluations with Nonrandomized Designs (TREND). (Available at www.cdc.gov/trendstatement).
Centers for Disease Control and Prevention. (2004). Transparent Reporting of Evaluations with Nonrandomized Designs (TREND). (Available at www.cdc.gov/trendstatement).×

Cross Over RCTs

Senn, S., (2002). Crossover Designs in Clinical Research. Wiley.
Senn, S., (2002). Crossover Designs in Clinical Research. Wiley.×

Single-Subject Trials

Chambless, D. L. & Hollon, S. D. (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology, 66(1), 7 [Article] [PubMed]
Chambless, D. L. & Hollon, S. D. (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology, 66(1), 7 [Article] [PubMed]×
References
Cook, T.D. & Campbell, D.T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Boston, MA: Houghton Mifflin Company.
Cook, T.D. & Campbell, D.T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Boston, MA: Houghton Mifflin Company. ×
Hulley, S., Cummings, S., Browner, W., Grady, D. & Newman, T. (2007). Designing Clinical Research. Philadelphia: Wolters Kluwer.
Hulley, S., Cummings, S., Browner, W., Grady, D. & Newman, T. (2007). Designing Clinical Research. Philadelphia: Wolters Kluwer. ×
Portney, L.G. & Watkins, M.P. (2009). Foundations of Clinical Research: Applications to Practice (3rd Ed.). Upper Saddle River, NJ: Pearson.
Portney, L.G. & Watkins, M.P. (2009). Foundations of Clinical Research: Applications to Practice (3rd Ed.). Upper Saddle River, NJ: Pearson. ×

Good precision, good accuracy

Good precision, good accuracy

Good precision, good accuracy

×

Good precision, poor accuracy

Good precision, poor accuracy

Good precision, poor accuracy

×

Poor precision, good accuracy

Poor precision, good accuracy

Poor precision, good accuracy

×

Poor precision, poor accuracy

Poor precision, poor accuracy

Poor precision, poor accuracy

×

A straightforward challenge in logical conclusion

A straightforward challenge in logical conclusion

A straightforward challenge in logical conclusion

×