EPSY 5221

A Model of Item Validity Evidence: Qualitative Evidence

Type of Evidence

Activity

Evidence Needed

Content Definition

Role delineation, task/job analysis; practice analysis completed

Documentation of the method(s) used to select item content

Test Specifications

Table of specifications/test blueprint created

Documentation of systematic link of test content to test specifications / test blueprint

Item Writer Training

Develop training materials and methods; train item writers

Documentation of methods, principles, written materials, and sample items

Adherence to Item-Writing Principles

Standard item-writing rules adopted

Evidence of compliance with rules and documentation of process used to review items

Cognitive Behavior

Cognitive classification system used to classify items

Documentation of system used and its rationale; reports of any research using system

Item Content Verification

Content experts review and judge items

Content experts’ credentials’ records of content-expert review process

Item Editing

Review items and professionally edit

Credentials and experience of editors; editorial and style guidelines, documentation of edit/review cycle

Bias/Sensitivity Review

Bias/Sensitivity review policies and procedures developed

Documentation of bias/sensitivity review; rationale for policies; credentials of reviewers

Item Tryout / Pretesting

Pretest/pilot test/field test items; item performance data; examinee interviews

Documentation of examinee pilot test data; examinee and item characteristics

Key Validation/ Verification

Correctness of keyed answer verified by panel of content experts

Policy and procedures for key verification; documentation of key validation results

Test Security Plan

A test security policy and set of procedures is developed

Copy of policy and procedures manual that specifies how items are protected from security lapses

Downing, S. M., & Haladyna, T. M. (1997).  Test item development: Validity evidence from quality assurance procedures.  Applied Measurement in Education, 10(1), 61-82.