EPSY 5221
A Model of Item Validity Evidence: Qualitative Evidence
Type of Evidence |
Activity |
Evidence Needed |
Content Definition |
Role delineation, task/job analysis; practice analysis completed |
Documentation of the method(s) used to select item content |
Test Specifications |
Table of specifications/test blueprint created |
Documentation of systematic link of test content to test specifications / test blueprint |
Item Writer Training |
Develop training materials and methods; train item writers |
Documentation of methods, principles, written materials, and sample items |
Adherence to Item-Writing Principles |
Standard item-writing rules adopted |
Evidence of compliance with rules and documentation of process used to review items |
Cognitive Behavior |
Cognitive classification system used to classify items |
Documentation of system used and its rationale; reports of any research using system |
Item Content Verification |
Content experts review and judge items |
Content experts’ credentials’ records of content-expert review process |
Item Editing |
Review items and professionally edit |
Credentials and experience of editors; editorial and style guidelines, documentation of edit/review cycle |
Bias/Sensitivity Review |
Bias/Sensitivity review policies and procedures developed |
Documentation of bias/sensitivity review; rationale for policies; credentials of reviewers |
Item Tryout / Pretesting |
Pretest/pilot test/field test items; item performance data; examinee interviews |
Documentation of examinee pilot test data; examinee and item characteristics |
Key Validation/ Verification |
Correctness of keyed answer verified by panel of content experts |
Policy and procedures for key verification; documentation of key validation results |
Test Security Plan |
A test security policy and set of procedures is developed |
Copy of policy and procedures manual that specifies how items are protected from security lapses |
Downing, S. M., & Haladyna, T. M. (1997). Test item development: Validity evidence from quality assurance procedures. Applied Measurement in Education, 10(1), 61-82.