Psychometric Consulting
EYEBROW TEXT
Psychometric Services for Certification and Licensing Bodies
Defensible, rigorous psychometrics are the foundation of any credentialing program. The Caviart Group test developers and psychometricians bring nearly four decades of experience in constructing high-stakes certification and licensure examinations to meet program needs and industry best practices. We design and develop programs that meet and exceed NCCA, ANSI, and ISO accreditation standards.
We offer a variety of psychometric services, from standard setting to comprehensive evaluations of examination performance, all to ensure the programs we support are designed and developed to be legally defensible as valid assessments of candidate ability.
Standard Setting /Cut Score Studies
Establishing a cut score is a complex decision, one that must take into account the stakeholder judgment, psychometric data and the environment in which the examination program is being offered. We offer a range of psychometric approaches to standard setting, including the following:
-
We use a panel of subject matter experts (SMEs) to review and score each examination item on its level of difficulty, determining which percentage of minimally competent practitioners would answer the question correctly.
-
This method builds on the Angoff method by adding discussion between subject matter experts on why they gave an item a particular rating. This discussion can help create more consistent results as SMEs may update their initial rating after the insight provided through the discussion.
-
In this method, exam items are arranged from easiest to hardest and a panel of subject matter experts determines where the “bookmark” is between a minimally competent practitioner passing and failing the examination.
-
For existing examinations, the most efficient way to establish an appropriate cut score is to measure the difficulty of each new test against the level of difficulty of previous examinations using a statistical process known as equating. This ensures that any differences in test difficulty from exam to exam are appropriately accommodated in the cut score process.
Ongoing Test Performance Monitoring
A well-constructed examination must be monitored continuously to ensure it continues to perform as designed. We monitor and analyze initial and ongoing test results to ensure that items are performing correctly and that the tests continue to perform as expected. Our statistical evolution process ensures that tests continue to improve statistically with each edition. Our statistical analysis includes:
P-value (P+) — the proportion of candidates answering an item correctly, used to evaluate item difficulty over time
Point-biserial correlation (PtBis) — a measure of how well each item discriminates between higher- and lower-performing candidates
Reliability coefficients — including Cronbach's alpha, which indicates the overall consistency of the examination
Differential Item Functioning (DIF) — statistical analysis to identify items that may perform differently across candidate subgroups, ensuring fairness across gender, ethnicity, and other demographics
Our statistical monitoring process ensures that each edition of your examination is measurably better than the last — and that any item flagged for underperformance is addressed before it affects candidate outcomes or program defensibility.
Exam Program Audits and Reviews
When you want to ensure your program is performing as it should, preparing for accreditation (NCCA, ANSI, etc), or it is simply time for a comprehensive review, a psychometric audit can identify vulnerabilities before they become problems. We frequently conduct audits of examination programs, thoroughly examining the test development process and psychometrically assessing the results, to determine if the program is fair, valid, and legally defensible.
If we identify areas of concern, we provide clear, prioritized recommendations and work with your team step-by-step to bring the program into alignment with current industry and accreditation standards.
Exam Security Psychometrics
Psychometric analysis of your examination data can help uncover breaches of examination security or just indicate when items have become known to the community at large. We conduct comprehensive psychometric studies specifically related to exam security.
Test Development Services
Our psychometric services integrate seamlessly with our test development process — including job task analysis and exam blueprint development — to support programs at every stage. Visit our test development page to learn more.
LET’S CONNECT