What are Computer Adaptive Tests (CATs)

What is a CAT?

A Computer Adaptive Test (CAT) is a type of assessment in which the questions are specifically tailored to each individual who takes the assessment. To select the most applicable next question, a CAT uses the Item Response Theory to provide questions that will sharpen the estimate of the final score [1]. CATs have a wide variety of uses including but not limited to certifications, college entrance exams, and patient-reported outcomes. See below for a further discussion on how CATs are used in Patient-Reported Outcome (PROs) assessments.

How do CATs work?

A CAT starts with an item bank, which is a collection of questions measuring an assessment. The length of each CAT can vary, however typically four (4) to seven (7) questions are administered for the individual to answer. The first question a CAT chooses to administer assesses an item with a middle-of-the-range function or symptom severity. Once the individual responds to the question, their estimated score is calculated. The CAT algorithm then selects the question it believes will best further refine the individual’s score. Once the individual responds to that question, their score is recalculated. A CAT will continue to provide questions and recalculate the individual’s score until either a specified level of measurement precision is met or a specified number of questions are administered.

How do CATs differ from regular PRO assessments?

Standard PRO assessments use the Classical Test Theory [4] which requires all questions on an assessment to be answered in order to calculate the final score. However, CATs use the Item Response Theory which is based on a probability model that assumes responses on a set of questions related to an unmeasured trait [3]. The Item Response Theory takes what is known about an individual’s responses and patterns of their responses to find the most likely level of trait being measured [1]. 

What are the benefits of using a CAT?

There are a number of benefits when using a CAT to administer PROs. 

• CAT is an efficient way to administer an assessment and due to the limited number of questions asked, it can be completed in a short period of time. This reduces the time and burden placed on an individual which then leads to increased compliance. 
• CAT uses tailored content and only asks the individual to answer relevant questions. Scores from CATs are able to be compared against each other, even if two (2) individuals answer an entirely different set of questions due to the high level of measurement precision gained by using a CAT. 
• CAT is able to cover a wide range of functions or symptoms, making the PROMIS® Physical Function and PROMIS® Upper Extremity great assessments to be administered using a CAT.

PROMIS® Physical Function

PROMIS® (Patient-Reported Outcomes Measurement Information System) Physical Function is an assessment that measures an individual’s physical, mental and social health. PROMIS® Physical Function can be used with the general population and individuals with chronic conditions to help evaluate their standard of living [2]. A standard PROMIS assessment requires the individual to answer all questions in order for a score to be calculated. By utilizing a CAT PROMIS® Physical Function assessment, the individual is asked an average of four (4) questions. This allows the CAT PROMIS® Physical Function to be completed in a short period of time while still providing accurate scores. 

PROMIS® Upper Extremity

The PROMIS® Upper Extremity assessment specifically evaluates activities that require the use of the upper extremities including shoulder, arm and hand [2]. This survey assesses actions such as writing, using buttons and opening containers.

How is a CAT PROMIS® Physical Function and CAT PROMIS® Upper Extremity administered?

A CAT PROMIS assessment relies on self-reported functions rather than actual performance and is intended to be completed by the individual themselves. The PROMIS® Physical Function assessment includes measurements to assess functioning of the upper extremity, lower extremity and central regions, along with activities of daily living (such as running errands). The item bank for the CAT PROMIS® Physical Function contains 165 items. The PROMIS® Upper Extremity assessment utilizes questions from the PROMIS® Physical Function item bank and is administered in the same way the PROMIS Global Health assessment is administered. The item bank for the CAT PROMIS® Upper Extremity contains 46 items.

How is a CAT PROMIS® Physical Function and CAT PROMIS® Upper Extremity scored?

In order to receive a score when taking the CAT PROMIS® Physical Function or CAT PROMIS® Upper Extremity, a minimum of four (4) questions must be administered and answered. Once a CAT determines the standard error is 3.0 or less or administers twelve (12) questions, it will stop the test and produce a T-score. The T-score is a standardized score that is based on the average score of fifty (50) and based on responses to the same questions by the general population. A T-score also has a standard deviation of ten (10) points. The average score for PROMIS® Physical Function and PROMIS® Upper Extremity is fifty (50) for the general population. For both assessments, the higher the T-score the better the outcome and function. As a result, a score of forty-five (45) or above indicates the patient has function within the normal limits and a score of below thirty (30) indicates the patient has severe limited function [5].

Glossary

• CAT: Computer Adaptive Test (CAT) a flexible, computer-driven measure that can use any questions in the item bank. A CAT selects only the questions that sharpen the estimate of the score. CAT length varies, but typically is four (4) to seven (7) questions

• Item: a question or statement and its response choices

• Item Bank: a collection of questions that provide an operational definition of a trait or construct. A good item bank covers all traits being measured, capturing different severity levels along the continuum

• Classical Test Theory: Requires all items to be answered in order to provide a final score

• Item Response Theory: Family of mathematical models that assume that responses on a set of questions are related to an unmeasured trait

References:

[1] Healthmeasures.net. 2020. Glossary. [online] Available at: <https://www.healthmeasures.net/resource-center/measurement-science/glossary> 

[2] Healthmeasures.net. 2020. PROMIS®. [online] Available at: <https://www.healthmeasures.net/explore-measurement-systems/promis>

[3] Healthmeasures.net. 2020. Item Response Theory (IRT). [online] Available at: <https://www.healthmeasures.net/resource-center/measurement-science/item-response-theory>

[4] Classical Test Theory. Available Here