Validation of CARS

A psychological instrument’s validity is the extent to which it accurately measures what it is designed to measure.

For CARS, there are two ways to think about validity. First, there is the question of whether diagnoses obtained through CARS are accurate. When CARS generates a report, it includes the following caveat on the first page: “CARS is a screening device and it does not provide a final diagnostic determination. Further evaluation is necessary to determine whether problems reported are sufficient to qualify as a disorder and to determine the level of problem severity.” This is because, while CARS assesses DSM criteria for psychiatric disorders, it is not necessarily completed by a clinician and thus might not provide as nuanced an assessment of disorder. However, by adapting CARS from content used in the Composite International Diagnostic Interview (CIDI), which has been tested for validity of its questions and diagnoses, we can ensure that the diagnostic criteria and questions CARS uses are evidence-based. In the future, we intend to run studies of CARS comparing its diagnoses to those obtained by a licensed clinician to further assess its validity.

The second way validity applies to CARS is when considering the CARS screener (CARS-SC).  During the development of the CARS screener, we have attempted to create a shorter assessment tool that can still provide valuable information about potential mental health issues. One way to measure this is through sensitivity and specificity. Sensitivity tells you how many people who have a disorder are correctly identified as positive screens. Specificity tells you how many people who do not have a disorder are correctly identified as not having the disorder. If a tool has high sensitivity and low specificity, then it casts a wide net and doesn’t miss anyone with a problem (which can be a very important property for a screener). However, highly sensitive tools, can mistakenly identify many individuals who might not have problems. If a tool has high specificity and low sensitivity, you can be very sure that the individuals it identifies do have problems; however these tools might also miss many individuals with problems. Our goal is to maximize both sensitivity and specificity for the CARS Screener.

The CARS screener is built in such a way that its sensitivity is necessarily near 100% (when compared to the CARS assessment). This is because the questions included in the screener are all necessary for diagnosis. In the future, we intend to run studies of CARS comparing positive screens generated by the screener to diagnoses obtained by the full CARS assessment for each disorder. We do not yet have enough data to make these comparisons. However, because we adapted CARS from the CIDI, we can, in most cases, recreate the CARS screener within the National Comorbidity Survey-Replication data set. This means that, within this general population sample, we can determine how many individuals who screened positive for a disorder according to the question in the CARS screener actually qualified for that disorder according to the CIDI. Below, we present the results of that analysis.