Patient-Reported Outcome Survey Response Rates – What You Need To Know

Patient-reported outcome surveys are used for a multitude of reasons from internal research and quality improvement to programs such as Joint Commission Certification and CMS’s 2023 IPPS ruling – both of which require a high PRO survey capture rate. At CODE we take great pride in the high survey capture rates we consistently deliver to our clients. We find that while there are many variables involved in patient engagement and survey participation, there are four keystones to creating a successful PRO program:

• Consistency
• Methodology
• Maintenance
• Patient Populations

 When comparing PRO vendors, it is important to understand these four elements and how they contribute to the vendor’s ability to successfully meet your capture rate expectations and the sustainability of your program overall.

 Consistency

The true test of a PRO vendor’s collection is the ability to produce consistent capture rates across a multitude of patient populations, surgeons and practices. When trying to understand how the vendor will perform for your organization, low levels of variability within populations and providers over time is a key indicator of the vendor’s performance. As is always true in data, it does not matter if a vendor has a strong capture rate for one or two of their clients – it is their median performance that matters! At CODE, we run a box plot that helps us visualize the capture rate by client (each hospital/practice that we collect for). Our account managers are responsible for ensuring that each site is performing as close to the CODE median as possible. When there is a deviation from our registry norm, we assess the account, diagnose the issue, suggest a treatment plan, and continue to monitor to confirm the plan is working.  When interviewing potential PRO vendors, make sure you ask for their median capture rate and that they clarify how many different organizations, surgeons and patient populations that includes. Doing so will help provide you with an accurate depiction of their capture rate variability.

Knee Replacement Capture Rate

Methodology

Capture rate methodology is more complex than one might think. Differences in methodology are prevalent across the industry, so it is important to understand a PRO vendor’s methodology and to make sure they are able to meet your program’s needs based on that methodology. Let’s use a practical example to show how different methodologies can make a big impact on how capture rate is measured, and what that could mean for your program.

 To set the scene, let’s say a hospital is participating in a certification program that requires 60% of all their knee replacement patients to complete a patient-reported outcome survey 3 months after their surgery. In this case, the hospital has decided to use a PRO vendor to help them with collection and is evaluating 2 different vendors – both who market an 80% response rate. For easy math, we will say there were 100 patients who had knee replacement surgery. Take a look at how the different methodologies each company uses makes a drastic difference in the hospital meeting their capture rate goals:

Company A methodology: The equation Company A’s methodology uses the number of patients who completed a survey as the numerator and the total number of patients receiving surgery as the denominator.

Company A determines that in order for a patient to be eligible to receive a post-op survey, they must complete a pre-op survey. Of the original 100 eligible knee replacement patients, 80 of them complete a pre-op assessment. Those 80 patients are then sent a 3-month post-op survey, of which 64 of them complete the survey. According to the company’s capture rate methodology, both the pre-op and 3-month post-op survey capture rate are 80%.

Okay, but did they meet the required capture rate? Yes, but just barely. The certification program’s metric is 60% of all total knee patients who complete a 3-month post-op PRO, not just those who complete a pre-op PRO. When you apply that methodology, the 3-month capture rate is 64% – not 80%.

 Company B methodology: Company B’s methodology uses the number of patients who completed a survey as their numerator and number of patients active in the PRO application as their denominator. An “active patient” is defined as someone who successfully downloads and logs into the PRO app after being emailed the instructions. The percentage of patients who are active varies, anywhere from 10-90%.

 Company B determines that a pre-op survey is not required to be eligible to receive a post-op survey. Of the original 100 knee replacement patients, 80% of active patients completed a 3-month survey. According to their capture rate methodology, they have an 80% capture rate!

Okay, but did they meet the required capture rate? No. Only 30 of the 100 patients became active in the application. Breaking it down, if 80% of the 30 active patients completed a 3-month assessment, that is only 24 completed assessments. This means that only 24/100 patients receiving surgery completed a 3-month assessment, making their 3-month capture rate 24% – not 80%.

 As you can see, both companies can deliver on the 80% capture rate they market – but how they calculate capture rate makes a huge difference in the total number of surveys captured. Of these two methodologies, neither of them is “right” or “wrong”, they are just different.

 While we are looking at different methodologies, it is important to also look at how your organization internally defines capture rate. This can be more nuanced than you think! For example, requiring a pre-op assessment be completed in order for a patient to qualify for a post-op assessment creates a complete data set that facilitates the study of functional improvement – or how a patient’s outcomes are impacted over time. However, if the comparison of pre-op to post-op PRO scores is not critical for your program’s needs, it might be best to collect post-op data regardless of pre-op participation. There are certainly pros and cons to both ways (Pun intended). What is important here is fully understanding the methodology – it could come back to haunt you later.

Registry Maintenance

There is a healthy amount of day-to-day maintenance required to obtain a high capture rate. Understanding upfront the tasks that need to be performed as part of this maintenance, and who will be responsible for working those tasks, is critical. Are FTE resources required on your end to perform this maintenance? Or does the vendor handle it as part of their service? Here are a handful of examples of registry maintenance needed for obtaining a high capture rate:

Patient Contact Information– the contact information at the time of enrollment is not always accurate or oftentimes it is updated after the patient is scheduled for surgery. Questions to ask your vendor regarding patient contact information include: What happens when an email bounces? Similarly, what happens when a phone number that previously worked for an SMS message, doesn’t go through? This is a very common situation that happens with post op data collection, especially with email addresses.

Change in patient status– it is not unheard of for a patient to have a change in their health status that makes them not eligible for participation, such as an accident that resulted in an amputation, or death. In cases such as these, you want to be extra sensitive to not reach out to these patients (or their family) to complete a survey. It is important to ask your vendor in cases like these who updates the PRO system and how can they be sure a patient is not contacted incorrectly?

Patient Education– we know educating patients on the process and importance of PROs is directly correlated with response rate. What isn’t talked about as much is the work necessary for patient education materials. Things to keep in mind when talking to a vendor is finding out who develops, monitors, and tracks the effectiveness of patient education?

Survey Fatigue– patients are being surveyed for each healthcare service they receive, and the demand for PROs is only going up. Survey fatigue is a response rate killer, and needs to be assessed and treated on an ongoing basis. We at CODE take survey fatigue seriously and through our years of experience have found some of the best ways to help combat it. We recommend asking your vendor about survey fatigue. How it is handled, who is monitoring for signs and symptoms of survey fatigue, and what recommendations they have to avoid it as much as possible.

Procedure Reschedules- sometimes patients are unsure of what day works best for their surgery. In those cases, schedulers may hold multiple spots for the patient until they decide. In scenarios such as this, it is important to understand how and when a vendor contacts a patient. If the patient is tentatively placed on the schedule for 3 different days, how does the vendor know to only reach out to the patient for 1 of those surgeries?

Patient Questions- as a patient is completing their assessment they may think of a question and hit reply to the email. As a practice, you want to know when your patients are asking questions and help them get a response. When talking with a vendor, make sure you understand what happens to patient replies when they are sent in response to a survey. Is someone monitoring and/or responding to them, or are they left in the abyss?

Is registry maintenance fun work? No. But it is necessary to keep that capture rate up, especially the long term post op PRO data. The most important thing to understand is who is responsible for each part of registry maintenance and how that will impact your PRO collection. If you are required to maintain your registry it is vital to factor in the FTE costs and workload levels. What happens if that FTE moves to a different role? Who will take over the maintenance and who will train them on that?

Patient Populations

As clinicians we understand that patient compliance to medical advice varies by individual patient and by patient population – our CODE registry shows this is very true for PROs as well. Each population is unique with its own set of patients, recovery expectations and yes, even different expected median capture rates! For example, patient age has a bell-shaped curve effect on capture rate. Younger patients are busy and not as likely to respond to their surgeon’s request and older patients are less likely to have access to technology (although this is slowly changing). Tucked in the sweet spot between these age groups are retirement aged patients who have time on their hands and are happy to fill out a PRO assessment for their surgeon! 

Capture Rate Percentage

At the end of the day, it is important to understand your organization’s internal capture rate methodology and associated goals, and then select a vendor that is appropriately matched to your needs. When evaluating or working with a vendor, ensure you ask questions to fully understand what their capture rate means and how that may apply to your program. We recommend evaluating how consistent their capture rate is across populations and organizations – an 80% capture rate at one facility may be an outlier, not their average. Beyond that, understand their capture rate methodology – how do they get to that capture rate and with the way they calculate it will your organization meet your registry goals? Additionally, it is important to clarify who owns what piece of registry maintenance. We know it may not be the most glamorous part of collecting PROs but it is one of the most important pieces and it’s vital to your program.