Attitudes about use of preoperative risk assessment tools: a survey of surgeons and surgical residents in an academic health system

Abstract

Background

Formal surgical risk assessment tools have been developed to predict risk of adverse postoperative patient outcomes. Such tools accurately predict common postoperative complications, inform patients and providers of likely perioperative outcomes, guide decision making, and improve patient care. However, these are underutilized. We studied the attitudes towards and techniques of how surgeons preoperatively assess risk.

Methods

Surgeons at a large academic tertiary referral hospital and affiliate community hospitals were emailed a 16-question survey via REDCap (Research Electronic Data Capture) between 8/2019-6/2020. Reminder emails were sent once weekly for three weeks. All completed surveys by surgical residents and attendings were included; incomplete surveys were excluded. Surveys were analyzed using descriptive statistics (frequency distributions and percentages for categorical variables, means, and standard deviations for continuous variables), and Fisher’s exact test and unpaired t-tests comparing responses by surgical attendings vs. residents.

Results

A total of 108 surgical faculty, 95 surgical residents, and 58 affiliate surgeons were emailed the survey. Overall response rates were 50.0% for faculty surgeons, 47.4% for residents, and 36.2% for affiliate surgeons. Only 20.8% of surgeons used risk calculators most or all of the time. Attending surgeons were more likely to use prior experience and current literature while residents used risk calculators more frequently. Risk assessment tools were more likely to be used when predicting major complications and death in older patients with significant risk factors. Greatest barriers for use of risk assessment tools included time, inaccessibility, and trust in accuracy.

Conclusions

A small percentage of surgeons use surgical risk calculators as part of their routine practice. Time, inaccessibility, and trust in accuracy were the most significant barriers to use.

Background

Accurate preoperative assessment of a patient’s surgical risk for postoperative complications is a challenging but important task for surgeons. Although surgical risk assessment tools such as the American College of Surgeons’ Surgical Risk Calculator, [1] the Veterans Affairs Surgical Quality Improvement Program (VASQIP) Patient Risk Calculator, and the Surgical Risk Preoperative Assessment System (SURPAS) [2] accurately predict risk of adverse postoperative patient outcomes, these tools have not been widely adopted by surgeons to routinely assess preoperative surgical risk [3, 4]. Surgeons are accurate at predicting preoperative risk in low-risk patients, but they are inaccurate when predicting risk of adverse events in high-risk patients. [2, 5] This is problematic as high-risk patients are the most likely to benefit from pre-operative counseling, informed consent discussions, and potential risk mitigation efforts.

Implementation of formal preoperative risk assessment may optimize understanding of risk of adverse postoperative outcomes by both surgical teams and patients [3, 4]. Patients desire to understand their risks, and report increased shared decision making with use of such tools [6,7,8,9]. The use of personalized risk assessment increases the patient’s understanding of the proposed operation and improves the communication of informed consent and individualized patient risks [10], which improves patient anxiety, comfort about the operation, and satisfaction. Surgeon use of such tools has been shown to lead to more accurate judgments of operative risk [11, 12]. These tools have allowed the operative decision-making process to become more objective in evaluating for potential postoperative complications [13]. Therefore, increased understanding of surgeons’ attitudes about use of risk assessment tools should be useful for further development of these tools.

The purpose of this study was to collect information on surgeons’ attitudes toward surgical risk tools, including how often they are used compared to other methods of risk assessment, how risk is communicated to patients, what factors increase the likelihood of using surgical risk tools, what are barriers to use of surgical risk tools, and which features enhance usefulness of surgical risk tools. It is also important to determine if level of training (e.g., resident vs. attending surgeon) impacts attitudes about surgical risk assessment. We hypothesized that: (1) a small percentage of surgeons routinely use preoperative risk assessment tools; (2) that attending surgeons are more likely to rely on prior experience to guide risk assessment, and residents are more likely to use surgical risk tools; and (3) that time is a significant barrier to use.

Methods

This research was conducted after review and approval from the Colorado Multiple Institutional Review Board (COMIRB#: 19-1068).

This study was a prospective survey developed by the research team under the guidance of a qualitative specialist with extensive experience in survey design (co-author ALK). The team designed a 16-question multiple choice survey to elicit opinions of and attitudes towards use of formalized risk assessment tools (Fig. S1). We used a series of multiple-choice questions that assessed the respondent’s frequency of risk discussions with patients, methods used, time spent, the sources of the risk information, useful features of a risk assessment tool, and barriers to use. The survey was designed such that respondents could choose to remain anonymous.

The survey was administered to surgical faculty and residents in the Department of Surgery at the University of Colorado Hospital (UCH), the Rocky Mountain Regional Veterans Administration Medical Center, Children’s Hospital of Colorado, Denver Health Hospital, and to affiliate appointments of the Department of Surgery in the years 2019-2020. We included surgical faculty and residents within the surgical specialties of general, trauma/ acute care, surgical critical care, surgical oncology, vascular, cardiac, thoracic, plastic reconstructive, colorectal, transplant, endocrine and urology. Affiliates included academic-affiliate hospitals within the UCHealth system. Survey response was incentivized through two $50 gift card raffles, one for surgical faculty and one for surgical residents. Reminder emails were sent once weekly for three weeks after the initial survey was disseminated. Incomplete survey responses were excluded. Surveys were disseminated and responses collected through REDCap (Research Electronic Data Capture, Vanderbilt University, Nashville, TN), a software tool for designing and collecting data for a research database. In addition to the 16 survey questions, we also collected the following data about the surgeon responders: age, gender, years of practice, specialty, education, hospital type, and hospital bed size.

Descriptive statistics, including frequency distributions and percentages for categorical variables, means, and standard deviations for continuous variables, were used to summarize the survey results. Group comparisons were performed using Fisher’s exact test for categorical variables, and unpaired t-test or Wilcoxon rank sum test for continuous variables. Results were considered statistically significant if the p-value of the test was ≤0.05. All statistical analyses were performed using SAS version 9.4 [SAS Inc, Cary, NC].

Results

A total of 261 surveys were emailed (108 to faculty surgeons, 95 to residents, and 58 to affiliated surgeons). One hundred twenty (46.0%) were completed (54 faculty surgeons, 45 residents, 21 affiliate surgeons). Response rates were 50.0% for faculty surgeons, 47.4% for residents, and 36.2% for affiliate surgeons. Two additional surveys were received but excluded due to incomplete data or the surgeon no longer being in clinical practice.

Table 1 presents the demographic information of survey participants. Approximately 40% of residents and affiliated surgeons were female, but only 26% of faculty surgeons. Average age and median years of practice of faculty and affiliate surgeons were similar (49.6 and 10 years vs. 46.0 and 12 years, respectively). The 120 surgeons were from general surgery (70.0%), urology (7.5%), vascular (6.7%), cardiac (3.3%), and other specialties (12.5%); and from an academic hospital (UCH) (78.3%), urban community hospital (15.8%), or rural hospital and other settings (5.8%).

Table 1 Survey participants’ and hospital characteristics

Table 2 presents the 120 respondents’ practices for risk discussion with their patients. The respondents were to record an answer for all categories, and the categories were not mutually exclusive. For comparisons, faculty and affiliate surgeons (i.e., non-trainees) were combined and compared to residents. Among all 120 surgeons, 45.0% spent 5-9 minutes discussing risk, and 36.6% took more than 10 minutes; 72.5% used prior experience (all percentages are most or all of the time, unless otherwise specified) and 60.0% used current literature as their sources of risk estimates, while only 20.8% used online risk calculators as a source of risk estimates. The primary methods (i.e., methods used most or all of the time) that the surgeons used to communicate risk to patients included face to face (90.0%) and pre-anesthesia clinic (51.6%). Phone calls (4.1%), residents (19.2%), advanced practice providers, (4.2%), handouts/pamphlets (19.2%), and classes/videos (4.1%) were used much less frequently. Only 20.8% of the surgeons communicated risk back to the referring provider most or all of the time. Residents spent less time discussing risk with patients (20.0% vs. 46.7% >10 minutes, p = 0.004), used risk calculators more often (62.2% vs. 49.4% sometimes to always, p = 0.002), and used prior experience (62.2% vs. 78.6%, p > 0.0001) and current literature (44.4% vs. 69.3%, p = 0.004) less often than non-trainees. Residents also mainly used face to face to communicate risk to patients and used residents more often (46.6% vs. 2.7%, p < 0.0001) and pre-anesthesia clinic (42.2% vs. 57.3%, p = 0.02) and handouts/pamphlets (4.4% vs. 28.0%, p < 0.0001) less often compared to non-trainees. Residents communicated risk less often to referring providers (6.6% vs. 29.4%, p < 0.001).

Table 2 Duration of operative risk discussion, source of risk estimates, methods of communicating risk to patients, and communicating risk to referring provider

Table 3 presents factors related to when surgeons might be more likely to use risk assessment tools. Among all 120 surgeons, 90.0% were more likely to use risk assessment tools when patients had significant risk factors, 76.7% when patients were >65 years old, 70.0% to dissuade patients/families from surgery, 69.2% when patients asked about risks, and 49.2% when patients needed emergency operations. There were no statistically significant differences between faculty and affiliate surgeons and residents on these attitudes.

Table 3 Factors related to likelihood of using risk assessment tool

Table 4 presents respondents’ attitudes towards barriers in using risk calculators and the utility of certain features of risk calculators. Among all respondents, 65.9% thought time was a moderate or significant barrier, 61.7% not being integrated into the electronic health record (EHR) or inaccessibility during patient visit, 59.2% trust in accuracy, 46.7% inability of patient to understand, and 32.5% patient language barrier. Compared to faculty and affiliate surgeons, residents thought that inaccessibility of the tool during the patient visit and native language of patient were more moderate or significant barriers. Among all 120 surgeons, 93.3% thought prediction of major complications and mortality made risk calculators very or extremely useful, 80.9% automatic integration of risk factors from the EHR, 80.0% prediction of post-surgery infection, 77.5% automatic recording of results into the EHR, 75.0% prediction of a good surgical outcome, and 48.3% prediction of minor complications. There were no statistically significant differences in attitudes toward features that make risk calculators useful between the faculty and residents.

Table 4 Potential barriers to use of risk assessment tool and features that make risk assessment tool useful

We also examined surgeon attitudes toward risk discussions with patients and use of risk assessment tools between the 21 affiliate and the 54 faculty surgeons. The attitudes were similar for all questions, with the exceptions of use of residents to communicate risk and whether they were more likely to use risk assessment tools for emergent operations. Faculty used residents more to communicate risk (70% for faculty vs. 14% for affiliate surgeons rarely to always, p < 0.0001) and they were more likely to use risk assessment tools for emergent operations (63% vs. 38%, p = 0.05). We also examined surgeon characteristics (gender, age, years of practice, education, type and bed size of facility practicing in) related to whether they used risk assessment tools never/rarely (n = 55), sometimes (n = 40), or most of the time/always (n = 25) (data not presented). We found no consistent trends between any of the surgeon characteristics and their use of risk assessment tools.

Discussion

This study found that a small percentage of surgeons (20.8%) used on-line risk calculators most or all of the time; that attendings were more likely to use prior experience and current literature and residents more likely to use risk calculators as a source of risk estimates; and that time was the number one barrier to use of risk calculators. The surgeons were also more likely to use risk calculators when patients were older, had significant risk factors, to dissuade patients and families about surgery when risk was high, or when patients asked about risk. Risk calculators were thought to be more useful for prediction of major complications, mortality, and post-surgical infection; when they were built into the EHR, when they automatically recorded results into the EHR; and when they predicted a good surgical outcome; they were less useful for minor complications. Top barriers for use of risk calculators, in addition to time, were thought to be accessibility and not being in the EHR, trust, inability of patients to understand, and language barriers.

It is not surprising that residents would rely more on risk calculators and less on their prior experience in estimating preoperative risk. The average age of the residents was 30.7 years, so they might be more in tune with computer technology, and they had more limited prior clinical experience. The fact that the surgeons were more interested in risk prediction tools for older patients with more risk factors and to dissuade patients with very high risk agrees with our previous research which showed that surgeons thought a risk calculator was more useful for high-risk patients and less useful for low-risk patients [2, 4].

Meguid, et al, [14,15,16,17,18] have developed a surgical risk calculator that has some of the features thought to make a risk calculator useful and to avoid some of the perceived barriers. Called the Surgical Risk Preoperative Assessment System (SURPAS), the risk calculator covers 9 surgical specialties (general, gynecology, neurosurgery, orthopedics, otolaryngology, plastics, thoracic, urology, and vascular), only requires the input of 8 preoperative variables (4 related to the operation—CPT-specific event rate, work RVU, inpatient/outpatient, and specialty; and 4 related to the patient—age, ASA class, functional health status, and emergency), and calculates a patient’s risk for 12 postoperative adverse events (30-day mortality, overall morbidity, unplanned readmission, discharge not to home, and the 8 specific complications of infection, cardiac, pulmonary, renal, UTI, VTE, bleeding, and stroke). Thus, the calculator is easy to use, taking only a few minutes to input data (time barrier); covers the major postoperative complications; is built into the EHR at our local health system and can be integrated into the EHR at other institutions; and writes a note into the patient’s EHR chart. Further, multiple validation studies have been done using internal [17,18,19,20,21] and external [22,23,24,25] validation methods, to address the issue of “trust” in the accuracy of the models. Several pilot studies have also been completed recently showing that after SURPAS patients understand their risks for surgery very well [2] and have improved satisfaction, comfort, and anxiety about their operations [10]. If surgeons at other institutions wish to use SURPAS, they can either use an on-line version at https://surpass.agilemd.com, or if they want SURPAS to be built into their own local EHR, they can access the SURPAS statistical models at https://github.com/katiecolborn/SURPAS.

Other studies have shown that risk assessment tools are underutilized by trainees, favoring traditional models of communication due to lack of an electronic health record and clinical workflow interruption [26]. This remains true for attending surgeons as well. A patient’s native language may also pose a barrier to using risk calculator tools in hospitals where interpreter services are not readily available. A formal handout given at the end of the visit, complete with visuals of risk, may help reinforce the preoperative risk discussion. When surgeons forego using a surgical risk calculator, they misestimate a patient’s risk of postoperative adverse events regardless of level of training [11].

In order to successfully implement surgical risk tools, especially in a high-volume clinic setting, accessing and using these tools must be quick, seamless, and accurate. Both ACS-NSQIP and VASQIP require manual entry of over 20 pre-operative variables, including laboratory values that may not be readily available for all pre-operative patients. These factors might preclude widespread use in a large volume pre-operative clinic setting where surgeons may only have 15 minutes allotted for the entire patient encounter.

This study had two major limitations: (1) Respondents were from just one health system (UCHealth). However, the health system did have different types of hospitals, including a large university hospital, several community hospitals, and a few rural hospitals; and (2) The overall response rate was only 46.0%. However, this was somewhat better than the typical response rates of 20-30% observed in other physician surveys [27].

Conclusions

Only twenty percent of the surveyed surgeons used a risk calculator most or all of the time. Attending surgeons were more likely to use prior experience and current literature as sources for risk information, while residents were more likely to use risk calculators. Time, accessibility, and trust in accuracy were cited as the most important barriers to use of a risk calculator.