A Case Study of the BAUS Audit on Prostatectomy
Recently, there has been ongoing debate around the impact of case volumes on surgical outcomes. A previous blog post (The Cost Debate in Robotic Surgery and the Impact of Skills) discussed a 2013 study published in the New England Journal of Medicine by Dr. John Birkmeyer, et al, which looked at skill levels between surgeons and identified that surgeons in the lower quartile completed three times fewer operations compared to surgeons in the top quartile.
In December 2012, the UK Government outlined plans to publish surgeon-level outcomes data, taken from national clinical audits, in ten specialty areas, which included Urology. This is known as the Consultant Outcomes Publication (COP) programme.
The British Association of Urological Surgeons (BAUS) has since published a number of audits on surgical outcomes in areas such as Prostatectomy, Nephrectomy, Stress Urinary Incontinence, and Urethroplasty. These audits are available for the general public to review the volumes and outcomes of a wide variety of surgeons.
The 2015 Prostatectomy audit, which looked at 2014 cases was published in September 2015 and the results are summarized below (as published on the BAUS website).
- The data collection period was from January 1, 2014 to December 31, 2014
- 6,161 cases were submitted in total, of which 5,814 were from England; these 5,814 cases came from 147 consultants at 62 sites, and include 230 private patients from 37 consultants
- Hospital Episode Statistics (HES) for 2014 indicate that there were 6,651 radical prostatectomies undertaken in England, so data was collected from 87% of the radical prostatectomies undertaken in England in 2014
- 5% were robotic assisted, 26.7% laparoscopic, 13.4% open and in 1.4% of cases the technique was not recorded
- Median number of cases per consultant: 32 (range 1 – 157)
- Median number of cases per center: 85 (range 1 – 250)
- The overall transfusion rate was 7% – for England only, 2.6%. In England only, the transfusion rates by technique were: open 5.4%, laparoscopic 0.8% and robotic 2.9%.
- 5,174 of the entries recorded whether there had been adverse events. The total post-operative complication rate was 5% (491 / 5174). Of these 491 cases, 364 recorded the Clavien Dindo grade (i.e. 127 or 26% did not). Complications classified as Clavien Dindo Grade III or above were seen in 1.6% of cases.
“Another interesting point to note was that there were differences in surgical positive margin rate between the three approaches,“ says Mr. Ben Challacombe MS FRCS (Urol) Consultant Urological Surgeon & Honorary Senior Lecturer, Guy’s Hospital & King’s College London. “Robotic surgery had an average of 13% while both the open and Laparoscopic approaches were at 19%. The length of stay was also seen to be lower for the robotic approach at a median of one day post operative”
We decided to go into each of the individual surgeons recorded on the web site and try and see if we could give any further insight into volumes carried out by the differing surgeons. Given that 12% of the cases do not have complications rate reported I did not try and see of there was a linkage between volume and complications due to the incompleteness of the data set available.
There were a number of interesting patterns.
Volume by procedural type:
As the table shows, more surgeons did Robotic cases and on average did more cases per year than the other approaches. We therefore decided to an analysis of the difference in procedures between the top and bottom quartile. A quartile was based on the number of surgeons doing the procedures so for example in the robotic cohort we compared the volume of the top 21 surgeons against the bottom 21 Surgeons.
The table below highlights the differences:
25% of the surgeons (37) with the highest volumes carried out over 50% of the cases. Interestingly the concentration was greatest in Open surgery where they carried out 69% of the cases. At the other end of the spectrum 37 surgeons with the lowest volumes only did just below 9 cases each or 5% of the total volume.
This picture is made slightly more complicated as clearly some surgeons will do more than one technique. In this sample 106 of the surgeons (73%) used only on technique while 36 used two techniques and three surgeons used all three techniques.
The overall surgical volumes increased as the number of techniques used increased. Those using one technique averaged 39 cases in the time period, those using 2 averaged 45 and those using 3 averaged 49 procedures. It is only natural that this occurs as surgeons move from one technique to another or believe that different patients are better suited to different techniques.
One of the big advantages of the 21st century is that data from surgical performance is becoming more transparent. This transparency is going to allow medical professionals to have visibility on a number of factors that affect patient outcomes, which will allow them to put in the correct protocols to ensure that the highest quality of care is always delivered. We believe that the increasing amount of data is showing that the use of validated simulation protocols and curriculum can ensure best results for patients.
On November 13, 2015, a panel of experts in robotic surgery met at the Académie Nationale de Chirurgie (ANC) in Paris to discuss training in robotic surgery. The meeting was moderated by Professor Michel Huguier and the speakers included:
- Professor Rolland Parc, Conseil de l’Ordre des Médecins
- Professor Jacques Marescaux of IRCAD (the institute for research into cancer of the digestive system), Strasbourg
- Professors Jacques Hubert and Laurent Bresler of the School of Surgery at Nancy
- Professors Xavier Cathelineau and Guy Vallancien of the École Européenne de Chirurgie (European surgical training centre), Paris
- Professor Jacques Belghiti from the HAS (the French national health authority)
- Dr Denis de Valmont from the insurance company SHAM
- Dr Yves Allioux of the Caisse Nationale d’Assurance Maladie (CNAM)
The full day session included discussions and presentations on the current status of robotic training in France as well as an overview of the adoption and current state of robotic surgery in a number of key specialties from Urology to Thoracic.
Their group identified some fundamental needs:
- Training requirements should be based on the established protocols for training in surgical robotics (drawn up by the teams from Nancy, France)
- It is essential to anticipate the arrival of new robotic platform
- It would not be helpful to increase the number of training centers. What is required would be several centers of excellence who are well equipped in platforms and personnel, with good reporting systems or registers.
The guiding principles of modern computer-assisted surgery, and thus of robotic surgery, should be the following:
- It is assumed that the clinicians should know how to operate and be competent in their surgical specialty
- Surgeons need to become familiar with the all aspects of the computer-assisted system
- Success will only be achieved through partnership with the manufacturers
- However, maintaining professional ethics and independence and avoiding all conflicts of interest is essential
- Being able to justify scientifically the evolution of treatment approaches thus being able to satisfy financial policymakers, and to defend surgeons against whom the HAS starts disciplinary proceedings.
Conclusions from the discussions:
The training in robotic surgery currently provided by the manufacturers is not a legally binding qualification. Their only obligation, as with any equipment manufacturer, is to explain to the purchaser how their product works. This training, according to published literature, is generally too short, and does not include any assessment of surgeons’ ability to use these robot systems. The responsibility of monitoring this training should, therefore, fall to the scientific societies and the universities in partnership with the manufacturers, and should include the evaluation of teams who will be tasked with these, using these new technologies. Training in robotic surgery can be provided by both public or private institutions, bearing in mind that it requires a substantial investment in equipment. It appears that university budgets alone will not be enough to meet this investment, and that public institutions could enter into partnerships with the private sector to meet the demand.
Robotic surgery is put into practice by surgeons and their teams, and their training should cover 5 areas:
1 – Surgical training is the remit of the existing schools of surgery
2 – Basic training in the use of a “robot” is common to all specialties that plan to use the system. It should be validated by a document certifying that the surgeon attended a course of basic training involving learning about the machine and the relevant techniques, with time on a simulator and on the robot in “dry lab” and “wet lab”. This stage of training should finish with an assessment
3 – In robotic surgery the surgeon is removed from the operative field, and there is, therefore, a loss of visual communication with the rest of the team. This makes training of the other members of the surgical team (team training) indispensable
4 – The clinical training specific to each specialty and procedure will be carried out in centers that have robots and having “proctors” (“Advanced Courses”)
5 – Surgical practice involves lifelong learning, which requires that the surgeon maintain his skills throughout his or her career. The question of re-certification, like that imposed on aircraft pilots following a period of inactivity or when they don’t practice their skills on a regular basis, does not currently exist in medicine. It is likely that in future the development of simulators will enable surgeons in these situations to refresh or maintain their technical skills.
Growing in popularity, robotic surgery is still not without challenges. Before the benefits of robotic surgery can be fully realized, the highest level of patient safety must be ensured, while remaining cost effective and at the same time allowing new surgeons the ability to be trained and access the technology without impacting safety and cost-effectiveness.
The Halstedian Method of “see one, do one, teach one” is clearly no longer sufficient for surgical training. Many comparisons between the training of pilots and the training of surgeons have been made over the years. In 2013, the FAA updated their rules to state that to be qualified as a First Officer, a pilot needed 1,500 hours total time. This includes both real and approved simulation time. Looking at a typical Resident training program, the calculations for a general residency that will last 4 years is approximately 16,600 hours. If within this a surgical trainee chose to focus on Gynecology for 20 months they would receive 6,400 specialty hours. If a surgeon focused on minimally invasive surgery, such as robotics, the Accreditation Council for Graduate Medical Education (ACGME) guidelines recommend 105 hours exposure to a variety of cases. Even tripling this minimum, a surgeon would be only at 300 hours of surgery, which is only a fraction of the 1,500 hours the FAA recommended training time for pilots.
Just as in aviation, simulation has been seen to be a solution allowing surgeons to develop their skills without impacting patient safety. Mimic’s MSim software, found on both the dV-Trainer and the da Vinci Skills Simulator, has been one of the most researched and validated simulation software in the surgical field. Table 1 below shows the range of validation studies that have been carried out on either platform as well as other simulators.
The studies, and in a particular the predictive validity study looking at simulation and operative outcomes, carried out by Dr. Culligan, have helped shape recommendations for surgical training being developed by medical societies such as those developed by the American Association of Gynecologic Laparoscopists (AAGL) in 2014.
All of these studies and simulation programs focused on the psychomotor component of learning how to “drive the robot” and not necessarily the cognitive training requirements that would help train the next generation of surgeons. Augmented reality was developed under the Maestro AR name to help solve this issue. It is best to think of Maestro AR as a curriculum incorporating both psychomotor tasks and cognitive questions supported by a moderated guide on a procedural approach and technique.
The basic premise is that a student will learn more if the psychomotor skills that they require are placed within their procedural context as opposed to in a vacuum. As students are learning how to use the robotic device they are also being tested on tissue recognition, procedural choreography, as well as learning from the narration about the decision making process behind this specific approach.
The Benign Hysterectomy Maestro AR module, for example, is divided into 9 modules starting with a Pelvic Anatomy survey and working through clear steps on how to deal with the ligaments and uterine vasculature before finishing with the Colpotomy and the Vaginal Cuff Closure.
“Maestro AR addresses the next frontier of training by developing a pathway that incorporates Didactics with Augmented Reality through virtual reality simulation,” says Mireille Truong, Virginia Commonwealth University Medical Center, “I am confident that research will show that adding didactic elements to simulation training will continue to improve surgeon performance when they enter the OR.”
As with the airline industry, simulation is becoming a vital part of the armarmentarium required for surgeons to ensure that through all stages of their career, they have the correct level of skills for the task ahead of them. Just as pilot is able to land a plane in virtually any airport around the world on their simulator, it is hoped that surgeons will also be able to develop their skills within a procedural scenario in an augmented reality environment.
As robotic surgery has increased in popularity and adoption, so has the recognition that training for robotic surgeons is critical. Also critical is ensuring that when these surgeons perform their first case on live patients they will have been exposed to and trained for all aspects of the system.
In Europe, ERUS (the European Robotic Urology Section of the EAU) has been designing and developing a structured training program and curriculum to help surgeons who wish to engage in performing a robot-assisted radical prostatectomy (RARP).
The program is divided into six clear sections. Starting and finishing with evaluation protocols, participants go through online modules and observations, simulation training aligned with wet and dry labs, as well as modular console training and training on the full procedural steps.
When implemented, the program took approximately 12 weeks for participants to complete, including one week dedicated to simulation training and dry and wet lab tasks. After the intensive week of hands-on training was completed, the fellows returned to their institutions carrying out specific parts of the procedure and mastering them before moving onto other steps and finally a complete procedure. Their final procedure was assessed by their mentors as well video recorded for review by independent assessors.
In total, of the ten students who took part in the study 3 were residents, 5 were fellows and 2 were staff members. Based on the final assessment, two of the residents were not deemed sufficiently proficient to carry out a RARP independently but the remaining eight (80%) of the participants were deemed proficient including three (30%) who achieved being considered able to complete complex cases.
Throughout the 12-week program, the participant’s skills were measured on a regular basis and over time showed significant improvement. The graphs below illustrate how the skills improved over the duration of the fellowship in two of the more complex exercises.
Given the importance of simulation to this curriculum, Mimic has been pleased to work with the EAU and ERUS to offer exposure to a short simulation curriculum during the a number of EAU congress in 2014 and 2015. At seven various educational events between September 2014 and December 2015 Mimic was able to offer 11 days of robotic simulation initiation. 107 participants were able to sit down at the simulator and complete and average of 8 exercises each. The initial courses focused only on console skills, with later courses including laparoscopic skills for bed side assistants.
All exercises were completed and the average pass rate was 28%, though it ranged from 60% to 0% depending on the student.
Robotic-assisted surgery has grown over the past few years with over 600,000 procedures carried out in 2015. Increasingly, the debate over the clinical value of robotic-assisted surgery is being replaced by a focus on the cost of robotic surgery over other types of intervention. In an infographic published in 2015, the ECRI institute estimated that the average robotic-assisted case incurred an additional $3,000 to $6,000 per procedure.
This article is not meant to debate cost accounting and whether or not this number is correct, but rather to look at what some of the factors are that could be driving these costs and what can be done about it.
CAVA robotics is an organization that a focuses on delivering proprietary, unprecedented, diagnostic and prescriptive robotic change management programs for hospital administration, their surgeons and OR crew. CAVA’s approach is data-driven and allows hospitals to analyze what is truly driving their costs.
“Generally speaking there are two primary factors that drive costs” says Dr. Rick Low, CAVA Medical Director. “On the one hand, you have costs that are driven by the operational inefficiencies of bringing a large system into the OR. These are driven by consumables, OR times, setup times, and the lack of standardization of procedures around the implementation of the system. This accounts for 50% of the cost driver to the hospital. The other 50% of cost is tied to the technical skills of the surgeons, the first assistants and the OR teams, in general,” concludes Dr. Low. “This excludes variables like complications, readmissions, and medical malpractice which most hospitals do not factor into the bottom line analysis of the surgical options and may actually be in some cases much more costly.”
Recent data analysis has shown that there is a relationship between surgical skills and patient outcomes. In a study published by NJM in 2013, Dr. Birkmayer asked 20 surgeons from the state of Michigan to provide a video of their bariatric surgical cases. These cases were then evaluated by a group of independent surgeons and grouped by skill level into quartiles. This stratification was then used to study approximately 10,300 cases that the group had carried out over a 7 year period. Those in the bottom quartile had 2.5 times more readmissions (6.7% vs 2.7%), 3 times more complications (14.5% vs 5.2%) and 5 times more mortalities (0.26% vs 0.05%). Interestingly there was also a significant difference in case volumes between the two groups.
CAVA Robotics has been able to do a similar analysis on robotic surgery thanks to its database gathered from 36 institutions, with 205 surgeons, and over 200,000 cases. They found a similar pattern, using case volume as a delimiter, and found differences in operating times, complications rates, length of stay, and re-admissions.
The tables below illustrate the data for two procedures, Cholecystectomies and Benign Hysterectomies.
An interesting similarity is the same approximate 3-fold difference in complications that was found in both the CAVA group and the Berkmayer paper.
One can also estimate the cost impact of the skills variation by taking average costs for OR time, complications, LOS and re-admissions, on a very conservative estimate the difference between the top and the bottom performers could be between $7,000 and $10,000 per case when these factors are taken into account. With the average US hospital doing 250 robotic cases a year and 62 of these cases being carried out by the bottom quartile, this could cost the institution $500,000 more than those cases being carried out by the top 25%.
Faced with this scenario what are hospitals to do? One option, which is being actively implemented by some institutions, is to severely limit access to the new technology only to surgeons who have demonstrated a high level of skill and who have completed a high volume of cases. Another option is put in proficiency thresholds (using simulation) that surgeons must meet before they are allowed to continue operating.
In a paper on the predictive validity of simulation training by Dr. Patrick Culligan, it was demonstrated that a properly developed curriculum with expert level benchmarks would allow new surgeons to achieve expert level clinical performance starting with their first case. However, the study showed that an average of 20 hours of simulation was required and in some cases nearly 40 hours to achieve these results. In another study done by Dr. Andrea Moglia, the fact that surgeons might have different innate abilities was confirmed and the study was able to identify that only 6% of a student population had native expert level skills, while for the majority it was more about how much training time they would need to reach the same level of skills.
In short, any new technology as it is introduced to into an organization will have a cost impact, be it through the disruption it has to that organization or whether it is caused by the skill variation of its users. Making sure that all team members are properly trained to validated proficiency levels, with the support of simulation systems, will help reduce unfavorable outcomes for the patient and reduce the costs for the institutions.