- Research article
- Open Access
High rates of early treatment discontinuation in hepatitis C-infected US veterans
BMC Research Notes volume 7, Article number: 266 (2014)
Patients with chronic hepatitis C (HCV) frequently discontinued dual therapy with pegylated interferon alfa (Peg-IFN) plus ribavirin (RBV) before reaching the recommended duration of 48 or 24 weeks for genotypes (G) 1/4 or 2/3, respectively. We quantified rates of discontinuation despite efficacy (non-LOE) versus lack of efficacy (LOE) versus discontinuation for unknown reasons in a national database of United States veterans.
We identified a population-based cohort of U.S. veterans with encounters from 2004 through 2009 who had lab-confirmed HCV infection and initiated therapy with Peg-IFN plus RBV in Veterans Health Administration medical centers. Pharmacy data were used to determine therapy duration, defined as the sum of Peg-IFN days supplied. Patients “discontinued” if they failed to receive at least 44 (G1/4) or 20 weeks (G2/3) of therapy. We classified discontinuations as due to non-LOE, LOE, or unknown reasons using a classification rule based on treatment duration and laboratory confirmed response.
Of 321,238 diagnosed HCV patients during the evaluation period, 9.7% initiated therapy and 6.4% met all other inclusion criteria. 54.9% of patients discontinued early; of these, 41.2% discontinued due to non-LOE reasons, 12.5% discontinued for LOE reasons, and 46.3% discontinued for unknown reasons. Among non-LOE discontinuers, most (60.1%) discontinued in the first 4 weeks of therapy, which constitutes 13.6% of all treated patients.
We observed a high proportion of early discontinuations with dual-therapy regimens in a national cohort of HCV-infected veterans. If this trend persists in the triple-therapy era, then efforts must be undertaken to improve adherence.
Until the 2011 approval of direct-acting antiretrovirals, the recommended first-line treatment for chronic hepatitis C virus (HCV) infection for genotypes 1–4 was a combination of pegylated interferon alfa (Peg-IFN) and ribavirin (RBV), for a duration of 24–48 weeks depending on genotype; treatment discontinuation at 12 weeks in patients failing to show at least a 2-log viral load reduction was advised . This dual therapy was associated with a relatively high rate of treatment failure, largely due to poor adherence, early therapy discontinuation, and/or dose reduction due to adverse events [2, 3].
The advent of triple therapy for genotype 1 patients appears to demonstrate improved response rates in clinical trials, [4, 5] but because the new antiretrovirals are added to existing dual therapy regimens, poor adherence and early therapy discontinuation driven by adverse effects may persist in routine clinical practice. Furthermore, to our knowledge, no epidemiologic studies of the US veteran population have characterized the earliest discontinuers. In this study, we describe the duration of treatment and the frequencies, rates, and types of treatment discontinuation for dual therapy in a US veteran population, with particular attention paid to early discontinuations for reasons other than lack-of-efficacy (i.e., non-LOE discontinuations). The rate of early non-LOE discontinuations in dual therapy may carry over into triple therapy regardless of its increased effectiveness over dual therapy.
Design and datasets
We conducted descriptive analyses of dual-therapy patients from every geographic region of the Veterans Affairs (VA) healthcare system, comprising a national cohort. Datasets were acquired and analyzed within the VA INformatics and Computing Infrastructure (VINCI) environment and included Corporate Data Warehouse (CDW; height, weight, laboratory results, and patient care notes for liver-clinic patients), Medical SAS (inpatient and outpatient visits, including procedures and diagnosis codes), and Decision Support Systems (DSS; outpatient pharmacy data).
All relevant ethical safeguards have been met in relation to patient or subject protection. IRB approval for this study was obtained through the University of Utah’s IRB and the VA’s Office of Research and Development.
For patients with missing genotype in the structured laboratory data, natural language processing (NLP) techniques were used to mine the notes for the missing information. First, treated patients in the cohort with no genotype result in structured laboratory data were identified and liver clinic notes for those patients were extracted and randomized. Clinical experts then manually reviewed a subset of the randomized notes focusing on identifying sections of the note that typically contained genotype and on the typical patterns in which it is represented. Next, we used the identified sections and pattern types to build an NLP system using a character window surrounding the identified target words and processed a larger sample of the randomized notes, outputting the word window string and the extracted genotype. Clinical experts then compared the machine-extracted genotypes with the liver clinic notes to correct and flag any errors. This reference standard was used to correct identified errors in the NLP system. We repeated this iterative loop until acceptable measures of machine performance were reached. The NLP system was then used to process a final set of un-annotated randomized notes. To calculate the accuracy of the tool, clinical experts reviewed a subset of the genotype observations and checked them against the original notes to identify any errors. The system output was compared to the human-created reference standard and descriptive statistics for the performance of the tool were assessed. Accuracy was calculated as the number of observations correctly determined by the NLP tool divided by the total number of observations.
We identified all United States (US) veterans with inpatient/outpatient encounters in 2004–2009 who had HCV diagnoses and laboratory-confirmed genotypes 1–4. Diagnosis codes from the 9th revision of the International Classification of Diseases (ICD-9) were used to identify HCV. The subset that initiated dual therapy with RBV plus Peg-IFN 2a or 2b during the study period was included. Patients were considered to have received dual therapy if they received the first prescription for one drug within 30 days of the other.
The index date was the first prescription fill for Peg-IFN. To maximize the chances that patients would be treatment naïve, we excluded patients with prior Peg-IFN or RBV recorded in the VA dataset between 2002 and 2004 or those without at least 1 outpatient visit to the VA 180+ days prior to the index date, which we considered a demonstrated pattern of receiving routine care in the VA. We also excluded patients whose pharmacy data contained missing or invalid National Drug Code (NDC) numbers, such that we could not determine the quantities dispensed or days supplied of Peg-IFN, as well as patients who died before the end of their treatment duration (24 weeks for G2/3; 48 weeks for G1/4).
The primary outcome was non-LOE discontinuation as determined by a discontinuation algorithm based on pharmacy treatment duration and laboratory patient response data (Table 1). Treatment duration was defined as the sum of Peg-IFN days supplied, which has been determined to be an adequate proxy for patient adherence to medication in large population studies . Discontinuers were those who failed to complete at least 44 weeks (G1/4) or 20 weeks (G2/3) of therapy. Gaps in treatment longer than 60 days following the end of previous days supplied defined discontinuation. Discontinuations were further classified as “non-LOE” (those that occurred in the first 10 weeks before 12-week laboratory measures, or occurred after lab results documenting at least a 2-log decrease in viral load); “LOE” (those that occurred after week 10 and after lab results documenting less than a 2-log decrease in viral load), or “unknown” (those that occurred after week 10 but with insufficient laboratory evidence to determine a response). We also classified non-LOE discontinuation events by time to discontinuation: ≤4 weeks, 4–10 weeks, and ≥10 weeks. We used the 10-week threshold for the 12-week laboratory measurement to ensure that laboratory measures that were taken early were still included.
Descriptive statistics were used to characterize patient demographics. Discontinuation frequencies were reported overall and by time to discontinuation. The percentages of patients who discontinued at each time point between genotype groups (G1/4 versus G2/3) were compared using a Chi-square test.
A total of 66,151 liver clinic notes for 4,410 patients were processed using the NLP tool, which yielded 75,187 observations of genotype in 4,186 patients. Human annotations on a random sample of N = 1,000 revealed an accuracy of 96.8% (95% CI = 95.7% to 97.9%) for the extracted genotype. Extracted genotypes were incorporated into the structured data for determining patient study eligibility.
As shown in Figure 1, out of 321,238 HCV patients with an ICD-9 diagnosis code, 31,215 (9.7%) initiated dual-therapy with Peg-IFN plus RBV. Of those, we were able to identify genotype in 24,839 (79.6%) including 20,653 using structured laboratory data and 4,186 using NLP. The subset meeting all other eligibility criteria was 20,499 (6.4%; including 4,092 patients with NLP-extracted genotype), and was included in the analysis. Baseline characteristics of the cohort are summarized in Table 2. The mean (SD) age of the cohort was 53.5 (6.0), and 96.1% were male. Race was known in 42.9% of the cohort: 26.2% white, 13.4% black, and 3.3% Hispanic. The vast majority of patients were overweight (41.9%) or obese (32.7%). The most prevalent comorbidities in the cohort were tobacco use (36.6%), alcohol use (27.4%), mental health issues (24.3%), and drug abuse (24.3%).
As shown in Figure 2, out of the 20,499 patients who initiated treatment, over half (54.9%) failed to complete at least 44 (G1/4) or 20 (G2/3) weeks of therapy. By genotype, a much higher percentage of patients with G1/4 failed to complete therapy (64.0%) compared to G2/3 (30.7%). Our discontinuation algorithm enabled us to infer the reason for discontinuation for 53.7% of non-completing patients. Due to missing laboratory data (i.e., neither in structured data nor in clinic notes), the reasons for discontinuation could not be inferred for 25.4% of patients overall, 30.6% of patients with G1/4, and 11.5% of patients with G2/3. Among discontinuers, 41.2%, 12.5%, and 46.3% of patients were classified “non-LOE”, “LOE”, and “unknown”, respectively. By genotype, these numbers were 37.4%, 14.7%, and 47.9% for G1/4 and 62.5%, 0.2%, and 37.3% for G2/3 (see Table 2). Very few G2/3 patients were classified as LOE discontinuers.
Among the 4,637 non-LOE discontinuers, 2,785 (60.1%) discontinued in the first 4 weeks of therapy, which constitutes 13.6% of all treated patients (see Figure 2). This finding was consistent across genotype groups (58.8% of G1/4 non-LOE discontinuers and 64.3% of G2/3 non-LOE discontinuers). Discontinuation for non-LOE reasons occurred less after the initial 4 weeks of treatment, but the vast majority still discontinued before 10 weeks. The rates of non-LOE discontinuation between 4 and 10 weeks of therapy were 30.8% overall, 30.4% for G1/4, and 31.9% for G2/3. The proportions of patients who discontinued for non-LOE reasons after a documented 2-log lowering in viral load beyond week 10 were 9.2% overall, 10.8% for G1/4, and 3.8% for G2/3. Differences in the proportions who discontinued at each time point were statistically significantly for G1/4 versus G2/3 (p < 0.01).
We have documented dual therapy discontinuation rates for HCV patients in a national cohort of US veterans and estimated the percentages that discontinued for LOE and non-LOE reasons. We found that, while 69.3% of patients with G2/3 completed therapy, only 36.0% of patients with G1/4 did so. Furthermore, of those who could be classified as non-LOE discontinuers, 90.9% discontinued before the first follow-up lab test typically performed at 10–12 weeks, and 59.5% discontinued within the first month.
The high, early, non-LOE discontinuation rates we observed are very concerning. We theorize that poor tolerability of the regimen is a likely explanation. A recent survey of managed-care patients found that 69% of patients listed physical side effects and 36% listed mental side effects as the most difficult aspects of dual therapy . The addition of protease inhibitors to the regimen will likely do nothing to mitigate the adverse effect profile of HCV treatment, and may worsen it.
These findings have important implications for the new triple therapy regimens. While clinical trials have shown the new protease inhibitors substantially improve treatment efficacy, [4, 5] early discontinuations could limit the potential effectiveness of these regimens in clinical practice. Treatment algorithms for boceprevir-based triple therapy regimens recommend a 4-week lead-in phase with dual therapy before adding the protease inhibitor;  our findings suggest that a large proportion of patients may discontinue in the first month, and hence would not continue to the boceprevir combination phase. For telaprevir-based regimens, guidelines recommend the initiation of the protease inhibitor on day 1 along with Peg-IFN and RBV . In this case, if the high early discontinuation rates observed with dual therapy persist in the context of triple therapy, then the cost of a month or more of a protease inhibitor will simply be added to a failed attempt at treatment in a large proportion of veterans. This suggests that adherence-enhancing initiatives will be crucial for ensuring that the efficacy benefits of protease inhibitors can be translated from clinical trials to the real-world setting.
The superior completion rates and the lower rate of unknowns among patients with G2/3 compared to G1/4 are most likely explained by two factors. First, the treatment duration for G2/3 is shorter than for G1/4 (24 weeks vs. 48 weeks), and in our study, patients with G2/3 had to persist only 20 weeks, compared to 44 weeks for G1/4. However, even at 4 weeks, the discontinuation rate was higher for patients with G1/4 compared to G2/3 (14.1% vs. 12.3%; p-value <0.001). The second explanation has to do with the response rates of the different genotypes. Since patients with G2/3 are more likely to have a rapid virological response (RVR, defined as undetectable HCV RNA at week 4 of treatment; 70-75% of G2/3 achieve RVR) compared to genotype 1 (15-25% achieve RVR), [1, 9, 10] we theorize that earlier positive feedback in the form of RVR may encourage patients to persist. We were nonetheless surprised to find that merely three G2/3 patients discontinued due to LOE.
To put our findings in the context of other research, only one other study has evaluated discontinuation rates in a national cohort of HCV-infected US veterans . In their evaluation, Beste and colleagues focused only on genotype 1 and reported that only 30.9% discontinued early (defined as receiving refills for less than 80% of the expected treatment duration) despite evidence of a response to treatment. However, their study included only patients who had received at least 2 fills of Peg-IFN, and consequently most early discontinuers were not considered. Ours is the first study to characterize discontinuation rates for all veterans with genotypes 1–4 who received at least an initial fill of Peg-IFN + RBV.
Our study’s discontinuation rate also falls within the range cumulatively established by studies of other populations of U.S. HCV patients. Estimates of early discontinuation rates include 24% among methadone-maintained patients in California , 36% within a national managed care health plan , and 59% among a non-randomized sample of African Americans and Hispanic patients in New York . Discontinuation rates also vary in U.S. veteran populations. In a regional cohort of veterans undergoing substance abuse treatment, early discontinuation estimates were 60% among G1/4 patients and 27-32% among G2/3 patients . In two U.S. cohorts of primarily genotype 1-infected veterans treated with interferon or Peg-IFN, more than 75% failed to complete 48 weeks of treatment [16, 17], though discontinuers are not distinguished by their reason for discontinuation. Backus et al. report cumulative days’ supply instead of weeks on treatment for genotype 1-, 2-, and 3-infected veterans; 43% of genotype 1 patients, 30% of genotype 2 patients, and 25% of genotype 3 patients did not have 100% cumulative days’ supply .
Although we did not collect data on SVR rates for our study, our 36.0% and 69.3% therapy completion rates for G1/4 and G2/3, respectively, are also not inconsistent with the VA SVR rates reported by Backus et al. . They found SVR rates of 35%, 72%, and 62% for genotypes 1, 2, and 3, respectively. However, they only included patients who had a “post-treatment HCV RNA test”, which would be a subset of all early discontinuers. This might inflate both the proportions of therapy completion as well as SVR because of the smaller denominator.
Our study is subject to several limitations. First, although we carefully tracked medication possession through pharmacy refill records, as is commonly done in large-scale, epidemiologic database studies, we have no way of knowing how many doses were actually taken. Pharmacy refill records are widely accepted as a proxy for adherence in such situations . Another significant limitation of our study concerns missing data: not only the high proportion of patients missing demographic data such as race (57.1% of our cohort), but especially patients with unknown reasons for discontinuation (i.e., patients with insufficient laboratory data for assessing response at week 12). We were unable to classify discontinuation outcomes for 30% of our cohort, or 46% of all discontinuers. The implications of this limitation depend on the underlying causes of the missing observations. If patients with missing laboratory response data (i.e., neither in structured data nor in clinic notes) were mostly responders, then the proportion of non-LOE discontinuers after week 10 could be much higher than we estimate here. If patients with missing laboratory response data were mostly non-responders, then we would have a much higher percentage of LOE discontinuers. However, there would be no difference in the proportions of early discontinuers reported here. A final limitation is our lack of data on the specific non-LOE reasons for treatment discontinuation in those patients, whether side effects, inconvenience, or other reasons. It is possible that a proportion of non-LOE discontinuers would have the reason recorded in their clinical notes; future work may include developing an NLP tool to extract reasons for non-LOE discontinuation.
Our decision to include the “acute hepatitis C infection” diagnosis code as an inclusion criterion may have been controversial. In fact, almost one-third of our cohort had at least one diagnosis code for acute infection. We arrived at the decision to include them because clinician experts on our research team felt that acute hepatitis C is very rare and that most patients with this diagnosis were likely misclassified; this conclusion is also supported by research showing that most acute infections are asymptomatic and are thus undiagnosed . In addition, while there is little consensus on how to approach treatment of acute infections, most recommend a 4–6 month waiting period to see if the patient spontaneously clears the virus. After that, among those who fail to clear the virus, treatment options range from a course of monotherapy with interferon  to treatment with the full 24- or 48-weeks of dual therapy. Those treated with monotherapy would have been excluded (since a second inclusion criterion was that they had to have initiated dual therapy) while those treated with dual therapy would have been representative of the discontinuation rate we were trying to characterize, and thus it was legitimate to include them. Thus, our analysis is robust to this concern.
We observed a high proportion of early discontinuations with dual-therapy regimens in a national cohort of HCV-infected veterans. If this trend persists in the triple-therapy era, then efforts must be undertaken to improve adherence so that the efficacy benefits of the new protease inhibitors can be realized by patients.
Ghany MG, Strader DB, Thomas DL, Seeff LB: Diagnosis, management, and treatment of hepatitis C: an update. Hepatology. 2009, 49 (4): 1335-1374. 10.1002/hep.22759.
McHutchison JG, Manns M, Patel K, Poynard T, Lindsay KL, Trepo C, Dienstag J, Lee WM, Mak C, Garaud JJ, Albrecht JK: Adherence to combination therapy enhances sustained response in genotype-1-infected patients with chronic hepatitis C. Gastroenterology. 2002, 123 (4): 1061-1069. 10.1053/gast.2002.35950.
Manns MP, McHutchison JG, Gordon SC, Rustgi VK, Shiffman M, Reindollar R, Goodman ZD, Koury K, Ling M, Albrecht JK: Peginterferon alfa-2b plus ribavirin compared with interferon alfa-2b plus ribavirin for initial treatment of chronic hepatitis C: a randomised trial. Lancet. 2001, 358 (9286): 958-965. 10.1016/S0140-6736(01)06102-5.
Jacobson IM, McHutchison JG, Dusheiko G, Di Bisceglie AM, Reddy KR, Bzowej NH, Marcellin P, Muir AJ, Ferenci P, Flisiak R, George J, Rizzetto M, Shouval D, Sola R, Terg RA, Yoshida EM, Adda N, Bengtsson L, Sankoh AJ, Kieffer TL, George S, Kauffman RS, Zeuzem S: Telaprevir for previously untreated chronic hepatitis C virus infection. N Engl J Med. 2011, 364 (25): 2405-2416. 10.1056/NEJMoa1012912.
Poordad F, McCone J, Bacon BR, Bruno S, Manns MP, Sulkowski MS, Jacobson IM, Reddy KR, Goodman ZD, Boparai N, DiNubile MJ, Sniukiene V, Brass CA, Albrecht JK, Bronowicki JP: Boceprevir for untreated chronic HCV genotype 1 infection. N Engl J Med. 2011, 364 (13): 1195-1206. 10.1056/NEJMoa1010494.
Steiner JF, Prochazka AV: The assessment of refill compliance using pharmacy records: methods, validity, and applications. J Clin Epidemiol. 1997, 50 (1): 105-116. 10.1016/S0895-4356(96)00268-5.
Murphy R, Shvachko V, Manos M: Patient perspectives about the treatment of Hepatitis C (PATH-C). 2011, San Francisco, CA: Hepatology: American Association for the Study of Liver Diseases
Ghany MG, Nelson DR, Strader DB, Thomas DL, Seeff LB: An update on treatment of genotype 1 chronic hepatitis C virus infection: 2011 practice guideline by the American Association for the Study of Liver Diseases. Hepatology. 2011, 54 (4): 1433-1444. 10.1002/hep.24641.
Ferenci P, Fried MW, Shiffman ML, Smith CI, Marinos G, Goncales FL, Haussinger D, Diago M, Carosi G, Dhumeaux D, Craxi A, Chaneac M, Reddy KR: Predicting sustained virological responses in chronic hepatitis C patients treated with peginterferon alfa-2a (40 KD)/ribavirin. J Hepatol. 2005, 43 (3): 425-433. 10.1016/j.jhep.2005.04.009.
Shiffman ML, Suter F, Bacon BR, Nelson D, Harley H, Sola R, Shafran SD, Barange K, Lin A, Soman A, Zeuzem S: Peginterferon alfa-2a and ribavirin for 16 or 24 weeks in HCV genotype 2 or 3. N Engl J Med. 2007, 357 (2): 124-134. 10.1056/NEJMoa066403.
Beste LA, Ioannou GN, Larson MS, Chapko M, Dominitz JA: Predictors of early treatment discontinuation among patients with genotype 1 hepatitis C and implications for viral eradication. Clin Gastroenterol Hepatol. 2010, 8 (11): 972-978. 10.1016/j.cgh.2010.07.012.
Sylvestre DL, Clements BJ: Adherence to hepatitis C treatment in recovering heroin users maintained on methadone. Eur J Gastroenterol Hepatol. 2007, 19 (9): 741-747. 10.1097/MEG.0b013e3281bcb8d8.
Mitra D, Davis KL, Beam C, Medjedovic J, Rustgi V: Treatment patterns and adherence among patients with chronic Hepatitis C virus in a US Managed Care Population. Value Health. 2010, 13 (4): 479-486. 10.1111/j.1524-4733.2009.00691.x.
Satapathy SK, Lingisetty CS, Proper S, Chaudhari S, Williams S: Equally poor outcomes to pegylated interferon-based therapy in African Americans and Hispanics with chronic hepatitis C infection. J Clin Gastroenterol. 2010, 44 (2): 140-145. 10.1097/MCG.0b013e3181ba9992.
Huckans MS, Loftis JM, Blackwell AD, Linke A, Hauser P: Interferon alpha therapy for hepatitis C: treatment completion and response rates among patients with substance use disorders. Subst Abuse Treat Prev Policy. 2007, 2: 4-10.1186/1747-597X-2-4.
Butt AA, McGinnis KA, Skanderson M, Justice AC: Hepatitis C treatment completion rates in routine clinical care. Liver Int. 2010, 30 (2): 240-250. 10.1111/j.1478-3231.2009.02156.x.
Iqbal SU, Cunningham F, Lee A, Miller DR, Li NC, Cheung R, Kazis L: Persistence with hepatitis C therapy in the Department of Veterans Affairs. J Clin Pharm Ther. 2008, 33 (3): 251-261. 10.1111/j.1365-2710.2008.00912.x.
Backus LI, Boothroyd DB, Phillips BR, Belperio P, Halloran J, Mole LA: A sustained virologic response reduces risk of all-cause mortality in patients with hepatitis C. Clin Gastroenterol Hepatol. 2011, 9 (6): 509-516. 10.1016/j.cgh.2011.03.004. e501
Vogel M, Deterding K, Wiegand J, Gruner NH, Baumgarten A, Jung MC, Manns MP, Wedemeyer H, Rockstroh JK: Initial presentation of acute hepatitis C virus (HCV) infection among HIV-negative and HIV-positive individuals-experience from 2 large German networks on the study of acute HCV infection. Clin Infect Dis. 2009, 49 (2): 317-319. author reply 319
Wiegand J, Buggisch P, Boecher W, Zeuzem S, Gelbmann CM, Berg T, Kauffmann W, Kallinowski B, Cornberg M, Jaeckel E, Wedemeyer H, Manns MP: Early monotherapy with pegylated interferon alpha-2b for acute hepatitis C infection: the HEP-NET acute-HCV-II study. Hepatology. 2006, 43 (2): 250-256. 10.1002/hep.21043.
This material is the result of work supported with resources and the use of facilities at the George E. Wahlen Department of Veterans Affairs Medical Center, Salt Lake City, Utah. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States government. This work was sponsored by a grant from Genentech. Dr. LaFleur has also been supported by Agency for Healthcare Research and Quality (AHRQ; HS018582) and Amgen during the writing of this manuscript. Dr. Morgan has received research funding to perform clinical trials from Genentech, Hoffmann-LaRoche, Merck, Bristol-Myers-Squibb, Gilead and Vertex. Dr. Pandya has received research funding from Genentech, Hoffmann LaRoche, and Merck. Dr. DuVall has received research grants from Anolinx, LLC, Genentech, Roche, Amgen, and Shire, and was a Partner in 2010–2011 for Clinical Methods. Dr. Nelson was supported in part by funding from the National Institutes of Health and the National Cancer Institute grant 1 KM1CA156723 and the National Institutes of Health Office of the Director grant 5TL1RR025762-03.
This work was sponsored by a grant from Genentech. Co-authors Robert Hoop and Eli Korner are employees of Genentech and own Genentech stock. Dr. LaFleur has also been partially supported by Agency for Healthcare Research and Quality (AHRQ; HS018582) during the writing of this manuscript and has received research grants from Amgen, Genentech, Merck, Novartis, and Anolinx. Dr. Morgan has received research funding to perform clinical trials from Genentech, Hoffmann-LaRoche, Merck, Bristol-Myers-Squibb, Gilead and Vertex. Dr. Pandya has received research funding from Genentech, Hoffmann LaRoche, and Merck. Dr. DuVall has received research grants from Anolinx, LLC, Genentech, Roche, Amgen, and Shire, and was a Partner in 2010–2011 for Clinical Methods. Ms. Knippenberg and Ms. Hayden are University of Utah employees but have no other competing interests to declare. Dr. Nelson was supported in part by funding from the National Institutes of Health and the National Cancer Institute grant 1 KM1CA156723 and the National Institutes of Health Office of the Director grant 5TL1RR025762-03. No other competing interests are declared.
JL: took the lead in planning/conducting the study, analyzing and interpreting the data, and drafting the manuscript. She has approved the final draft submitted. RH: assisted in planning the study, interpreting the results, and contributed to drafts of the manuscript. He has approved the final draft submitted. TM: assisted in planning the study and interpreting the results, and contributed edits to drafts of the manuscript. He has approved the final draft submitted. PP: assisted in planning the study and interpreting the results, and contributed edits to drafts of the manuscript. He has approved the final draft submitted. EK: assisted in interpreting the results and contributed edits to drafts of the manuscript. He has approved the final draft submitted. KK: contributed substantially to the writing and editing of the manuscript. She has approved the final draft submitted. CH: contributed to analyzing the data and writing/editing the manuscript. She has approved the final draft submitted. SD: oversaw the Natural Programming Language (NPL) component of this study and contributed to drafts of the manuscript. He has approved the final draft submitted. RN: contributed substantially to planning the study, analyzing the data, interpreting results, and drafting/editing the manuscript. He has approved the final draft submitted. JL is the author accepting full responsibility for the conduct of the study. She has had access to the data and control of the decision to publish. All authors read and approved the final manuscript.