Refinement of a pharmacogenomics app for dosing guidelines for oncology: findings from the usability evaluation

Introduction

At the present time, more than 77,000 genetic tests are on the market and around 10 new tests enter the marketplace daily (1,2). Pharmacogenetic testing is a distinct classification of genomic testing utilized to individualize prescriptive treatments (3). This type of testing assists with the evaluation of drug toxicity and efficacy before prescribing a specific drug (4). The FDA has approved over 300 drugs with pharmacogenetic information included in the drug labels (5). Moreover, over 100 of these approved pharmacogenetic FDA recommendations are specific to oncology medication. As part of a long-established custom, prescriptions have typically been based on a consensus of data rather than one’s genetic information (6). Furthermore, medications have been connected with adverse drug reactions, but pharmacogenetics, a subset of precision medicine, provides a course of action to individualize dosages and provide tailored drug therapy. This new insight in prescribing techniques has resulted in the emergence of a knowledge deficit for many health care providers on the correct way to use pharmacogenetics in practice. Providers are at the forefront of patient care, which makes them well situated to educate patients about newly discovered technologies associated with their health. Therefore, health care providers play an essential role in the integration of pharmacogenetic testing and genotype-guided therapy into customary practice.

Pharmacogenetic testing in oncology

Pharmacogenetic testing within oncology has been observed in the early phases of this field. In 2007, a study by Fargher et al. (7) revealed that two-thirds of providers surveyed in England used a pharmacogenetic test for the genetic variant of thiopurine methyltransferase (TPMT) prior to Azathioprine treatment. Most recently, the focus has shifted to cost-effectiveness studies related to the use of pharmacogenetic testing in oncology due to the vast prevalence of this type of precision medicine. According to Farugue et al., pharmacoeconomic studies on fluoropyrimidine, 6-mercaptopurine, irinotecan, carboplatin, cisplatin, erlotinib, gefitinib, cetuximab, panitumumab, and trastuzumab were conducted in Asia, Europe, Canada, the United States, and Mexico (8). Overall, these studies showed cost-utility, cost-effectiveness, and cost-minimization outcomes. The use of pharmacogenetic testing as compared to usual prescribing methods had more cost saving in studies conducted in the United States (n=9), Europe (n=6), and Asia (n=1).

Despite the frequency of genetic variants seen within the vast population and more specifically within oncology, a major obstacle to clinical implementation of precision medicine exists related to the convenient access to clinical practice guidelines. Poor or limited access is a barrier to the diffusion of innovation. One such barrier is the lack of these dosing algorithms on databases within the medical center. Furthermore, Clinical Pharmacogenetics Implementation Consortium (CPIC) provides guidelines to make specific prescription decisions, but this information is only available on one website (9). CPIC guidelines entail valuable information to help health care providers translate evidence to formulate specific prescriptive dosages (10). Although both regulatory and clinical practice rely on scientific data to guide decisions, the lack of convenient access to pharmacogenetic guidelines and funding and expertise for integration within current electronic medical record systems may, in part, explain heterogeneity in clinical uptake of pharmacogenetic testing following labeling updates (11,12). Therefore, an easily accessible and cost-effective solution via mobile app could improve access to information surrounding pharmacogenetic testing.

Over the past few years, the market for mobile apps has more than doubled its size. This evolution has applied pressure to the app developers, because with more users comes more applications to choose from. A new study revealed that the main motives why users stop using applications include “I found a better app” and “technical problems” (13). These statements imply that the users are progressively insisting on applications that fulfill their needs and expectations. Based on this information, it is obvious that ignoring usability may result in losing the users’ interest or prevent users from utilizing the potential benefits of the application in general. Therefore, usability needs to be considered when developing mobile applications.

The guiding framework for this study is Rogers’ diffusion of innovation theory to examine adoption of a decision support app by health care providers (14). To encourage guideline adoption by providers, a readily accessible decision support tool should be developed and tested. The use of decision support tools utilized for prescribing medication is sparse within the literature; however what evidence is available is encouraging. Among five studies, an improvement of outcomes and reduced costs when prescribing antibiotics were noted (15-19). In another study, Manca et al. found clinical decision support tools prevent complications in patients with chronic illnesses (20). No studies have been conducted to understand the utility of such a tool in precision medicine. Use of mobile apps to prescribe medication has dramatically increased since the introduction of digital decision support tools (21,22). Studies have shown awareness and adoption of these tools are crucial for widespread diffusion among health care providers (21,23). Therefore, successful implementation of mobile apps depends on the attitudes of health care providers. However, after an extensive market research, there are no current mobile apps that provide dosing recommendations based on pharmacogenetic variants. Once adoption occurs, broader use of mobile apps for pharmacogenomics is expected.

This research team has had prior success in the development a clinical decision support tool via mobile app for one gene variant, TPMT and its associated drug, Mercaptopurine. A protype of the clinical decision support tool has been effectively developed and preliminary usability testing with focus groups within the oncology field has been completed. A sample of 10 nurse practitioners who actively prescribe medication were recruited via email from previous partnerships with local healthcare systems (6). In addition to this convenience sampling strategy, a snowballing approach was used to recruit more participants. Ultimately, the sample size for the qualitative aspect of the study was determined by the level of saturation. Semi-structured interviews were conducted with each nurse practitioner. The interviews followed a semi-structured interview guide to capture information necessary to understand use of clinical decision support tools. These initial findings have provided information on the extreme interest for this tool within the oncology field as well as key areas within the app to revise to encourage adoptability of this app within the oncology field (6).

Based on the evidence, the absence of readily accessible guidelines for precision medicine and the advantages associated with this clinical decision support tool warrant refinement of this mobile app. Due to the vast array of gene variants associated with drug interactions and the feasibility of constructing a mobile app, only two gene variants (CYP2D6 and DPYD) associated with a pharmacogenomic dosing algorithm in oncology was targeted. Therefore, a project to expand a precision medicine clinical decision support tool for oncology medication was proposed. Usability of the clinical decision support app was evaluated by health care providers in oncology. The findings from this study will hopefully inform clinical practice by providing accessible guidelines for prescribing appropriate medications to reduce adverse reactions as well as a tool to educate practitioners and patients on the rationale for specific medication choices.

Methods

Design

Based on information from the literature and preliminary work, a mixed method design was used. Usability testing is divided into two main categories, inspection and testing methods. The testing evaluation was conducted to evaluate usability, measured by the System Usability Scale, and six open-ended questions were included to measure perceptions of the mobile app related to opinions of the usefulness in practice, user-friendliness, barriers to implementation, suggestions for revisions, and suggestions for additions to the mobile application. Ethical approval was received through the University of North Carolina-Wilmington’s IRB (Study # 17-0153). Participants were given an informed consent before taking part in the study. Furthermore, the study conformed to the provision of the Declaration of Helsinki (as revised in 2013).

Sample and setting

The sample included both nurse practitioners and registered nurses. Eligibility criteria were English-speaking and practice in oncology in North Carolina. The setting included both inpatient and outpatient oncology practices. The needed sample size was determined via power analysis. The confidence interval was set at 95% with an effect size of 0.8, which resulted in a calculation of 49. Usability testing was conducted via an online anonymous survey to obtain a better understanding of the needs of oncology practitioners in their use of clinical decision support tools to improve refinement of the mobile app.

Research variables

The main research variable was usability as measured by effectiveness, efficiency, and satisfaction. Usability is a multidimensional quality of a product. Basically, it defines how usable a product is. According to the definition of the International Organizational for Standardization (ISO) standard 9241-11, usability is the “extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” (24). The ISO definition talks about a product, which in this study is referred to as an application or an app. Effectiveness assesses how accurately the users perform actions within the app to complete a task. Efficiency relates to the use of resources, where resources can refer to time, effort or costs. In the case of a mobile application, the resources are usually time, including both physical and mental effort. Thus, efficiency explains how easily the users are able to complete tasks on the application accurately. Satisfaction is the most complicated of the measures due to its subjective nature. To some extent, the user’s own perception of satisfaction is indispensable, and his/her satisfaction with the application is affected by the effectiveness and efficiency of it. All aspects of the application affect the user’s satisfaction. Essentially, as a usability measure, satisfaction portrays how pleasant the application is to use, and how comfortable the users are with all the different parts as well as the interface in general.

Instruments

The System Usability Scale (SUS) (25) measured usability. The SUS is a 10-item scale used to measure users’ subjective perceptions of the usability of an information system. The published psychometric measures of this scale are a Cronbach’s Alpha of 0.85 and the SUS has demonstrated sufficient content and concurrent validity (26). Similar psychometric results have been found within studies that tested the usability of mobile apps for acute stroke guidelines and posttraumatic disorder (27,28). The SUS was completed immediately after the subject used the mobile app to find a recommended dosage based on a scenario of a patient’s genetic testing information. Subjects were asked to record their immediate response, rather than think about items for a long time. In addition, six open-ended questions were asked to analyze common themes regarding usability. Subjects were instructed to answer all items and if they cannot respond to a particular item, should mark the center point of a scale. Furthermore, demographic variables pertaining to discipline of practice, number of years in current profession, age, and sex were collected.

Data collection schedule and procedures

The data collection process entailed two phases. During the first phase, a faculty member from Computer Science refined the functioning mobile application that was previously created by adding two additional gene variants (DPYD and CYP2D6) commonly seen within the oncology field that is associated with an algorithm based on the CPIC guidelines and an educational platform was added to include additional information on pharmacogenetic testing and clinical practice guidelines associated with this type of genetic testing. After the refinement was completed, an expert in the field of personalized medicine from a prominent cancer institute provided his expertise to assess the quality and safety of the mobile app. The recommendation for modifications were completed. Once the extension of the mobile app was completed, a heuristic evaluation by four evaluators from computer science was completed and updates to the mobile app was conducted based on newly published evidence-based practice guidelines. The heuristic evaluation inspected the user interface, comparing the elements and aspects of it to a set of principles, heuristics, as a guideline to evaluate the usability of the mobile app. This inspection method is extensively used according to a survey conducted by the User Experience Professionals Association and can be used to uncover usability in the early stages of development (29).

During Phase 2, an electronic recruitment flyer about the research study was sent via email to oncology health care providers found within the North Caroling Board of Nursing database. A link to the anonymous electronic survey was provided. A follow-up email was sent two weeks later. Within the recruitment email, each participant was given a scenario of an oncology patient’s genetic information that corresponds with a gene-drug interaction programmed within the mobile app. Furthermore, a link to the mobile app was provided. The participant was asked to obtain a recommended drug dosage for the corresponding medication. Once the participant has completed review of the mobile app, the participant completed the System Usability Scale and six open-ended questions. All elements of the data that could be tied to the health care provider was eliminated. No identifying aspects was located with the data. All data was duplicated, and a backup was created.

Statistical analysis

Descriptive statistics of frequency, range, and central tendency were conducted with the demographic variables. Furthermore, measures of central tendency were conducted with the SUS scores. Finally, a thematic analysis of the qualitative data was completed.

Results

Quantitative component

A purposive sampling of 51 health care providers in the oncology setting was successfully recruited. Descriptive statistics were completed with the demographic variables and the results can be found in Table 1. The respondents ranked each of the 10 questions from 1 to 5 based on how much they agreed or disagreed with the statement they were reading. A score of 1 indicated that they strongly disagreed with the statement; a score of 5 indicated that they strongly agreed with the statement. Each respondent’s answers are scored to give an overall usability score out of 100. A SUS score greater than 68 is considered to have successful usability and user-friendliness; scores less than 68 indicate a system that has usability concerns. The results were normalized through percentile ranking.

To calculate the SUS score for each respondent, we subtracted 1 from the score of each of the odd-numbered questions and subtracted the value of the answer from 5 for the even-numbered questions. The sum of the 10 calculated scores is calculated, and this value is multiplied by 2.5 to obtain a SUS score out of 100. This is not a percentage, but a clear way to evaluate the usability score. Overall, the SUS score obtained from the respondents was a 72.99. This showcased successful usability and user-friendliness. Furthermore, the nurse practitioners rated the mobile app with a mean usability score of 73.59 and the registered nurses had a mean score of 72.5. Figure 1 represents the usability scores of each of the 51 participants.

Figure 1 Usability scores.

Qualitative component

Following the data collection process, all open-ended responses were read to identify common themes expressed regarding the respondents’ thoughts regarding the usability of the mobile app and areas for future improvement. The major themes identified related to the modifications to be made to the mobile app were the addition of nursing specific interventions included on the recommendations, addition of more information on each medication included in the mobile app, and a simple explanation on the process to use the mobile app. However, an overwhelming majority of comments focused on lack of need for modifications and the desire to be able to utilize this within practice. Finally, the major barrier noted by the respondents regarding the implementation of this mobile app and pharmacogenetic testing in general was the need for education on when to pharmacogenetic testing should be ordered for patients.

Conclusions

In this study, the participants found the GeneRx application to be highly usable, eliciting the notion of satisfaction with the simplicity of use. This usability rating suggests that the nurses perceive the mobile app to be beneficial to their practice. Furthermore, although the small sample size of the amount of nurse practitioners, the higher usability score found among the nurse practitioners as compared to the registered nurses showcase a clinical difference that may be related to the more frequent use of prescribing oncology medications. In addition to prior discoveries that health care providers are convinced that a usable clinical decision support system for genomics would benefit a patient’s care outcomes, these findings support the fact that efficacious tools will enhance the uptake of pharmacogenomics in patient care settings (30). Ultimately, the biggest strengths of this clinical decision support mobile app are the perceived user friendliness and the portability of this tool within the health care setting. Furthermore, once the provider education portal is fully developed, this can serve as an educational tool for providers who are novice to the field of pharmacogenetics. Although several strengths were identified, one of the biggest weaknesses of this clinical decision support mobile app is the continual need for updates based on novel findings within pharmacogenetic testing. This weakness will be mitigated by updating the mobile app when new or revised clinical practice guidelines are published.

Unlike former studies, this study focused on the application’s potential for adoption among practitioners rather than merely measuring the application interface usability. The respondents were given specific scenarios for determining the prescription of a drug based on pharmacogenomic algorithms. Therefore, their opinions of usability reflected the respondent’s interaction with the mobile app interface as well as how the app would specify the correct dosing from the genetic information given. However, training on the mobile app was not conducted prior to this study. Future research should focus on training health care providers to employ clinical judgment in accepting recommended dosages given by GeneRx. This training would be a valuable addition to education for all health care providers, including pharmacists and physicians and GeneRx could serve as a training tool that pharmacists could use to educate fellow heath care providers. Furthermore, based on the information obtained by the qualitative statements of the respondents, additional improvements to the mobile app will be made to improve the ease of use by adding a calculator function for dosages that require soft biometric traits such as height and weight. Finally, a simple step-by-step explanation of the process for using the mobile app will be added.

Limitations

Two main limitations of this study are the lack of representation from health care providers outside the realm of nursing and the presentation of only three sets of gene-drug algorithms within the GeneRx application. Despite these limitations, this study did provide insight into the usability and prospective uptake of the GeneRx app among oncology nurses. Further studies with a more diverse population of health care providers outside of nursing as well as the addition of more gene-drug algorithms within the GeneRx app are scheduled to acquire more insight into the adoption of mobile apps in pharmacogenomics among health care providers.

Implications

This research study has not only gained valuable information on the perception of nurses related to this specific mobile app, this study has also highlighted the needs of nurses related to the educational barrier of determining when to order pharmacogenetic testing. This information highlights the broader necessity of tools to help simplify the decision-making process for ordering pharmacogenetic test when appropriate. In addition, the formation of this interprofessional team among computer science and nursing will be a catalyst for the development of these useful tools to improve adoption of pharmacogenetic into routine practice. Finally, the inclusion of a nursing perspective provides a unique lens to enhance the applicability of these decision support instruments to improve the healthcare outcomes of our patients.

Acknowledgments

Funding: This work was supported by the Oncology Nursing Foundation Grant: #G58166.

Footnote

Data Sharing Statement: Available at https://atm.amegroups.com/article/view/10.21037/atm-2022-68/dss

Peer Review File: Available at https://atm.amegroups.com/article/view/10.21037/atm-2022-68/prf

Conflicts of Interest: Both authors have completed the ICMJE uniform disclosure form (available at https://atm.amegroups.com/article/view/10.21037/atm-2022-68/coif) and report receiving funding to complete the study by the Oncology Nurses Foundation which was paid to the institution. The authors have no other conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. Ethical approval was received through the University of North Carolina-Wilmington’s IRB (Study # 17-0153). Participants were given an informed consent before taking part in the study. Furthermore, the study conformed to the provision of the Declaration of Helsinki (as revised in 2013).

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. National Library of Medicine [Internet]. Bethesda (MD): Medline Plus. c2021 [2021 Jul 28]. Available online: https://medlineplus.gov/genetics/understanding/testing/genetictesting/
2. Phillips KA, Deverka PA, Hooker GW, et al. Genetic Test Availability And Spending: Where Are We Now? Where Are We Going? Health Aff (Millwood) 2018;37:710-6. [Crossref] [PubMed]
3. National Human Genome Research Institute [Internet]. Bethesda (MD). c2019 [2019 Feb 13]. Available online: https://www.genome.gov/19516567/faq-about-genetic-testing/
4. Mayo Clinic [Internet]. Rochester (MN). c1998-2022 [2020 Sep 3]. Available online: https://www.mayoclinic.org/healthy-lifestyle/consumer-health/in-depth/personalized-medicine/art-20044300
5. Food and Drug Administration [Internet]. Silver Spring (MD). c2021 [2021 Aug 20]. Available online: https://www.fda.gov/drugs/science-and-research-drugs/table-pharmacogenomic-biomarkers-drug-labeling
6. Dodson CH, Baker E, Bost K. Thematic analysis of nurse practitioners use of clinical decision support tools and clinical mobile apps for prescriptive purposes. J Am Assoc Nurse Pract 2019;31:522-6. [Crossref] [PubMed]
7. Fargher EA, Tricker K, Newman W, et al. Current use of pharmacogenetic testing: a national survey of thiopurine methyltransferase testing prior to azathioprine prescription. J Clin Pharm Ther 2007;32:187-95. [Crossref] [PubMed]
8. Faruque F, Noh H, Hussain A, et al. Economic Value of Pharmacogenetic Testing for Cancer Drugs with Clinically Relevant Drug-Gene Associations: A Systematic Literature Review. J Manag Care Spec Pharm 2019;25:260-71. [Crossref] [PubMed]
9. PharmGKB [Internet]. Stanford (CA). c2001-2021 [2021 Aug 13]. Available online: https://www.pharmgkb.org
10. Caudle KE, Klein TE, Hoffman JM, et al. Incorporation of pharmacogenomics into routine clinical practice: the Clinical Pharmacogenetics Implementation Consortium (CPIC) guideline development process. Curr Drug Metab 2014;15:209-17. [Crossref] [PubMed]
11. Hess GP, Fonseca E, Scott R, et al. Pharmacogenomic and pharmacogenetic-guided therapy as a tool in precision medicine: current state and factors impacting acceptance by stakeholders. Genet Res (Camb) 2015;97:e13. [Crossref] [PubMed]
12. Schuck RN, Marek E, Rogers H, et al. Clinical and regulatory considerations in pharmacogenetic testing. Am J Health Syst Pharm 2016;73:1999-2006. [Crossref] [PubMed]
13. Aggarwal R [Internet]. Austin (TX): Localytics. c2021. [2015 Nov 11]. Available online: http://info.localytics.com/blog/new-study-reveals-what-app-users-want-localytics
14. Rogers E. Diffusion of innovations. 5th ed. New York: Free Press, 2003.
15. Demonchy E, Dufour JC, Gaudart J, et al. Impact of a computerized decision support system on compliance with guidelines on antibiotics prescribed for urinary tract infections in emergency departments: a multicentre prospective before-and-after controlled interventional study. J Antimicrob Chemother 2014;69:2857-63. [Crossref] [PubMed]
16. Gulliford MC, van Staa T, Dregan A, et al. Electronic health records for intervention research: a cluster randomized trial to reduce antibiotic prescribing in primary care (eCRT study). Ann Fam Med 2014;12:344-51. [Crossref] [PubMed]
17. Helmons PJ, Coates CR, Kosterink JG, et al. Decision support at the point of prescribing to increase formulary adherence. Am J Health Syst Pharm 2015;72:408-13. [Crossref] [PubMed]
18. Litvin CB, Ornstein SM, Wessell AM, et al. Adoption of a clinical decision support system to promote judicious use of antibiotics for acute respiratory infections in primary care. Int J Med Inform 2012;81:521-6. [Crossref] [PubMed]
19. Warholak TL, Rupp MT, Zale A, et al. Check it out: A practical tool for improving medication safety. J Am Pharm Assoc (2003) 2015;55:621-5. [Crossref] [PubMed]
20. Manca DP, Campbell-Scherer D, Aubrey-Bassler K, et al. Developing clinical decision tools to implement chronic disease prevention and screening in primary care: the BETTER 2 program (building on existing tools to improve chronic disease prevention and screening in primary care). Implement Sci 2015;10:107. [Crossref] [PubMed]
21. Charani E, Kyratsis Y, Lawson W, et al. An analysis of the development and implementation of a smartphone application for the delivery of antimicrobial prescribing policy: lessons learnt. J Antimicrob Chemother 2013;68:960-7. [Crossref] [PubMed]
22. Zaidi ST, Marriott JL, Nation RL. The role of perceptions of clinicians in their adoption of a web-based antibiotic approval system: do perceptions translate into actions? Int J Med Inform 2008;77:33-40. [Crossref] [PubMed]
23. Shelton RC, Trivedi MH. Performance improvement CME: algorithms and EMRs in depression. J Clin Psychiatry 2011;72:e29. [Crossref] [PubMed]
24. International Organization for Standardization [Internet]. London. c2018. [2021 Nov 12]. Available online: https://www.iso.org/standard/63500.html
25. Healthcare Information and Management Systems Society [Internet]. Chicago. c2012. [2021 Mar 10]. Available online: https://www.himss.org/file/1306816/download?token=neHwcuTW
26. Lewis JR, Sauro J. International Conference on Human Centered Design. Berlin: Springer, 2009.
27. Price M, van Stolk-Cooke K, Ward HL, et al. Tracking post-trauma psychopathology using mobile applications: A usability study. J Technol Behav Sci 2017;2:41-8. [Crossref] [PubMed]
28. Shkirkova K, Akam EY, Huang J, et al. Feasibility and utility of an integrated medical imaging and informatics smartphone system for management of acute stroke. Int J Stroke 2017;12:953-60. [Crossref] [PubMed]
29. User Experience Professionals Association [Internet]. Houston. c2016. [17 Dec 2021]. Available online: https://uxpa.org/article/2016-ux-salary-survey-results
30. Baker EW, Dodson CH. Prototype Development and Usability Evaluation of a Clinical Decision Support Tool for Pharmacogenomic Pharmacy in Practice. Comput Inform Nurs 2021;39:362-6. [Crossref] [PubMed]