Appropriateness of tumor marker request: a case of study

Introduction

Appropriateness, defined as “the outcome of a process of decision-making that maximizes net individual health gains within society’s available resources” (1), is a crucial issue in health care systems, encompassing both clinical and organizational aspects, that intersects all health services. Appropriateness is the key to fulfill the requirement of delivering efficient and high-quality health services at affordable costs, which is the goal of all health care systems. Over the last decades, with the exponential increase of knowledge in the biomedical fields and the consequential dramatic technological advances, appropriateness in health care has taken on a more stringent connotation. In fact, health care systems are facing major challenges of offering innovation and clinical advances at individual and population levels in the frame of a global scenario of resource constraint. In this context health care systems may reduce costs through either an unbalanced, linear restriction of services currently furnished, or a selective restriction of inappropriate expenditures. The former option is simple and rapid, but would lead to a progressive deterioration of the quality of health care; the latter, although it could be certainly more complex and time consuming, is the proper approach to guarantee a long-term sustainable development of health care systems.

Laboratory medicine represents a health care sector of special interest for investigation and management of inappropriateness for several reasons. First, knowledge growth is translated into technological innovations with a probably higher rate in laboratory medicine than in any other clinical field. Second, while requiring a minimally invasive approach, laboratory tests have a pivotal role in the diagnostic process of many diseases and in different clinical settings. In parallel, laboratory tests have a limited cost when considered in isolation and are not restrained by organization barriers—such as waiting lists—which could “physiologically” limit their access, as for example may occur in the case of imaging tests. Not surprisingly, laboratory tests represent the single highest-volume of the procedures related to medical activity (2). Finally, laboratory tests may be requested by several professionals, such as general practitioners, hospital clinicians, consultants and other health care professionals as the case of waived tests (3). In addition, direct-to-consumer testing is also expected to rapidly grow in response to consumer demands and declining prices (4). All the above variables are potentially related to an increased risk of inappropriate use.

General variables (i.e., country, type of test, health care organization) possibly related to different ordering pattern have been investigated. Zhi et al. did not find significant differences of appropriateness rates among studies performed in different countries, neither among the different diagnostic area that were explored (5). A previous study of our group, performed at a regional scale, showed that the appropriateness rate of tumor marker orders was not dependent on factors related to the institution (i.e., territorial area covered by the institution, presence of oncology facilities, declared adherence to guidelines) or laboratory organization (i.e., certification and/or accreditation process) (6).

Possible causes of the continuous increase in medical laboratory testing have been extensively investigate and both patient-related and doctor-related factors have been identified as causal variables (7). Of special interest are some subjective factors, such as patient need for reassurance, fear of missing an unexpected diagnosis and fear of litigation, which are prominent when dealing with serious diseases, perceived as potentially life threatening. In this scenario, the use of circulating tumor biomarkers for the diagnosis and management of cancer can be cited as an emblematic example, and may represent an interesting case of study for appropriateness.

The multifaceted aspects of inappropriateness

Inappropriate testing is a complex phenomenon that can occur at different levels, such as: (I) test ordering; (II) patient preparation; (III) specimen collection, handling and storage; (IV) assay of the proper analyte; (V) interpretation and clinical use of test result. Several of these phases are managed within quality assurance systems and are under the responsibility of laboratory staff. Inappropriate actions in the pre-analytical and analytical phases are therefore regarded as process errors, identified, amended and prevented through routine quality assurance procedures. In contrast, test ordering and utilization of test results in the medical decision process are typically managed by clinicians and the control by the laboratory personnel may be performed only in an indirect way. Laboratory may influence appropriateness by acting on two sides: (I) an effective translation of knowledge through regular educational and informative interventions; and (II) monitoring either ordering patterns, or actions undertaken on the basis of a test result, or both. The assessment of ordering pattern is a typical process analysis, in which the monitored action—test ordering—is examined per se, whereas the appraisal of downstream actions following a test result fits in the area of outcome analysis. Although actions undertaken on test results must certainly be considered crucial for the appropriate use of laboratory resources, they are in fact driven by a multiplicity of factors, making difficult to evaluate the real contribution of the laboratory result on clinical outcomes.

On the other hand, monitoring test ordering and assessment of appropriateness seem to be feasible tasks, as the clinical question and the test request should be two sequential, strictly logically related, steps. In accordance with these assumptions, the assessment of inappropriateness of laboratory tests has been focused mainly on ordering patterns.

Test ordering may be inappropriate in two opposite ways, implicating over-ordering and under-ordering, respectively. The former refers to tests not appropriate but actually requested, whereas the latter refers to tests not requested even when appropriate. The inappropriateness (either over- or under-ordering) of test requests may be examined in relation to several factors: (I) the clinical question, including the type of disease and the clinical stage; (II) age; (III) gender; (IV) the setting, considering for instance outpatient, inpatient or emergency; and (V) the scenario, considering initial testing on patients at their first referral or repeated testing of patients followed-up or periodically monitored subjects with increased risk of a given disease.

The consequences of inappropriate laboratory test ordering

Both over-ordering and under-ordering have deleterious consequences. Not ordering the appropriate tests can be considered as malpractice, since it is associated with the concrete risk of missing or delaying a diagnosis, with probable harm to the patient. Notably, the effects of under-ordering may be crucial independently of the rate of inappropriateness; missing a diagnosis may be vital also if it occurs in a few or even in one patient. Ordering a not indicated test has several negative effects as well, some of which are a consequence of the ordering itself, some others occur only when the test has a positive result. The former includes the waste of resources, inefficiency of the health services overloaded by not necessary diagnostic tests and the loss of business hours of patients. The latter encompasses false positive results and over-diagnosis, with possible health harms due to anxiety and side effects of avoidable invasive investigations or procedures. False positive results may occur in a certain percentage of healthy subjects, since positive/negative threshold levels are calculated as a given point of distribution of test results (usually from the 90th to the 95th percentile) in a cohort of control subjects (ostensibly healthy people). In other words, in a predictable percentage of cases, healthy people may be erroneously classified as possibly affected by the disease. Conversely, in the case of over-diagnosis, despite the result of the test is correctly identified as positive because the patient is affected by the disease, it pertains to asymptomatic subjects which are diagnosed with a disease that will never cause symptoms, nor eventually reduce their life expectancy. In both instances—false positive and over-diagnosis—the positive result of the test classifies subjects in good health as diseased patients. The effects of overordering are increasingly negative as the rate of overordering increases. For instance, cost of laboratory tests accounts for a minimal portion of total health care expenditure, being less than 2% of total spending on health care in Italy and across Europe. Nevertheless, a positive result of inappropriate requested test necessarily prompt downstream diagnostic steps to either confirm or confute the suspicious diagnosis, which amplifies costs and services overload. Noticeably, the relevance of these consequences will increase dramatically when the rate of overordered tests is elevated. Not surprisingly, research on over-diagnosis is receiving increasing interest within the Scientific Community (8-10) and in the United States research has been recognized as part of the scientific direction of the National Cancer Institute (NCI) (11).

The consequences of inappropriate test ordering are dissimilar in inpatients and outpatients. Inpatients are theoretically ill subjects admitted to the hospital with symptoms or suspicious clinical signs. Therefore, the result of a given laboratory test is considered in the frame of both a complete clinical evaluation and several additional diagnostic data. In the case of tumor markers, the meaning of a false positive result is considerably mitigated and most probably will not induce any clinical decision alone. Conversely, outpatient is mainly represented by healthy people and frequently laboratory tests are performed alone for check-up or screening. In the case of tumor markers, a false positive result necessarily induces further, not previously planned, clinical or instrumental investigations. Due to the low prevalence of cancer in a non-selected general population, a high rate of false positive results of tumor marker assessment is hence predictable (12). To be noted, only a minor percentage of tumor markers is requested in inpatients, ranging from <1% of total PSA requests to 10% of total CEA requests (M. Gion, unpublished data). In conclusion, from the above reasons, the control of inappropriateness of tumour markers seems a crucial issue mainly in outpatients.

How to measure inappropriate ordering of tumor markers

The appropriateness of a health care intervention is assessed with reference to the clinical question that prompted the intervention itself, by weighing the medical decision against evidence based indications of clinical practice guidelines (CPGs). In the case of laboratory tests, appropriateness of requests should be appraised by matching the requested test with the underlying clinical issue. This approach is conceptually ideal, but is unfortunately barely usable as clinical information supplied with the test requisition form is usually poor and generic, when not lacking at all.

Monitoring appropriateness of laboratory requests presents different features in outpatient and in inpatient. First, information on the clinical question for inpatients, although not regularly provided, may be conveniently obtained contacting in real time the ward ordering the test, or retrospectively by consulting the hospital informatics system, as clinical information and outcome data of any admission episode are regularly filed. Conversely, information available on the test requisition form are generic and/or incomplete in the majority of cases in outpatients. Contacting the physician requesting the test may be troublesome and time consuming. Moreover, electronic records with clinical information of outpatients are neither available nor accessible by the laboratory. Second, the laboratory may autonomously modify the type or the number of test requested by the ward physician, when seemingly inappropriate. Unlike this case, the test requisition form for outpatients has official (legal) value, and so cannot be modified by the laboratory receiving the request. Thus, while monitoring inappropriateness through individual clinical information may be a realistic goal in inpatients, it seems more difficult to perform in outpatients.

Revision of published studies on inappropriateness of laboratory test ordering should identify proper indicators to be used on a regular basis. However, only three systematic review have been published on this issue so far (5,13,14). One review focused on ongoing evolution of laboratory test audits, examining methodology, study design and role of different professionals involved, but did not report rates of inappropriate texting (14). Two other review examined appropriateness of testing reporting mixed data, with inappropriateness rates scattered in a wide range (5,13). Overall, only 86 papers met the methodological criteria of a study on appropriateness required in two systematic reviews over the past 46 years, thus proving that appraisal of appropriateness on the basis of the clinical question remains challenging. In the more recent review, Zhi et al. selected and examined 40 studies; 18 were focused on inpatient settings only, while 22 considered both in- and outpatients: of these, 6 studies evaluated inappropriate repetitions of specific tests and 16 examined appropriateness of initial requests. These latter studies were principally focused on specific clinical scenarios, i.e., drug monitoring (3 studies), plausibility of multiple testing in infectious diseases (4 studies) and very detailed clinical questions (4 studies: 1 on thyroid screening in Down syndrome, 1 on liver function monitoring in patients on statin therapy and 2 on venous thromboembolism). Two studies were focused on general panels of tests and only 3 concerned tests that may be requested in the general practice: 1 was on anti-neutrophil cytoplasmic antibodies, the second examined 2,425 requests of prostate specific antigen (PSA) and the third studied a panel of 5 tumor markers in 373 patients (5). From the settings of the studies selected by Zhi et al. (5) it appears that the assessment of inappropriateness in outpatients is much less common than in inpatients and seems feasible mainly for some specific—and narrow—diagnostic areas. In conclusion, proper indicators for monitoring inappropriate tumor marker ordering are still demanding.

Developing indicators to monitor appropriateness of tumor marker requests

Lack of reliable clinical information on the test requisition form of tumor markers hampers the assessment of appropriateness comparing the requested test with the clinical question. Accurate recording of clinical data provided to the laboratory is not expected to improve in the next future due to the increasing automation of laboratory workflow and the diffusion of the “hub-and-spoke” organization paradigm, with many peripheral phlebotomy centres referring samples to a central laboratory. Indirect approaches to appraise appropriateness could circumvent the need for thorough information with any individual requests.

In fact, though accurate clinical data reported in individual order forms of outpatients is lacking, relevant information can be obtained by mining structured databases regularly filled in for administrative or epidemiologic purposes. Our group developed an innovative model to estimate appropriateness based on the comparison between the actually ordered and the expected requests of tumor marker (15). Laboratories of public hospitals of two Italian regions were surveyed and 1,891,070 requests registered over a year in laboratory informatics systems of 66 laboratories were extracted and examined. Epidemiologic figures of malignancies were obtained from Cancer Registers (16). Requests of CA15.3, CA19.9 and CA125 were compared with prevalence or incidence figures of the diseases in which the markers can be considered. The model identified much higher requesting rates than expected of both CA19.9 (expected vs. observed difference, +1,842%) and CA125 (expected vs. observed difference, +130%). The model was effective in demonstrating overordering of tumor markers, possibly associated with inappropriate use (15).

On the basis of these findings, our research group carried out a confirmatory study testing and validating the model on a national scale (17). The number of tumor markers ordered in Italy over 2 years (2011 and 2012) was obtained from the Ministry of Health, whereas cancer prevalence was obtained from the Italian Association of Cancer Registries (16). The number of requested tumor markers was matched with those expected, calculated on the basis recommendations of CPGs applied to figures of cancer prevalence. Tumor markers ordered in Italy were 13,207,289 in 2012 (221.3/1,000 individuals). Given an estimated prevalence of 2,243,953 cancer cases, 7.04 tumor markers appear to be requested for every prevalent case. Meaningful region-to-region variations were also identified, which do not correspond to any known variation of cancer prevalence. The model, validated at a national scale, showed that tumor markers are overused in Italy, and that their ordering pattern is not related to cancer prevalence, thus providing a proxy indicator of inappropriateness (17).

The model was developed using data on cancer prevalence assuming that (I) prevalent cases were represented by patients with the most probable clinical status; and (II) the limit of acceptable appropriateness was the number of tumor markers recommended by general practitioners (CPGs) in the most probable clinical status. Using these assumptions, overordering of CA19.9, CA125 and carcinoembryonic antigen (CEA) was shown, whereas ordered and expected CA15.3 were quite similar (17). In the case of breast cancer, the assumptions were further focused in accordance with current CPGs that recommend against CA15.3 determination in the follow-up of asymptomatic patients, whereas they consider using the marker to monitor the response to treatment of metastatic disease (18). Prevalence rate includes all cases with a given malignancy, but does not provide information on their disease status. Therefore, the number of prevalent cases, with or without advanced disease, may only be indirectly inferred on the basis of data regarding mortality and survival rate. The number of expected CA15.3 requests was estimated on the basis of data on mortality rate and survival rate of women with metastatic breast cancer under anticancer treatment. According to a published algorithm (19), approximately 36,000 prevalent cases are expected to be affected by metastatic breast cancer disease in Italy. Assuming that every case with metastatic disease is monitored monthly with CA15.3 for therapy response, the expected number of CA15.3 determinations would be 432,000. Therefore, the reported number of CA15.3 requested per year (1,078,864) (17) resulted in fact considerably higher than the expected one with an over-prescription rate of 149.7% (19). The above findings show that the proposed model is adequately flexible to be adapted to diverse epidemiological data and also suggest that it can be suitable to develop indicators for appropriateness in the use of tumor markers in different clinical conditions or organizational setting.

Population-based assessment has been suggested as an ideal approach for measurement of appropriateness in order to generalize the evaluation of test ordering within a given geographic area. However, the developed epidemiological-based model does not provide a direct measure of appropriateness, since it reveals areas of overutilization probably related to inappropriate use, in which it is necessary to confirm the inappropriateness through a deeper analysis. Our group developed a complementary methodological approach to deeply examine the inappropriateness of tumour markers ordering in outpatients (20). The method was based on the formulation of performance indicators which were tested by mining cohort-wide structured information in the electronic health records (EHRs) of the Italian National Health Service, which include demographic data as well as disease-associated codes registered to manage costs and reimbursement. The benchmark for defining the frame of appropriate ordering was grounded on recommendations of CPGs (19,21,22). The study considered currently used tumour markers requested to all outpatients referred to the Local Health Authority Aulss3 of Veneto Region over one year. In total, 80,813 tumour marker tests requested for 52,536 patients were examined. Indicators were formulated in relation to age, gender, disease type and number of repetitions of the test in any individual subject. CA15.3 and CEA were found to be prevalently requested in patients with cancer, whereas the other tumour markers [alpha-fetoprotein (AFP), CA125, CA19.9] were largely requested also in patients without cancer. Multiple repetitions of AFP, CA125, CA15.3, CA19.9 and CEA were found to be prevalent in patients with cancer or benign diseases, in which the request of tumour markers may be appropriate, while PSA repetitions occur mainly in patients without cancer. The study by Gion et al. shows that the mining of EHRs is an effective strategy to assess appropriateness of tumour marker ordering and to optimize performance indicators (20).

It appears that inappropriateness is multifaceted, even within a specific category of laboratory tests such as tumour markers. For instance, CA15.3 and CEA, although overordered (as shown by the analysis of epidemiological data) (17), are mainly requested in patients with cancer (20), thus suggesting that interventions to enhance appropriateness should be addressed to oncologists. On the other side, overordering of AFP, CA125, CA19.9, and mainly of PSA, occurs in patients without cancer. In this latter case actions to improve appropriateness should be focused on general practice. Interestingly, in 69.3% of patients labelled with disease codes for “cirrhosis of liver and biliary” or “chronic hepatitis (active)” AFP was not requested (20), although being recommended by some CPGs (21). These findings show that both overordering and underordering may be investigated by exploring EHRs which integrate data captured for administrative purposes and structured information on every health care intervention.

Concluding remarks

In the last two decades evidence based medicine has acquired increasing popularity. Awareness of the need to base clinical decisions on CPGs has also significantly increased among physicians. Hauser et al. reported that organizational guidelines have become the predominant source of audit criteria over the last two decades, accounting for over 75% of audits reports in recent years (14). In contrast, Zhi et al. reported that inappropriateness of laboratory tests ordering did not show significant variations over a similar time period (5). Published data on tumour markers confirm that no significant improvement of test ordering appropriateness occurred over the last years (15,17,19,20,23-27). These findings support the hypothesis that recommendation of CPGs on tumour markers are largely disregarded and the process of knowledge translation presents on the whole significant margins for improvement.

The roadmap for a continuous improvement of appropriateness should consider to move from implementation of guidelines to integration of guidelines into the more formal context of knowledge translation (28). In a broad connotation, knowledge translation describes the process of putting knowledge into action (29) and has been described by a variety of terms, which contributed to reinforce the idea that knowledge translation is an unstructured and generic pathway, thus hampering the benefit it could actually provide (30). According to the Canadian Institutes of Health Research, knowledge translation is more accurately defined as ‘‘a dynamic and iterative process that includes the synthesis, dissemination, exchange and ethically sound application of knowledge to improve health, provide more effective health services and products and strengthen the healthcare system’’ (31). In agreement with this definition, knowledge translation is a formal approach that combines and manages several phases, including identification of problems and evaluation of their priority, assessment of barriers, implementation of evidence, monitor of evidence use and evaluation of outcomes. Notably, the formal definition of knowledge translation fits with that of “best practice” in health care, that has been defined as “the best way to identify, collect, evaluate, disseminate, and implement information about as well as to monitor the outcomes of health care interventions for patient population groups and defined indications or conditions. Information is required on the best available evidence on safety, efficacy, effectiveness, cost-effectiveness, appropriateness, social and ethical values and quality of the health care interventions” (32). These strong similarities confirm, from different points of view, the need for a weighted use of all valid and relevant information coupled with a careful and regular monitoring of outcomes of health care interventions (33). Implementation of evidence based information and monitoring their impact on the clinical practice are parts of the same, multistage, process aimed at the progressive improvement of health care.

Acknowledgements

We thank Dr. Antonette Leon for her contribution in editing the manuscript and Mrs Ornella Scattolin for administrative assistance.

Funding: This study was supported by Veneto Region (IT) through the ‘Programma Regionale per i Biomarcatori Diagnostici, Prognostici e Predittivi’ assigned to Azienda ULSS3 Serenissima—formerly Azienda ULSS12 Veneziana; partially supported by the Italian Association for Research on Cancer, Grant Special Program Molecular Clinical Oncology, 5×1000 (No. 12214) to M Gion; and also partially supported by Istituto Oncologico Veneto IOV-IRCCS, Padova, Italy and AVAPO Venezia Onlus, Venice, Italy.

Footnote

Conflicts of Interest: The authors have no conflicts of interest to declare.

References

1. Buetow SA, Sibbald B, Cantrill JA, et al. Appropriateness in health care: application to prescribing. Soc Sci Med 1997;45:261-71. [Crossref] [PubMed]
2. Alexander B. Reducing healthcare costs through appropriate test utilization. Critical Values 2012;5:6-8. [Crossref]
3. Howerton D, Anderson N, Bosse D, et al. Good laboratory practices for waived testing sites: survey findings from testing sites holding a certificate of waiver under the clinical laboratory improvement amendments of 1988 and recommendations for promoting quality testing. MMWR Recomm Rep 2005;54:1-25. [PubMed]
4. Liang BA, Mackey T. Direct-to-consumer advertising with interactive internet media: global regulation and public health issues. JAMA 2011;305:824-5. [Crossref] [PubMed]
5. Zhi M, Ding EL, Theisen-Toupal J, et al. The landscape of inappropriate laboratory testing: a 15-year metaanalysis. PLoS One 2013;8:e78962. [Crossref] [PubMed]
6. Franceschini R, Trevisiol C, Dittadi R, et al. Tumour markers requesting pattern with regards to different organizational settings in Italy: a survey of hospital laboratories. Ann Clin Biochem 2009;46:316-21. [Crossref] [PubMed]
7. Smellie WS. Demand management and test request rationalization. Ann Clin Biochem 2012;49:323-36. [Crossref] [PubMed]
8. Moynihan R, Doust J, Henry D. Preventing overdiagnosis: how to stop harming the healthy. BMJ 2012;344:e3502. [Crossref] [PubMed]
9. Carter JL, Coletti RJ, Harris RP. Quantifying and monitoring overdiagnosis in cancer screening: a systematic review of methods. BMJ 2015;350:g7773. [Crossref] [PubMed]
10. Chiolero A, Paccaud F, Aujesky D, et al. How to prevent overdiagnosis. Swiss Med Wkly 2015;145:w14060. [PubMed]
11. Kramer B. Current and future perspectives on cancer prevention research. National Cancer Advisory Board, 2012. Available online: http://deainfo.nci.nih.gov/advisory/ncab/161_0212/Kramer.pdf. Last accessed 6 May 2017.
12. Jackson BR. The dangers of false-positive and false-negative test results: false-positive results as a function of pretest probability. Clin Lab Med 2008;28:305-19. [Crossref] [PubMed]
13. van Walraven C, Naylor CD. Toward optimal laboratory use. Do we know what inappropriate laboratory utilization is? A systematic review of laboratory clinical audits. JAMA 1998;280:550-8. [Crossref] [PubMed]
14. Hauser RG, Shirts BH. Do we now know what inappropriate laboratory utilization is? An expanded systematic review of laboratory clinical audits. Am J Clin Pathol 2014;141:774-83. [Crossref] [PubMed]
15. Gion M, Franceschini R, Rosin C, et al. An epidemiology-based model to estimate the rate of inappropriateness of tumor marker requests. Clin Chem Lab Med 2014;52:889-97. [Crossref] [PubMed]
16. AIRTUM working group. Italian cancer figures, report 2010: cancer prevalence in Italy; Patients living with cancer, long-term survivors and cured patients. Epidemiol Prev 2010;34:1-188.
17. Gion M, Peloso L, Trevisiol C, et al. An epidemiology-based model as a tool to monitor the outbreak of inappropriateness in tumor marker requests: a national scale study. Clin Chem Lab Med 2016;54:473-82. [Crossref] [PubMed]
18. Gion M, Trevisiol C, Rutjes AW, et al. Circulating tumor markers: a guide to their appropriate clinical use - Comparative summary of recommendations from clinical practice guidelines (PART 2). Int J Biol Markers 2017;32:e1-52. [PubMed]
19. Trevisiol C, Gion M, Dittadi R, et al. Epidemiology-based assessment of tumor marker overordering in breast cancer: an algorithm to examine different disease conditions. Int J Biol Markers 2017. [Epub ahead of print]. [Crossref] [PubMed]
20. Gion M, Cardinali G, Zappa M, et al. Indicators of inappropriate tumour marker use through the mining of electronic health records. J Eval Clin Pract 2017. [Epub ahead of print]. [Crossref] [PubMed]
21. Gion M, Trevisiol C, Rutjes AW, et al. Circulating tumor markers: a guide to their appropriate clinical use - Comparative summary of recommendations from clinical practice guidelines (PART 1). Int J Biol Markers 2016;31:e332-67. [Crossref] [PubMed]
22. Gion M, Trevisiol C, Rutjes AW, et al. Circulating tumor markers: a guide to their appropriate clinical use - Comparative summary of recommendations from clinical practice guidelines (PART 3). Int J Biol Markers 2017;32:e147-81. [PubMed]
23. Gion M, Rampazzo A, Mione R, et al. Cost/effectiveness ratio of carcinoembryonic antigen – importanceof adequacy of routine requests of tumour markers. Int J Biol Markers 1992;7:179-82. [PubMed]
24. Gion M, Barioli P, Ponti A, et al. How tumor markers are used in the routine follow-up of breast and colorectal cancer. A survey of 29 Italian hospitals. Int J Biol Markers 1998;13:124-38. [PubMed]
25. Arioli D, Pipino M, Boldrini E, et al. Tumour markers in internal medicine: a low-cost test or an unnecessary expense? A retrospective study based on appropriateness. Intern Emerg Med 2007;2:88-94. [Crossref] [PubMed]
26. McDonnell M. Northern Ireland Regional Audit Group. An audit of tumour marker requests in Northern Ireland. Ann Clin Biochem 2004;41:378-84. [Crossref] [PubMed]
27. Walker PL, Crook M. Tumour marker requesting in primary care and the role of the laboratory. J Clin Pathol 2011;64:443-6. [Crossref] [PubMed]
28. Straus SE, Tetroe JM, Graham ID. Knowledge translation is the use of knowledge in health care decision making. J Clin Epidemiol 2011;64:6-10. [Crossref] [PubMed]
29. Canadian Institutes of Health Research. Available online: http://www.cihr-irsc.gc.ca/e/29418.html. Last accessed, 6 May 2017.
30. Graham ID, Logan J, Harrison MB, et al. Lost in knowledge translation: time for a map? J Contin Educ Health Prof 2006;26:13-24. [Crossref] [PubMed]
31. McKibbon KA, Lokker C, Keepanasseril A, et al. WhatisKT wiki: a case study of a platform for knowledge translation terms and definitions--descriptive analysis. Implement Sci 2013;8:13. [Crossref] [PubMed]
32. Perleth M, Jakubowski E, Busse R. What is 'best practice' in health care? State of the art and perspectives in improving the effectiveness and efficiency of the European health care systems. Health Policy 2001;56:235-50. [Crossref] [PubMed]
33. Lippi G, Bovo C, Ciaccio M. Inappropriateness in laboratory medicine: an elephant in the room? Ann Transl Med 2017;5:82. [Crossref] [PubMed]