Prof. David Moher: Guidelines for Reporting Health Research (A User’s Manual)
Editor’s note
In March 2021, AME Publishing Company translated the book “Guidelines for Reporting Health Research: A User’s Manual” into Chinese and completed the work in June 2021. While the Chinese edition is now beginning official publication, the AME editorial office launches alongside its publication interviews with the book editors and authors, hoping to highlight some updates on the status and trends of the reporting guidelines in the Chinese edition.
We take a pleasure in interviewing Dr. David Moher; here he shares his insights based on the book. Dr. David Moher is one of the Editors-in-Chief and seven chapters author of the book.
AME: Birth of the book. Could you share the story of this book, particularly the reason why you decide to take on the role as Editor-in-Chief of the book from the beginning and what your main work included? You may also share about how the book was started, how long it took for the publication, what the ultimate goal and expectation of publishing were, and if you have ever encountered any difference of opinions along the way.
Prof. David Moher: Why did we start the book is we aware that there were many reporting guidelines but there was no one place to try to pull them all together. We also knew that in the literature, there was not a lot of information about transparent reporting of research in 2000. We started this around 2012, when there was not a lot of information about the EQUATOR Network and we felt that a book would be quite useful. Therefore, it was a way for us to disseminate information about reporting guidelines.
AME: The organization of the chapters. How did the editorial board lay out the chapters of the book? Which is your most satisfactory or highly recommended chapter if any? Have you felt regretful for any reporting guidelines that were not included in this book due to its length? For example, how the PART I, PART II, and PART III were organized and how each representative reporting guideline were selected from the numerous reporting guidelines. Why would you include these reporting guidelines instead of others (e.g., the guidelines recommended by the EQUATOR Network, such as CARE, AGREEII, RIGHT, ARRIVE, CHEERS, etc. were not included)?
Prof. David Moher: The book is categorized into four parts. The first part of the book is some general issues about reporting. The second part of the book, which is the largest part of the book, is around specific types of reporting guidelines and there was one guidance for protocols of randomized trials. There’s one chapter about abstracts of trials, the other one about CONSORT and various CONSORT extensions, and also other chapters about different reporting guidelines. Chapter section three of the book is about analytical issues on reporting of research such as statistical analysis, presentation of tables and figures. And the last part of the book is a chapter about policy guidance for journals. That’s how the book is organized.
I don’t have a most highly recommended chapter. I think that all of the chapters provide useful information to researchers and authors.
There are many reporting guidelines not included in this book. As mentioned above, the book was started in 2012. For example, there are now a reporting guidance on multi-armed trials and a guidance on network meta-analysis. There’s lots of guidance that is not in the book because the book is several years old now. If we were to update the book, we would no doubt include other guidance and also the updated version of the currently included guidance, e.g., chapter 24 (PRISMA). Recently PRISMA was significantly updated and so this chapter was probably out of date and PRISMA 2020 is now the appropriate guidance for systematic reviews. I also note another very excellent guideline is PRISMA for scoping reviews. There’s also a PRISMA as for reporting search strategies. It’s simply because there are newer guidance and the book is older.
I think the chapters included particularly in part II and part III of the book were selected to cover as broad a spectrum as possible. For example, we wanted to make sure that there was information on reporting of diagnostic accuracy studies, on reporting guidance for clinical trial for systematic reviews for observational studies and qualitative studies. So, we wanted to select chapters that had a broad spectrum of interest across research. Not everybody does randomized trials. Some people will do observational studies but not everyone will do the same. For some people, their expertise is doing diagnostic accuracy like this. We want to make sure we have the spectrum in here.
AME: This book mentions the EQUARTOR Network many times, and we knew that you were also involved in the work of the EQUARTOR Network. What is your role in the EQUARTOR Network?
Prof. David Moher: My role presently is that I am a chair of the EQUATOR Network and EQUATOR Network Executive meets four times a year. Obviously, during the pandemic we meet virtually and we try to ensure that the respective EQUATOR centers are fulfilling their functions of providing educational input. I should also mention that in the past year or so, it’s been wonderful to have the development and the establishment of the Chinese EQUATOR center. My role is to connect the group of EQUATOR centers (Australia, Canada, China, France and the UK).
AME: What is the most impressive advancement of the reporting guidelines around the world in your mind? What is the biggest problem that needs to be improved?
Prof. David Moher: I think that there are many impressive advancements. For example, there are some very good examples of reporting guidelines that use artificial intelligence as the intervention, and the update of the PRISMA 2020 has been a very encouraging guidance. I think the UK EQUATOR center is doing a very important study of examining the scientific quality of reporting guidelines in the EQUATOR library. I think all these advancements are extremely important and I cannot single out a most impressive one.
AME: This book has been translated in Chinese by AME Publishing Company and the Chinese edition will be published soon. What do you want to say to the Chinese readers?
Prof. David Moher: I would first like to congratulate on publishing the book in Chinese. The Chinese researchers are an extremely important group in the research ecosystem of the world. Second, I would like Chinese authors and authors around the world to work very hard to make sure that their research is completely and transparently reported and it should be in such detail that a reader could reproduce their methods and results if they wanted. There is a lot of evidence that researchers do not pay enough attention to reporting their research.
Chapter 1 Importance of Transparent Reporting of Health Research
AME: As far as you know, what is the status quo and trend of transparent reporting guidelines in recent years?
Prof. David Moher: I think that the evidence is that authors are still not reporting their research transparently so it’s difficult to undertake evidence-based research and transfer it into evidence-based policy if we don’t have clear and transparent reporting. It’s also clear that if research is not transparently reported, it’ would be very difficult to reproduce it. We need to be much better at being able to reproduce research and that’s extremely important. I also would like to say that it’s not solely the responsibility of the author, but also the responsibility of the journal so that if an author submits a manuscript to a journal that is not well reported, the journal editor and the peer reviewers have a responsibility to make sure that it is well reported. In the declaration of Helsinki of 2013, it very clearly indicates that editors have a responsibility to ensure that what they publish is of the highest possible standards but we have quite a lot of evidence that’s not working out and there is still a lot of research not well reported.
AME: Transparent reporting guidelines are very important and require the collaboration of multiple parties, including journals, editorial boards, research funding agencies, academic institutions, authors, etc. Which area do you think needs to be strengthened?
Prof. David Moher: I think all of them need to be strengthened but academic institutions have a major responsibility in the sense they’re training the future generations of researchers. If I was to be invited to generate policy to improve the transparency of reporting, I would make it a mandatory that researchers in training have to take a course on reporting their research. I also think that reporting research could also be used in promotion and tenure assessment.
AME: The author has a lot of workloads to fill out the checklist. At the end of this chapter, you mentioned that you can use machine-readable language software to help you fill out checklist automatically. To our knowledge, it’s still in the experimental stage. How is the progress in this area and its current development in various journals?
Prof. David Moher: I think it’s very slow because many journals have not integrated automated procedures to do that. I think in the next few years hopefully that will improve as we see much greater sophisticated artificial intelligence being used. There are examples of journals using software. In many journals, they use software to check for fabrication or they check for just a software called authenticate and that will allow the journal to see whether any of the text language has been taken from somewhere else or used previously. So, I think it’s happening. It’s moving quite slow but I hope in the next few years that we will be able to automate this better.
Chapter 2 How to Develop a Reporting Guideline
AME: It is fabulous to see the chapter introducing how to make a reporting guidelines step by step. Could you please share with us how the formulation method (18 steps) was established?
Prof. David Moher: It was established based on our experience with developing reporting guidelines. It was developed many years ago and the paper was published in 2010. What I can share with you now is that work is being updated (led by Dr. Matthew Page) and I hope in the next year we will be able to publish an update of how to develop reporting guideline. The EQUATOR Network library consists of more than 400 reporting guidelines but several of them are not developed optimally. I think we need to work much harder at developing reporting guidelines appropriately and I hope our update of chapter 2 will help researchers develop reporting guidelines appropriately.
AME: We (AME Publishing Company) have launched the SUPER (Surgical technique reporting checklist and standards) project (https://www.thesuper.org) in August 2020, aiming to improve the reporting quality of surgical technical articles. The project involves collaborators (specialists) from multiple disciplines and follows the formulating method as mentioned in this chapter. As we have now entered the three round of Delphi survey, the degree of experts’ engagement is various (for example, some experts’ suggestions are very detailed, while some reveal that they may not read each item carefully). Do you have any advice on how to motivate the experts to participate in the Delphi survey more actively and earnestly?
Prof. David Moher: What we often use in our part of the world to motivate people to complete Delphi is incentive. For example, what we do for everybody who completes the Delphi, we enter their name into a raffle where they have the opportunity of getting a hundred-dollar Amazon gift certificate. If you completed, you are entered into a contest to win something from Alibaba so that’s how we do it in our part of the world. We use incentives what we call as a fiscal incentive.
AME: We totally agree that authors should consider publishing the reporting guidelines in multiple journals mentioned in this chapter. Do you know any cases where journals do not accept simultaneous publication of a reporting guideline in multiple journals? What would be your advice to deal with the situation should it happen?
Prof. David Moher: Some journals say that they don’t want to publish this reporting guideline but then we try to get it published in another journal. Maybe you have to try multiple journals. My first advice is that to develop a reporting guideline, you need a strong scientific rationale. Developing a reporting guideline for where there may be only 10 such studies reported in the world is not worth the investment because there’s very little incentive to publish that. I often receive emails from people saying that they’d like to develop a reporting guideline for people who have earache, but earache maybe a very small problem and they may be very few publications so it’s not worth it. But these people don’t listen and they just do it. The second issue is you need a strong rationale for why you’re developing a reporting guideline. And the second piece of advice would be to follow the guidance that we developed on how to develop a reporting guideline in that we clearly state what you need a strong rationale for and you can’t just develop a reporting guideline which people don’t think is good because they’re not going to use it.
AME: Do you have any other formulating methods or considerations recommended for creating reporting guidelines?
Prof. David Moher: Follow what we’ve published in that chapter and that’s the strongest advice I can give.
Chapter 3 Characteristics of Available Reporting Guidelines
AME: Nearly one-third of reporting guidelines did not report the consensus process. What do you think is the main reason for this result?
Prof. David Moher: Because they’re not developing guidelines according to what we outlined in chapter 2. I think what we find in biomedical research is that sometimes we have big egos. Researchers sometimes say ‘I know how to do this and I’m just going to do it’ so they make mistakes and they think their ways are fine.
AME: Only 13.6% have designed explanatory documents. We know that the preparation of explanatory documents is a very time-consuming task. Do you have any suggestions for production of more designed explanatory documents in the future?
Prof. David Moher: In terms of explanatory documents, for people who are coming to the consensus meeting (there are often 20 to 30 people), my suggestion would be to ask each of those 20 to 30 people to take one of the items on the checklist and to draft an explanation with some evidence. That’s much more sharing of the work and it’s much easier to do. We have just finished a reporting guideline for overviews (systematic reviews of systematic reviews) and the way we did that was after the consensus meeting. Each member at the consensus meeting took one or two items from the checklist and wrote the draft for that and provide evidence and that’s very helpful. It really shared a lot and is a very good way of doing the explanatory paper.
There are many reporting guidelines for types of research like for writing a systematic review, you could use PRISMA 2020. I think that’s very important but it’s very late in the day that the systematic review is done and all you can get somebody to do is to report it as clearly and transparently as possible. The most impact of reporting guidelines can be seen at the protocol stage. For example, there is PRISMA-P (PRISMA for protocol), which asks researchers who are thinking about doing the systematic review to think about all of these issues. This kind of guideline is going to be much more impactful because it can influence how the review is done. However, there are more than 400 reporting guidelines and less than 5 of them have guidance for protocols. Therefore, I would say stop doing reporting guidelines for completed research and start doing reporting guidelines for protocols, where you can have your biggest impact. For PRISMA “P”, we developed not only a checklist but also an explanatory document and that’s what I think should be done for a protocol.
AME: Have these data been updated recently? What is the trend?
Prof. David Moher: I don’t know what the trend is and I don’t have any plans to update it because to do this sort of research, it requires funding and it’s very difficult to get the funding. If your publishing company was interested in funding this, we would be very happy to do it but I don’t have any plans to update it so far.
Chapter 5 Ambiguities and Confusions Between Reporting and Conduct
AME: What is the current confusion between “the quality of research reporting” and “research quality”? Has the situation been improved?
Prof. David Moher: I think there is still a lot of confusion between conducting and reporting, and it requires an educational training opportunity for people to identify the difference between conduct and reporting. What we really need to encourage people is that whatever they’ve done, they need to report it clearly, transparently and honestly. We have a lot of research indicating that the way people do research is not the way they report it. They are often with big gaps. What we are trying to develop, in collaboration with the Chinese EQUATOR Center, a Massive Open Online Course (MOOC) where we are going to run 20 sessions on reporting of research and for people who successfully complete that, they will get a certificate from the Chinese EQUATOR Center and the Canadian EQUATOR Center.
AME: “The quality of research reporting” and “research quality” are just like the egg and chicken issue, suggesting the possible causal relationship behind each other. At present, which one do you think is more important (or egg first or chicken first)?
Prof. David Moher: Firstly, we need to work much harder on reporting guidance for protocols, Secondly, researchers need to that everything they’ve done is perfect and this is not true. I’ve been doing research for 30 years and I can assure you that every time we do research, it is not perfect and there are mistakes happening and people just need to feel comfortable about reporting. Perfection is not the goal while transparency is the goal.
Chapter 7 SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials)
AME: SPIRIT, as a reporting guideline for an interventional trial plan, entails many elements for consideration. From 59 items to 33 items, what are the main considerations in the modification?
Prof. David Moher: CONSORT is a guidance for reporting completed trials. SPIRIT is a reporting guideline for clinical trial protocols, not an extension for CONSORT.
AME: [Evidence of the effectiveness of the guidelines] pointed out that SPIRIT compliance assessment is currently lacking. Are there any updates in this aspect?
Prof. David Moher: The compliance with reporting guidelines is generally not good and that is because there are no incentives to use reporting guidelines. That’s a problem across almost all reporting guidelines so we need more incentives and rewards to use guidelines. For example, academic institutions and universities, they typically reward people for the number of papers they publish and they don’t reward them for the quality of papers they publish. Actually, universities could say we would like you to use reporting guidelines but they don’t.
AME: What is the progress in terms of the methodological aspects of the evaluation of compliance with the reporting guidelines? Does the evaluation method also need to be regulated?
Prof. David Moher: The evaluation methods for reporting guidelines are similar to the evaluation methods of any intervention. If you think of a reporting guideline like a drug, the best way to evaluate whether a drug is effective is when you do a randomized trial. The best way to evaluate whether an intervention like a reporting guideline works is to do a randomized trial and several have been done of reporting guidelines showing that the use of a reporting guideline improves the transparency and completeness of reporting. There are not enough randomized trials on reporting guidelines so there needs to be more, but how you evaluate a reporting guideline is just like you evaluate a drug. It’s actually the same way you would evaluate traditional Chinese medicine (TCM) via a randomized trial.
AME: Can you share the status of SPIRIT, such as update plans, future arrangements, etc.?
Prof. David Moher: We are currently updating SPIRIT and CONSORT. The SPIRIT Executive and the CONSORT Executive have joined forces. We are in the early stages of updating both reporting guidelines. We hope to be able to complete the update in 2023.
Chapter 9 CONSORT
AME: CONSORT can be said to be one of the best implemented reporting guidelines among numerous others. In your opinion, what is the main reason for its success?
Prof. David Moher: There are several reasons for CONSORT’s success. The first reason is it was the first reporting guideline aimed at trying to improve the completeness and transparency of randomized trials. The second reason is that some very influential editors of journals were highly supportive of it and endorsed it very early on. These journals said to prospective authors that if you want to submit a report of a completed trial to our journals, you need to use the CONSORT reporting guideline. There was a lot of incentives and rewards associated with using the reporting guideline by these very influential editors.
AME: How did you select the items in CONSORT?
Prof. David Moher: The items were selected based on evidence. For example, randomization was selected because there was evidence that the randomization process needs to be completely reported. We know that incompletely reported randomization methods, compared to completely reported randomization methods, introduces bias and exaggerates the results of a randomized trial.
AME: Except for the top-recommended items mentioned in the chapter (item 8-10, item 11, item 13). What other items do you think are also of particular importance?
Prof. David Moher: I think they’re all important.
AME: Regarding registration, do you suggest that registration must be mandatory? Do you think that registration before submission and registration after submission are both acceptable?
Prof. David Moher: I think registration should be mandatory and we have evidence that there is insufficient number of trials that are registered and publication bias is actually increasing. A recent study of the 36 medical centers in Germany that conduct randomized trials shows their publication bias on average was 60%. We also have new evidence coming from Canada, indicating that on average only 45% of trials are registered, which is not good. Trials need to be registered in several parts of the world. In Canada, trial registration is mandated. It’s required but people aren’t doing it so we need to work very hard to improve that. In general, I believe registration should be mandated and is mandated in certain parts of the world.
AME: Many researchers cannot provide a specific protocol. Do you have any advice?
Prof. David Moher: I think protocols need to be mandated and everybody who is conducting a randomized trial needs to make their protocol publicly available. There are many places to do this. Many universities have a repository for which a protocol could be placed. There is clinicaltrials.gov, which is a place that you could append the protocol. There’s other places such as open science framework where you could put the protocol. There is no excuse for not making a protocol of a randomized publicly available.
AME: There are many extended versions of CONSORT. Are there any special considerations for the scope and order of their extensions?
Prof. David Moher: Yes, every extension has its unique aspects. That’s why it’s an extension. For example, there is an extension for randomized crossover trials and there are extensions for certain types of interventions, TCM and non-pharmacological interventions respectively. Each of those extensions includes some unique items. They’re all important.
AME: What is the future update and expansion plan of CONSORT?
Prof. David Moher: We are going to update both SPIRIT and CONSORT. We’re in the process of doing that. We’ve done a lot of background research already and we hope to be able to update both of them in 2022.
Chapter 24 PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses)
AME: Would it be convenient for you to share your most impressive feelings and stories during the process of developing PRISMA?
Prof. David Moher: PRISMA is a reporting guideline for systematic reviews and meta-analysis. PRISMA is an update of some work that we had done earlier called “QUORUM”. QUORUM was a reporting guideline for meta-analysis. It’s great honor for me to see it being very widely accepted as a very useful reporting guideline. We updated PRISMA in 2020 so now there is a PRISMA 2020 published recently.
AME: Currently, PRISMA has received great recognition worldwide. For example, all journals of AME explicitly require authors to report on PRISMA and submit a checklist, which will be published together with the manuscript, but not every journal do the same. To your knowledge, how is the status of recognition of PRISMA in different journals (in terms of mandatory requirements, suggestions, completion of checklists, etc.)?
Prof. David Moher: To my knowledge, reporting guidelines are often encouraged in certain journals. In some journals, they’re recommended but in some journals, the language of recommendation is quite vague. For prospective authors, they’re not entirely sure what the journal means. What would be very good for China is that they use the same language about PRISMA across journals. Different journals don’t use different language so all of the journals should come together and use the same language of endorsement and requirements. That’s my recommendation to do this in China. The uptake of PRISMA has improved over time. The number of endorsements has improved and there are some evaluations that PRISMA has improved, such as the quality of reporting of systematic reviews. Initially, journals, universities and funders should come together and use the same language because what you find is that funders say one thing, journals say something else and universities often say nothing and that’s quite confusing to authors. It’s a mixed message. One of the stories we’ve learned about implementation is that if you want to get something implemented, the different stakeholders need to use the same language and the same message, so that’s what needs to happen. I think in a country like China, it may be easier to do that than in a country like Canada.
AME: For systematic reviews, the PRISMA guidelines do not emphasize that authors need to report all items. Are there any items that you think must be reported and some items that are not required?
Prof. David Moher: I think all of the items are important. There are some new items in PRISMA 2020. I would encourage you to look at that but what I’m going to do is to share with you the PRISMA 2020 statement, explanation and elaboration document. There’s the 2020 checklist and the 2020 flow diagram. There are a number of changes to PRISMA for the 2020.
AME: The 2020 PRISMA guideline has made many updates, including a 27-item checklist, and a four-phase flow diagram. Could you please share the main reason for the updates and why the flow diagram is separated in two types respectively for “databases and registers only” and for “databases, registers and other sources”? And why it is recommended to report the number of items per source?
Prof. David Moher: Because the philosophy that we use in developing reporting guidelines and updating reporting guidelines is based on evidence. If there is new evidence to report the items (whether there should be new items or new evidence about a particular item), we will include it. The update of PRISMA was done because it was at least a decade since we produced the original PRISMA statement and we felt that there was quite a lot of evidence about actually methodology of systematic reviews and statistical approaches, leading us to update PRISMA 2020. The reason why recommending the new items is because previously, when authors were reporting systematic reviews, they often said that we identified 6,000 records and we included four studies. We were always wondering if you included four studies, what happened to the 5,900 other studies that you didn’t include and why didn’t you include them, which is an important piece of information. That’s why we’re asking authors for these details in the flow diagram in PRISMA and it’s more advanced in PRISMA 2020.
AME: In terms of item updates, which item updates do you think are the most essential or distinctive?
Prof. David Moher: All of them are essential. I don’t want to single out any item. If you look at the BMJ publication PRISMA 2020, there is a box No. 2 section, which has a whole section on the differences between the original PRISMA (2009) and PRISMA 2020.
AME: On the one hand, given the importance of systematic reviews in evidence-based science, it is necessary for us to have more strict constraints on it. On the other hand, we can see PRISMA 2020 has included stricter requirements. Are you concerned that the new PRISMA will affect its popularity and compliance?
Prof. David Moher: No, the PRISMA 2020 was published about a year ago and it’s already been cited over a thousand times. So, the uptake is very strong. For many researchers, if their article is cited a hundred times, it is fantastic. The PRISMA 2020 been cited over a thousand times within the first 6 months, so we’re very confident that the uptake is good.
Expert introduction
Prof. Moher (Figure 1) is an internationally recognized leader and pioneer in the conduct and reporting of randomized controlled trials, methodology of systematic reviews and meta-analyses, development of reporting guidelines for health research, research on research integrity and for pioneering the emerging field of journalology (publication science). He is one of the most cited and impactful researchers in the world having published more than 700 peer-reviewed papers with an h-index of 160 and more than 400,000 citations (Google Scholar); he has received $100M in peer-reviewed funding throughout his career; he has been recognized several times as one of the world’s most highly influential biomedical researchers by Thomson Reuters/Clarivate Analytics. Professor Moher is a fellow of the Royal Society of Canada and the Canadian Academy of Health Sciences.
Acknowledgments
Funding: None.
Footnote
Provenance and Peer Review: This article was commissioned by the editorial office, Annals of Translational Medicine for the series “Insights from the reporting guidelines”. The article did not undergo external peer review.
Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://atm.amegroups.com/article/view/10.21037/atm-2022-24/coif). The series “Insights from the reporting guidelines” was commissioned by the editorial office without any funding or sponsorship. KJD, GSL, KZ, YL and FY report that they are all full employees of AME Publishing Company. DM has no conflicts of interest to declare.
Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.
