Examining the impact of health research facilitated by small peer-reviewed research operating grants in a women's and children's health centre

Although health research focuses on the development of evidence, there has been surprisingly little effort to evaluate the effects of research and research funding programs. Some research program evaluations of agencies such as the NIH[1] and the Health and Health Research Council of New Zealand[2] have relied on bibliometric methods to review their granting programs. These reviews typically consist of recording publications per funded project and using this information to compare the number of papers published among comparable institutes or grants[1, 2]. Although useful, these evaluations are limited in scope, and do not address the multidimensional aspects of research impact[3].

Research frameworks have been developed by several groups[3, 4] and institutions[5] as a means for evaluating of assessing a range of impacts resulting from health services research. health research. The Research Impact Framework[3] identifies four areas of impact: Research-related impacts include traditional measures of research success (publications, conferences, patents and awards) as well impacts on the research field itself through communication, knowledge translation and research networks; policy impacts include research that informs and influences policy; service impacts include improved methods and cost-effectiveness of delivering quality health-care; societal impacts include outcomes that affect the population beyond the research world. A clear research impact framework can contribute greatly to the evaluation of research grant programs by identifying the areas of potential impact and illustrating the relationships between them.

Agencies such as The Health and Health Services Research Fund of Hong Kong[6], Public Health Research and Development Committee of Australia[7] and the Alberta Heritage Fund for Medical Research[8] have employed similar frameworks in the evaluation of granting programs to demonstrate significant policy, service and research impacts[6–8]. However, these frameworks are typically geared toward large grants with equally large outcomes and are not easily transferable to the evaluation of smaller scale research projects designed to build research capacity. To our knowledge, there is no literature that assesses the impact of very small grants or suggests a framework within which to evaluate the impact of very small grants.

The IWK Health Centre is a women and children's academic health sciences centre, affiliated with Dalhousie University, serving Maritime Canada. The IWK has a vigorous research culture that thrives across disciplines and levels of expertise. Through the IWK's Research Services Department, a variety of granting opportunities are available within the research award program. The goal of this program is to promote the development of high quality research, consistent with the mission of the health centre. These grants are designed to support research that can be completed within a proposed budget or provide seed funding for the development of larger proposals for external funding. The grants include Category A grants funding up to ($4,000.00) and Category B grants (funding up to $15,000.00). These awards undergo rigorous peer review by the IWK Scientific Review Committee. Since 2002, over $800,000 has been awarded through 165 Category A and B grants, mostly to novice and junior researchers.

The research impact literature has no known research addressing the impact of very small research grants. Nor are there any suitable frameworks in which to measure the impact of very small grants. The purpose of the present study was to describe the impact of health research facilitated by Research Operating Grants at the IWK Health Centre within the context of a relevant research impact framework. To this end, an adaptation of the Research Impact Framework[3] was used to assess the dimensions most applicable to the impact of very small grants.

The results from this research evaluation demonstrate the effectiveness of small peer reviewed operating grants such as the IWK Research Services' Operating Grants Program. Despite the relative small funding amount, impacts comparable to larger grants were clearly apparent. For example, when considering the more traditional forms of knowledge transfer such as publishing in peer reviewed journals, the number of publications reported by participants (72%) was similar to those reported from granting agencies providing significantly larger funds. For example, Kwan et al. reported that 86.5% of projects funded by the Health and Health Services Research Fund in Hong Kong yielded peer-reviewed publications while Shah and Ward reported 82% of Australian Public Health Research and development Committee grants produced peer-reviewed publications [6, 7]. Many of the participants also reported additional forms of knowledge transfer, typically consisting of presentations to research teams, trainees or clinical staff at the IWK Health Centre, other hospitals or a local university. Presentations to the community were also common. Similar to knowledge transfer, various policy and practice impacts resulted from the research conducted by the participants. These policy and practice impacts ranged in scope from local to national and included: changes to policy regarding newborn screening processes to minimize false positives for genetic testing, policy changes regarding immunization against influenza for pregnant women and implementing disciplinary tracking systems for the Department of Education. Societal impact was seen through the dissemination of research results to the public, whether through popular press, conferences or other methods.

A unique focal point to this review was the area of personal impact. Nearly all of the participants reported a positive experience and felt that this research improved their research skills and developed their program of research. A positive impact at the level of the research is a fundamental component of ensuring that novice and junior researchers continue to pursue research opportunities. Perhaps most notable of the personal impacts was the opportunity to mentor and be mentored. Mentorship is essential to building the research careers of novice investigators by providing support and expertise.

Relationships between impacts demonstrate the interconnectedness of the areas of potential impact. The associations found between practice impacts and presentations, demonstrate a link between knowledge transfer and effects on practice. Similarly, the association between research experience and conference presentations indicates that as investigators become more experienced, they engage in more knowledge transfer activities.

The present study however, is not without limitations. The participation rate, although reasonable, may have contributed to bias. Investigators who had projects with little impact may have decided not to participate. The small sample size limited the use of inferential statistics that might provide more insight into the relationships between variables. Finally, the exclusive reliance on self report by the participants on the impact of their research may have biased the results.

The foundation of excellence in health service delivery is the evidence that informs clinical practice and policy change. It is essential that academic health centres engage actively in ensuring that a culture of research inquiry is maintained and that support is available to those researchers that may ultimately contribute to the excellence in health care. Typically, the purpose of small institutional grants is to provide the seed funding that will support new researchers in the development of their programs of research. The findings from this study indicate that the grants awarded through the IWK Health Centre provide researchers with the support they need to carry out their research, engage in effective knowledge translation and affect policy and clinical practice. Perhaps more importantly are the personal impacts that small-scale funding has on researchers such as the research capacity that is built through new collaborations, mentorship, and the confidence and skill to pursue further research endeavours.