Although health research focuses on the development of evidence, there has been surprisingly little effort to evaluate the effects of research and research funding programs. Some research program evaluations of agencies such as the NIH[1] and the Health and Health Research Council of New Zealand[2] have relied on bibliometric methods to review their granting programs. These reviews typically consist of recording publications per funded project and using this information to compare the number of papers published among comparable institutes or grants[1, 2]. Although useful, these evaluations are limited in scope, and do not address the multidimensional aspects of research impact[3].
Research frameworks have been developed by several groups[3, 4] and institutions[5] as a means for evaluating of assessing a range of impacts resulting from health services research. health research. The Research Impact Framework[3] identifies four areas of impact: Research-related impacts include traditional measures of research success (publications, conferences, patents and awards) as well impacts on the research field itself through communication, knowledge translation and research networks; policy impacts include research that informs and influences policy; service impacts include improved methods and cost-effectiveness of delivering quality health-care; societal impacts include outcomes that affect the population beyond the research world. A clear research impact framework can contribute greatly to the evaluation of research grant programs by identifying the areas of potential impact and illustrating the relationships between them.
Agencies such as The Health and Health Services Research Fund of Hong Kong[6], Public Health Research and Development Committee of Australia[7] and the Alberta Heritage Fund for Medical Research[8] have employed similar frameworks in the evaluation of granting programs to demonstrate significant policy, service and research impacts[6–8]. However, these frameworks are typically geared toward large grants with equally large outcomes and are not easily transferable to the evaluation of smaller scale research projects designed to build research capacity. To our knowledge, there is no literature that assesses the impact of very small grants or suggests a framework within which to evaluate the impact of very small grants.
The IWK Health Centre is a women and children's academic health sciences centre, affiliated with Dalhousie University, serving Maritime Canada. The IWK has a vigorous research culture that thrives across disciplines and levels of expertise. Through the IWK's Research Services Department, a variety of granting opportunities are available within the research award program. The goal of this program is to promote the development of high quality research, consistent with the mission of the health centre. These grants are designed to support research that can be completed within a proposed budget or provide seed funding for the development of larger proposals for external funding. The grants include Category A grants funding up to ($4,000.00) and Category B grants (funding up to $15,000.00). These awards undergo rigorous peer review by the IWK Scientific Review Committee. Since 2002, over $800,000 has been awarded through 165 Category A and B grants, mostly to novice and junior researchers.
The research impact literature has no known research addressing the impact of very small research grants. Nor are there any suitable frameworks in which to measure the impact of very small grants. The purpose of the present study was to describe the impact of health research facilitated by Research Operating Grants at the IWK Health Centre within the context of a relevant research impact framework. To this end, an adaptation of the Research Impact Framework[3] was used to assess the dimensions most applicable to the impact of very small grants.
Design
Retrospective on-line survey conducted between June 6, 2008 and August 22, 2008.
Participants
Principal Investigators of Research Operating Grants (valued at under $15,000) from the IWK Health Centre awarded between the years 2004-2006. Multiple IWK grant holders were queried on only one grant randomly selected by the investigators. After exclusions for multiple grants, 64 eligible grant holders were invited to participate. Seven investigators declined to participate and 18 failed to complete the questionnaire, leaving a final group of 39 (61%).
Measures: The Research Impact Assessment Questionnaire [additional file 1] was adapted from the Research Impact Framework[3] and included 5 subsections designed to measure various impacts of the research. Subsections included: (1) research (impact on research itself), (2) policy (impact and influence on policy), (3) practice (impact and influence on medical practice), (4) society (impact on society) and (5) personal (measure of self or career development). Additional sections were included for demographic information, difficulties encountered during the research and future plans for research.
The survey was hosted on a secure web server maintained by Dalhousie University. The Opinio Web Server uses Opinio survey software which allows researcher to produce and publish online surveys. Questionnaire items included multiple choice, short answer and yes/no questions. For almost all of the questions, a comment box was provided for participants to elaborate on their responses.
Procedure
Potential participants were identified and contacted using information provided in the IWK Research Services' database. Contact and consent were completed by email. Participants were provided a randomly generated identification number to access the questionnaire.
Reminders were sent via email to participants who had not completed the questionnaire after 2 weeks and again after 4 weeks. After a further 2 weeks of inactivity following the second reminder sent to Principal Investigators, Co-Investigators were invited to complete the questionnaire instead.