How big is the elephant in the room? Estimated and actual IT costs in an online behaviour change trial
© McCambridge et al; licensee BioMed Central Ltd. 2010
Received: 29 January 2010
Accepted: 18 June 2010
Published: 18 June 2010
The practical and methodological challenges inherent in online behaviour change studies are both novel and complex. We relate our experiences of estimating and managing information technology (IT) research and intervention costs in an ongoing internet trial in the hope that others will find this information useful.
Actual IT costs were approximately twice those that had been originally estimated by external contractors. These original estimates for IT costs allowed little scope for the identification of new needs, which was intrinsic to the iterative nature of the research enterprise.
Making greater provision for the uncertain nature of these costs in future studies is a key practical lesson for the planning of future online behaviour change studies.
Estimating research and intervention costs in the planning of any large study is a considerable practical challenge facing researchers. Overestimation may threaten the granting of research funding and underestimation can threaten the viability, quality and completion of the study. For most trials costs can be divided between those required to develop the intervention materials (both active and control) and the research evaluation. There is a particular need to take account of both the practical and methodological challenges inherent in online studies as the internet is increasingly used to both deliver healthcare and public health interventions, and to undertake research. Here the uncertainties unavoidably encountered in any relatively new field of study are acute, and this raises additional technical and financial issues. We relate our experiences of estimating and managing information technology (IT) research and intervention costs in an ongoing internet trial in the hope that others will find this information useful.
The Down Your Drink Trial as a case study
The Down Your Drink randomised controlled trial was funded by the U.K. Medical Research Council on behalf of the National Prevention Research Initiative (see acknowledgements for further details). This ongoing study compares the effectiveness of an enhanced website in helping people concerned about their drinking to achieve behaviour change in comparison with a standard alcohol website. The enhanced website was developed from an existing website which had been in operation for some years, with demonstrated potential for effectiveness. Thus IT costs presented here relate to the further development of an existing website for the "intervention" components of the trial and the new additional requirements related to the research evaluation. In practical terms, it was these additional costs that need to be estimated at the research proposal stage. For the research and for its economic evaluation component it is also necessary to be able to separate out research based costs (not included in the economic evaluation) from the intervention costs (which do need to be estimated and included). This will identify the actual costs of developing and maintaining the website to be incorporated into cost-effectiveness analyses.
Our chosen modus operandi involved a technical consultant liaising between the research team and the programmers who were based in a small private sector IT company. A competitive tendering process had occurred for the original establishment of the website some years previously and it would have been difficult for technical reasons to involve new personnel. As it was essential to have current expertise on web design and the user interface, we opted to work with the private sector rather than with an academic IT department. For this case study the use of a private company had the advantage that resources used in the trial were more specifically monitored than may have been the case in a large academic department. Communications between the various parties were effective, though constrained in ways to be described. We examined the literature for guidance on IT costs and had informal discussions with other e-health researchers, but at that time (during 2006) there was little collective experience of on-line trials. Some of the costs we present are specific to our model and our circumstances, though the lessons drawn from this experience are more clearly generalisable.
In a total research grant awarded of approximately £350,000* over three and a half years, we were awarded £21,400 for website development and maintenance costs, exactly as requested (N.B. all totals include Value Added Tax [VAT]; a general sales tax, which at the time of the study was 17.5%). We received £15,400 for website development and maintenance and £6,000 for the technical consultant who also fulfilled a project management function. These costings were based on estimates we received from both external contractors in the course of the research grant application, and before we started the work on redeveloping the existing website. Our funding award involved a pilot study being undertaken online 3 months after the initiation of the project, leading into a main trial after approximately one year.
IT Resource Use in Practice
Initial detailed work and costing plan
Implementation of the consent process and the randomization and group assignment system
Implementation of the primary and secondary outcome questionnaires, including the code
Basic e-mail reminders system and follow-up questionnaires system
Implementation of the control site (estimated about 15-20 pages of information)
Implementation of the three modules, based on restructuring some old content and adding more new content provided by research team
Technical consultancy & project management
AGREED SPEND AT OUTSET:
VAT @ 17.5%
These tasks were duly implemented. In so doing, it quickly became apparent that additional work would be required. This stemmed from the fundamentally iterative nature of the enterprise. Once particular tasks were completed, only then could it be established whether additional work was or was not needed. For example, testing new material with users was judged essential to the successful conduct of the study. It is difficult to know in advance, however, precisely what need for further work will emerge from this feedback and thus what the costs will be.
Additional work required
System to provide password protected downloadable data for the research team
Implementation of a participant withdrawal system
New randomization system
Incentives sub-study (incidental)
Changes to registration system following pilot study
Maintenance fee for the main trial
One Year follow-ups - implementation of the new incentives system
There was no overspend on the development of the Control Website
Modules development and content management system
Copyright fees for use of images
Additional features to enhance the attractiveness and functioning of the website
Set up standalone outcome measure site at http://tot-al.downyourdrink.org.uk (incidental)
Additional consultancy costs
VAT @ 17.5%
Some developments were also initiated or required by other parties. For example, one participant who wished to withdraw from the study contacted the ethical committee because they no longer wished to be bothered by automated e-mail messages. The ethical committee required that this be done, and we were very happy to do it, though this did involve unforeseen programming costs. We sought and obtained additional funding support for the newly identified extra work (see acknowledgements). If we had not been able to obtain additional funding, the conduct of the study would have been jeopardised.
There are two other types of research costs described in Table 2. Firstly, there were costs for the main trial that emanated from the pilot study, in which we altered our research decisions on the basis of accumulating research data. Secondly, other costs were attributable to incidental studies. These were sub-studies which were undertaken to optimise the research decision-making for the main trial. For example, we conducted a number of experimental studies examining various methods to minimise attrition, a central problem in internet longitudinal research. In both cases, the needs of the research study itself required unforeseen expenditures if it was to be done as well as possible.
Towards the end of the study, further work was identified to fulfil our initial commitment to make the website freely available after the end of the trial. This work was thus neither specifically an intervention nor a research cost and totalled £4,140. These were added to the total programming costs of £10,986 for research purposes, £19,448 on the development of the intervention website, £1,645 on the control website, and £8,460 on consultancy costs to give a final total for IT costs of £44, 679. Research-specific IT costs were much lower than were intervention IT costs, and costs for our Technical Consultant (which embraced both research and intervention activities) were far from trivial. Research staff costs have not been included here.
In total the actual costs of the IT work were more than twice what had originally been estimated (i.e. compared to £21,400). It was almost as if the elephant in the room was twice as big as we thought it would be, notwithstanding our efforts to obtain well informed estimates. With the benefit of hindsight our original estimates allowed little scope for the identification of new IT needs, beyond the need for maintenance costs, and thus were grossly inadequate. Perhaps we could have given more scrutiny to the estimates we received from contractors. Even so, the fundamental problem that we faced was that many of the additional costs could not have been specifically anticipated in advance as the need for them only became apparent during the course of the methodological and intervention design work that was the substance of the research itself. And we were reliant on the estimates provided by external expert IT contractors, precisely because they had expertise which we did not.
Current research funding systems rely on academic departments taking the risk for any underestimation of resources, while most research funding bodies have mechanisms to claw back those resources not used. This risk is managed by most universities through some pooling of risks across different projects and different funding streams to support research activities. However, these systems do not usually extend to flexibility in external contracts for specialist services such as IT support.
There is a limited pool of knowledge about how to do studies of this type effectively and efficiently and we certainly do not suggest that our particular model, involving an external technical consultant, should be taken up more widely. Particularly noteworthy for internet intervention developmental work is the fact that we deliberately eschewed advanced interactive features and videos on grounds of cost and user engagement. We were also aware of the need to limit not just the direct IT costs but also the hidden costs of research project staff time devoted to IT issues. Although these are difficult to separate from wider research costs, they may well exceed the IT specific costs and it was our experience that such staff time was extended by the inevitable communication problems encountered in bridging the worlds of academic research and the internet in addressing ongoing issues. Making greater provision for the essentially unknowable nature of these costs in future studies is thus a key practical lesson for the planning of future internet behaviour change studies.
*The value of financial costs may be approximately converted to U.S. dollars or Euros by multiplying by 1.5 or 1.2 respectively, though exchange rates obviously do vary.
This work was supported by the National Prevention Research Initiative, which includes the following funding partners: British Heart Foundation; Cancer Research UK; Department of Health; Diabetes UK; Economic and Social Research Council; Medical Research Council; Research and Development Office for the Northern Ireland Health and Social Services; Chief Scientist Office, Scottish Executive Health Department; and the Welsh Assembly Government http://www.npri.org.uk/. The Alcohol Education and Research Council provided additional funding to assist with developing the intervention site. No funder has had any role in the preparation of the manuscript or the decision to submit. We thank Alcohol Concern for their collaboration with the trial. We are grateful to Harvey Linke of Net Impact and Richard McGregor of Codeface Ltd. for their work in developing the intervention, comparator and trial websites, and Jo Burns for project management.
- Murray E, Khadjesari Z, White IR, Kalaitzaki E, Godfrey C, McCambridge J, Thompson SG, Wallace P: Methodological challenges in online trials. J Med Internet Res. 2009, 11 (1): e9-10.2196/jmir.1052.PubMed CentralPubMedView ArticleGoogle Scholar
- Murray E, McCambridge J, Khadjesari Z, White IR, Thompson SG, Godfrey C, Linke S, Wallace P: The DYD-RCT protocol: an on-line randomised controlled trial of an interactive computer-based intervention compared with a standard information website to reduce alcohol consumption among hazardous drinkers. BMC Public Health. 2007, 7 (1): 306-10.1186/1471-2458-7-306.PubMed CentralPubMedView ArticleGoogle Scholar
- Linke S, McCambridge J, Khadjesari Z, Wallace P, Murray E: Development of a psychologically enhanced interactive online intervention for hazardous drinking. Alcohol Alcohol. 2008, 43 (6): 669-674.PubMed CentralPubMedView ArticleGoogle Scholar
- Linke S, Brown A, Wallace P: Down your drink: a web-based intervention for people with excessive alcohol consumption. Alcohol Alcohol. 2004, 39 (1): 29-32.PubMedView ArticleGoogle Scholar
- Linke S, Murray E, Butler C, Wallace P: Internet-based interactive health intervention for the promotion of sensible drinking: patterns of use and potential impact on members of the general public. J Med Internet Res. 2007, 9 (2): e10-10.2196/jmir.9.2.e10.PubMed CentralPubMedView ArticleGoogle Scholar
- Thomas BA, McCambridge J: Comparative psychometric study of a range of hazardous drinking measures administered online in a youth population. Drug & Alcohol Dependence. 2008, 96: 121-127.View ArticleGoogle Scholar
- Cunningham JA, Kypri K, McCambridge J: The use of emerging technologies in alcohol treatment. Alcohol Research & Health.