Inclusion of mobile telephone numbers into an ongoing population health survey in New South Wales, Australia, using an overlapping dual-frame design: impact on the time series
BMC Research Notes volume 7, Article number: 517 (2014)
Since 1997, the NSW Population Health Survey (NSWPHS) had selected the sample using random digit dialing of landline telephone numbers. When the survey began coverage of the population by landline phone frames was high (96%). As landline coverage in Australia has declined and continues to do so, in 2012, a sample of mobile telephone numbers was added to the survey using an overlapping dual-frame design. Details of the methodology are published elsewhere. This paper discusses the impacts of the sampling frame change on the time series, and provides possible approaches to handling these impacts.
Prevalence estimates were calculated for type of phone-use, and a range of health indicators. Prevalence ratios (PR) for each of the health indicators were also calculated using Poisson regression analysis with robust variance estimation by type of phone-use. Health estimates for 2012 were compared to 2011. The full time series was examined for selected health indicators.
It was estimated from the 2012 NSWPHS that 20.0% of the NSW population were mobile-only phone users. Looking at the full time series for overweight or obese and current smoking if the NSWPHS had continued to be undertaken only using a landline frame, overweight or obese would have been shown to continue to increase and current smoking would have been shown to continue to decrease. However, with the introduction of the overlapping dual-frame design in 2012, overweight or obese increased until 2011 and then decreased in 2012, and current smoking decreased until 2011, and then increased in 2012. Our examination of these time series showed that the changes were a consequence of the sampling frame change and were not real changes. Both the backcasting method and the minimal coverage method could adequately adjust for the design change and allow for the continuation of the time series.
The inclusion of the mobile telephone numbers, through an overlapping dual-frame design, did impact on the time series for some of the health indicators collected through the NSWPHS, but only in that it corrected the estimates that were being calculated from a sample frame that was progressively covering less of the population.
Since 2002, information about the health of the NSW population— 7.3 million— has been obtained using the NSW Population Health Survey (NSWPHS) . This survey is a continuous sample survey of approximately 15,000 persons each year; with equal numbers selected from each of the strata — health administrative areas— using random digit dialing (RDD) of landline telephone numbers and computer assisted telephone interviewing (CATI). Prior to 2002, two surveys were conducted in 1997 and 1998. When the survey began in 1997 coverage of the Australian population by landline phone frames was estimated to be 96%, however by 2006 coverage had dropped to 91% and by 2009 coverage had declined even further to 85% [2, 3]. This decline in the coverage of the population by landline phone frames was consistent with what was occurring in other countries [4–6].
Although not routinely available in Australia, differences in health risk factor and health status prevalence estimates between adults covered in a landline phone frame, and adults who live in a household without a landline telephone connection on which to make and answer calls (ie mobile-only phone users), have been measured in the USA using the National Health Interview Survey (NHIS) since 2003 . This survey found substantial differences for the health indicators: five or more drinks in one day at least once in the past year (17.5% v 30.5% - 74% higher), current smokers (14.5% v 24.3% - 68% higher), and ever diagnosed with diabetes (10.8% v 6.2% - 43% lower) .
Based on this overseas experience, it was estimated that when the overall landline phone coverage dropped below 85% in Australia, the differences in health indicators between people who were covered in the landline phone frame and mobile-only phone users differed by more than 50%, and/or there were associations between phone usage and the health indicators, it would impact appreciably on the overall NSW prevalence estimates . In 2010, the landline phone coverage in Australia dropped below 85%, and an Australian study showed relative differences between those people who were covered by the landline phone frame, and mobile-only phone users, of 69% for current smoking (20.4% v 34.5%) and 56% for ever diagnosed with diabetes (11.7% v 5.2%) . This study also showed that for current smoking, even after adjusting by age and sex, the mobile-only phone users were more likely to be current smokers. So, because of these changes in landline phone coverage, mobile telephone numbers were included in the 2012 NSWPHS, using an overlapping dual-frame design . Within a stratum the landline sample was selected using equal probability of selection of landline telephone numbers and then random selection of one person from the selected household. In the mobile phone sample an equal probability sample of mobile telephone numbers in Australia was selected and screened for adult residents in NSW. If the respondent had one or more children one child was selected at random. Sample weights thus reflected the differing sampling probabilities. The sample weights of the dual phone-users were then adjusted so that the composite factor used to combine the estimates for this component obtained from the landline sample and the mobile phone sample, λ, was set at 0.5. Benchmarking to the reference population was then performed by adjusting the weights for differences between weighted estimates of the age and sex structure obtained from the combined landline and mobile phone sample and ABS mid-year population estimates for each stratum. More details about the weighting strategy are provided in Barr et al. .
This paper firstly provides prevalence estimates for type of phone-use, health risk factor and health status from the 2012 NSWPHS. It also examines if, as other authors have found, there are any associations between the health indicators and type of phone-use, adjusting for the weighting variables. Health estimates from the 2012 NSWPHS, and the landline frame sample, re-benchmarked to the NSW population, were then compared to the previous year’s estimates. The impacts on the time series of the change in design to an overlapping dual-frame design is then discussed as are possible approaches to handling these impacts.
Data from the NSWPHS for 2012 was used. This consisted of data on 15,214 respondents with 10,518 (69.1%) from the landline phone frame, of which 1,792, (17.0%) were landline only, and 4,696 (30.1%) from the mobile phone frame, of which 1,121 (23.9%) were mobile only. The overall response rate was 31.0%, co-operation rate was 63.4%, refusal rate was 17.9% and contact rate was 66.9%.
Calculation of prevalence estimates
Estimates for type of phone-use were calculated overall and by selected demographic characteristics. Health risk factor and health status indicators were selected from the questions asked in the survey as shown in Table 1. Prevalence estimates and 95 per cent confidence intervals using the SURVEYFREQ procedure in SAS, which uses the Taylor expansion method to calculate sampling errors of complex sample designs, were calculated for each indicator .
Associations between the health indicators and type of phone-use
Prevalence ratios (PR) for each of the health indicators were calculated using Poisson regression analysis with robust variance estimation by type of phone-use using the categories mobile-only, landline-only, dual phone users in the mobile frame, and dual phone users in the landline frame as the reference category. This analysis was then repeated adjusting for all of the weighting variables including age group, sex, administration area, number of eligible persons in the household, and number of phone lines.
This analysis used the GENMOD procedure in SAS. As the Poisson model uses the natural logarithm as the link function, exponentiation of the parameter estimates was used to obtain the PRs for the study factors [14–16].
Comparison of 2012 prevalence estimates with previous years
Estimates for health related variables for the 2012 NSWPHS, as well as using just the landline frame sample, re-benchmarked to the NSW population, were then compared to the 2011 NSWPHS. Significant differences were identified by comparing the differences between the two estimates, divided by the standard error of the differences, calculated as √[SE(E2011PHS)2 + SE(E2012PHS)2]), with the standard normal distribution . The full time series was examined for health indicators where there was an association between type of phone-use and the indicator, and both significance and direction changed between sampling designs.
Two solutions were considered to adjust the time series for the expansion of the coverage of the survey. The first being the backcasting method  where the 2012 figures were used and a correction factor was applied to each of the proceeding annual time points using the formula where is the estimate from the landline phone frame, is the revised estimate, A is the landline frame, and b is the mobile-only phone users. The difference is the relative difference measured in this study, which is , and is the coverage each year as reported by the Australian Communication and Media Authority (ACMA) [2, 3]. The second method was the minimal coverage method, which only allowed inclusion of point estimates into the time series where there was adequate population coverage, with adequate population coverage being defined as 85% or above .
Prevalence estimates for 2012
It was estimated from the 2012 NSWPHS that 20.0% (95%CI 18.3%-22.0%) of the NSW population were mobile-only phone users, 9.6% (95%CI 8.8%-10.4%) were landline only, and 70.2% (95%CI 68.4%-72.1%) were dual phone users. Table 2 shows estimates for type of phone-use overall, and, for selected demographic characteristics. As shown in Table 2, mobile-only phone user rates were highest in young people, unmarried people, and Aboriginal and/or Torres Strait Islander peoples.
With regard to health risk factor and health status indicators, it was estimated from the dual frame 2012 NSWPHS as shown in Table 3, that 11.1% (95%CI 9.9%-12.2%) of the population drank five or more drinks of alcohol in a day, 27.6% (95%CI 25.9%-29.3%) drank more than two alcoholic drinks in a day, 53.4% (95%CI 51.5%-55.3%) met the recommended fruit intake, 10.0% (95%CI 8.8%-11.1%) met the recommended vegetable intake, 17.1% (95%CI 15.6%-18.6%) were current smokers, 56.2% (95%CI 54.2%-58.1%) did adequate physical activity, 82.4% (95%CI 81.2%-83.6%) had positive self-rated health status, 10.1% (95%CI 9.1%-11.1%) had current asthma, 8.4% (95%CI 7.5%-9.2%) were ever diagnosed with diabetes, and 49.7% (95%CI 47.7%-51.6%) were overweight or obese.
Associations between the health indicators and type of phone-use
Table 3 also shows the health indicator prevalence estimates for the 2012 NSWPHS overall, and for those with landline phones, who may also have a mobile phone, and those who are mobile-only phone users. As shown in Table 3 there were relative differences of more than 50% for five or more drinks of alcohol in a day (9.0% v 19.3%, 114% higher), and current smoking (14.0% v 28.3%, 103% higher).
Table 4 shows null PRs and adjusted PRs, for weighting variables, by type of phone-use for each of the selected health indicators from the 2012 NSWPHS. As shown in Table 4, after adjusting by the weighting variables of age group, sex, administration area, number of phone lines, and number of eligible persons in the household, mobile-only phone users were more likely to: drink five or more drinks of alcohol in a day (PR, 1.29; 95%CI, 1.04-1.59) and be current smokers (PR, 1.39; 95%CI, 1.20-1.63), and mobile-only phone users were less likely to meet the recommended vegetable intake (PR, 0.65; 95%CI 0.50-0.85) and be overweight or obese (PR, 0.90; 95%CI 0.83-0.97) than dual phone users from the landline frame. Also, after adjusting by the weighting variables of age group, sex, administration area, number of phone lines, and number of eligible persons in the household, dual phone users from the mobile frame were significantly different to the dual phone users from the landline frame: for more than two drinks of alcohol in a day (PR 0.88; 95%CI 0.80-0.97), current smoking (PR, 0.85; 95%CI 0.74-0.98), and current asthma (PR 0.77; 95%CI 0.65-0.91) which further supports the use of overlapping dual-frame designs, rather than screening dual-frame designs.
Comparison of 2012 prevalence estimates with previous years
Table 5 shows the health indicators estimates from the 2012 NSWPHS, compared to the 2011 NSWPHS. Significantly higher estimates were found in 2012 for: recommended fruit intake (from 50.4% to 53.4%, p = 0.016), recommended vegetable intake (from 8.4% to 10.0%, p = 0.026), current smoking (from 14.7% to 17.1%, p = 0.011), positive self-reported health status (from 80.3% to 82.4%, p = 0.010), and significantly lower estimates for overweight or obese (52.2% to 49.7%, p = 0.047).
Table 5 also shows the health indicators estimates, using just the landline frame sample for 2012, re-benchmarked to the NSW population, compared to the 2011 NSWPHS. Significantly higher estimates were again found for recommended fruit intake (from 50.4% to 55.9%, p < 0.001) and recommended vegetable intake (from 8.4% to 12.3%, p < 0.001), and significantly lower estimates for more than two alcoholic drinks in a day (29.6% to 27.1%, p = 0.042). However current smoking, positive self-reported health status, and overweight or obese were no longer statistically significantly different, and the difference had changed in direction for current smoking and overweight or obese.
Table 6 shows a summary of the factors used to predict if the sampling frame change is likely to impact on the time series. These factors are; 50% difference or more for non-landline frame persons, association between type of phone-use and the indicator, change in significance between sampling designs, and change in direction between sampling designs. Based on this analysis, the two indicators for which the time series was most likely to be affected, were current smoking and overweight or obese.
Looking at the full time series as shown in Figure 1, if the NSWPHS had continued to be undertaken only using a landline frame, overweight or obese would have been shown to continue to increase and current smoking would have been shown to continue to decrease. However, with the introduction of the overlapping dual-frame design in 2012, overweight or obese increased until 2011 and then decreased in 2012, and current smoking decreased until 2011, and then increased in 2012.
Adjusting the time series
The backcasting method, applying relative differences of 103% for current smoking and -21% for overweight or obese across all years and landline coverage of 96% from 1997-2002, 95% in 2003, 93.5% in 2004, 92.4% in 2005, 90.1% in 2006, 89.3% in 2007, 87.6% in 2008, 84.6% in 2009, 83.1% in 2010, 80.6% in 2011 and 77.8% in 2012, resulted, as shown in Figure 2, in the trend for current smoking continuing to decrease in 2012 and the trend for overweight or obese increasing until 2008, and plateauing thereafter. Lines of best fit were: y = -0.0112x2 + 44.411x – 43981, R2 = 0.9315 for current smoking, and y = -0.0523x2 + 210.18x – 211256, R2 = 0.9503 for overweight or obese where x = year-1996.
The minimal coverage method, removing years 2010 and 2011 when the population coverage was less than 85% resulted, as shown in Figure 2, in the trend for current smoking continuing to decrease in 2012, but at a lesser rate and the trend for overweight or obese increasing until 2008 and plateauing thereafter. Lines of best fit were: y = -0.0112x2 - 0.3169x + 25.34, R2 = 0.9156 for current smoking, and y = -0.0641x2 + 1.7555x + 39.093, R2 = 0.9393 for overweight or obese where x = year-1996.
Preliminary estimates for the first quarter of 2013 were also included for current smoking and overweight or obese; to examine which of the adjustment methods would best predict the 2013 estimates. Both methods were very close for overweight and obese, and the backcasting method was slightly better for current smoking. However, both methods had similar trajectories into the future and both were getting further away from the landline frame trajectory.
We had previously found that the dual-frame gave a more representative sample . The type of phone-use estimates from the 2012 NSWPHS were similar to those published for Australia in 2012 by ACMA from the Roy Morgan Single Source Survey— 19.9% for mobile-only, 8.0% for landline-only and 69.8% for dual phone users .
In this study, we found relative differences of over 50% between people who were covered by the landline frame and mobile-only phone users for health indicators: five or more drinks of alcohol in a day, and current smoking. We also found that type of phone-use was associated with many of the health indicators, in particular mobile-only phone users were significantly different for: drink five or more drinks of alcohol in a day, current smoking, recommended vegetable intake, and overweight or obese, even after adjusting for the weighting variables. Our results were consistent with other studies [10, 20, 21].
When we compared the health indicators estimates from the 2012 NSWPHS to the 2011 NSWPHS, we found significant differences for recommended fruit intake, recommended vegetable intake, current smoking, positive self-reported health status, and overweight or obese. However, when we compared the health indicators estimates using only the landline frame sample, re-benchmarked to the NSW population, to the 2011 NSWPHS, current smoking, positive self-reported health status, and overweight or obese were no longer significantly different, and the difference had changed in direction, for current smoking and overweight or obese.
So, how do we interpret these changes? Did current smoking really increase in 2012, and did overweight or obese really decrease in 2012, or is it a consequence of the design change? Our examination of the time series for current smoking and overweight or obese showed that it was a consequence of the design change and not a real change.
The backcasting method was best able to predict the 2013 figures for current smoking. This method appears superior to the minimal coverage method in that it not only corrects the years when the landline frame coverage was sub-optimal, but it also adjusts the estimates to what they should have been for all the other years, if mobile-only phone users were included. However the backcasting method also requires the making of several assumptions, that being that the relative difference between people covered by the landline phone frame and mobile-only phone users has remained constant over time, and, that the landline phone coverage estimates for Australia were appropriate for NSW. Also, for the backcasting method, a more complex formula would need to be used if it was being used for demographic groups— requiring for each group which is not currently available— and could quite quickly become very complex with numerous assumptions needing to be made. The minimal coverage method does not require any additional assumptions to be made; it just requires a decision on what is considered adequate population coverage by the sample frame or frames.
Figure 2 shows that the difference between the landline frame time series and the adjusted dual-frame time series, for current smoking and overweight or obese, are widening over time. A recent study on the European telephone surveys has concluded that coverage bias from surveys using only landline frames in Europe are increasing over time . This study highlighted the need for mobile telephone number augmentation of the sample to occur prior to the landline phone coverage becoming sub-optimal.
The inclusion of the mobile telephone numbers through an overlapping dual-frame design did impact on the time series for some of the health risk factors and health status estimates, in that it corrected the estimates that were being calculated from a sample frame, which was getting progressively less representative of the population. Therefore, continuing to use only landline frames in Australia, although maintaining the same design, is not keeping the estimates the same because of the decreasing coverage of landline frames.
Health estimates from surveys using only landline frame sampling are progressively getting “further from the truth”.
MLB is a PhD student with the National Institute for Applied Statistics Research, University of Wollongong, Wollongong, Australia.
Computer assisted telephone interviewing
National Health Interview Survey
New South Wales
NSW Population Health Survey
Random digit dialing
Roy Morgan Single Source Survey.
NSW Ministry of Health, NSW Population Health Surveys.http://www.health.nsw.gov.au/surveys/Pages/default.aspx,
Australian Bureau of Statistics: Population Survey Monitor. Catalogue no 4103.0. 1996, Canberra: ABS
Australian Communications and Media Authority (ACMA): Communications report 2010–11. 2011, Melbourne: ACMA,http://www.acma.gov.au/webwr/_assets/main/lib410148/communications_report_2010-11.pdf,
Kuusela V, Callegaro M, Vehovar V: The influence of mobile telephones on telephone surveys. Advances in Telephone Survey Methodology. Edited by: Lepkowski J, Tucker C, Brick M, De Leeuw E, Japec L, Lavrakas PJ, Link M, Sangste R. 2007, Hoboken, NJ: Wiley, 87-112.
Lee S, Brick JM, Brown ER, Grant D: Growing cell-home population and non-coverage bias in traditional random digit dial telephone health surveys. Health Serv Res. 2010, 45 (4): 1121-1139.
Lynn P, Kaminska O: The impact of mobile phones on survey measurement error. Institute for Social and Economic Research Working Paper Series No 2011-07. 2011, Essex: University of Essex,https://www.iser.essex.ac.uk/publications/working-papers/iser/2011-07.pdf,
National Health Interview Survey.http://www.cdc.gov/nchs/nhis.htm,
Blumberg SJ, Luke JV: Wireless substitution: Estimates from the National Health Interview Survey. January - June 2012. 2012, National Centre for Health Statistics,http://www.cdc.gov/nchs/data/nhis/earlyrelease/wireless201212.PDF,
Barr ML: Predicting when declining landline frame coverage will impact on the overall health estimates for the NSW Population Health Survey. 2008, Sydney: Centre for Epidemiology and Research, NSW Department of Health,http://www.health.nsw.gov.au/surveys/other/Documents/predicting-when-mobile-only-impacts-2008.pdf,
Pennay D, Bishop N: Profiling the ‘mobile phone only’ population: A study of Australians with a mobile phone and no landline telephone. 2009, Melbourne: The Social Research Centre Pty Ltd
Barr ML, van Ritten JJ, Steel DG, Thackway SV: Inclusion of mobile phone numbers into an ongoing population health survey in New South Wales, Australia: design, methods, call outcomes, costs and sample representativeness. BMC Med Res Methodology. 2012, 12: 177-
Barr ML, Ferguson RA, Hughes PJ, Steel DG: Inclusion of mobile phone numbers into an ongoing population health survey in New South Wales, Australia: final weighting strategy. University of Wollongong Working Paper. 2014,http://niasra.uow.edu.au/content/groups/public/@web/@inf/@math/documents/doc/uow167062.pdf,
SAS Institute: The SAS System for Windows version 9.2. 2009, Cary, NC: SAS Institute Inc, Further information available from http://www.sas.com
Zou G: A modified Poisson regression approach to prospective studies with binary data. Am J Epidemiol. 2004, 159: 702-706.
Lee J, Tan CS, Chia KS: A practical guide for multivariate analysis of dichotomous outcomes. Ann Acad Med Singapore. 2009, 38: 714-719.
Cole SR: Analysis of complex survey data using SAS Computer Methods Programs. Biomed. 2001, 64: 65-69.
Altman DG, Bland JM: Interaction revisited: the differences between two estimates. BMJ. 2003, 326: 219-
Van den Brakel JA, Smith PA, Compton S: Quality procedures for survey transitions – experiments, time series and discontinuities. Surv Res Methods. 2008, 2 (3): 123-141.
Australian Communications and Media Authority (ACMA): Communications report 2011–12. 2013, Melbourne: ACMA,http://www.acma.gov.au/webwr/_assets/main/lib550049/comms_report_2011-12.pdf,
Pennay D: Profiling the ‘mobile phone only’ population: Results from a dual- frame telephone survey using a landline and mobile phone sample frame, ASCPRI Social Science Methodology conference proceedings. ASCPRI. 2010, 2010: 2010-
Livingston M, Dietze P, Ferris J, Pennay D, Hayes L, Lenton S: Surveying alcohol and other drug use through telephone sampling: a comparison of landline and mobile phone samples. BMC Med Res Methodol. 2013, 13: 41-
Mohorko A, de Leeuw E, Hox J: Coverage bias in European Telephone Surveys: Development of landline and mobile phone coverage across countries and over time. Survey Methods. Insights from the Field. 2013, Retrieved from http://surveyinsights.org/?p=828
We acknowledge the interviewing staff and supervisors at the Centre for Epidemiology and Evidence, NSW Ministry of Health for conducting the interviews. We also acknowledge the respondents for participating in the survey.
The authors declare that they have no competing interests.
MLB developed the overall concepts and planned the study; analysed the data, wrote the methods and results, wrote the introduction and discussion, and finalised the manuscript. RAF managed the data, checked the analysis programs and commented on drafts of the manuscript. DGS provided development and analysis advice, checked the underlying logic of the statistics, and commented on drafts of the manuscript. All authors read and approved the final manuscript.
About this article
Cite this article
Barr, M.L., Ferguson, R.A. & Steel, D.G. Inclusion of mobile telephone numbers into an ongoing population health survey in New South Wales, Australia, using an overlapping dual-frame design: impact on the time series. BMC Res Notes 7, 517 (2014). https://doi.org/10.1186/1756-0500-7-517