- Short Report
- Open Access
Test-retest reliability and construct validity of the ENERGY-parent questionnaire on parenting practices, energy balance-related behaviours and their potential behavioural determinants: the ENERGY-project
BMC Research Notes volume 5, Article number: 434 (2012)
Insight in parental energy balance-related behaviours, their determinants and parenting practices are important to inform childhood obesity prevention. Therefore, reliable and valid tools to measure these variables in large-scale population research are needed. The objective of the current study was to examine the test-retest reliability and construct validity of the parent questionnaire used in the ENERGY-project, assessing parental energy balance-related behaviours, their determinants, and parenting practices among parents of 10–12 year old children.
We collected data among parents (n = 316 in the test-retest reliability study; n = 109 in the construct validity study) of 10–12 year-old children in six European countries, i.e. Belgium, Greece, Hungary, the Netherlands, Norway, and Spain. Test-retest reliability was assessed using the intra-class correlation coefficient (ICC) and percentage agreement comparing scores from two measurements, administered one week apart. To assess construct validity, the agreement between questionnaire responses and a subsequent interview was assessed using ICC and percentage agreement.
All but one item showed good to excellent test-retest reliability as indicated by ICCs > .60 or percentage agreement ≥ 75%. Construct validity appeared to be good to excellent for 92 out of 121 items, as indicated by ICCs > .60 or percentage agreement ≥ 75%. From the other 29 items, construct validity was moderate for 24 and poor for 5 items.
The reliability and construct validity of the items of the ENERGY-parent questionnaire on multiple energy balance-related behaviours, their potential determinants, and parenting practices appears to be good. Based on the results of the validity study, we strongly recommend adapting parts of the ENERGY-parent questionnaire if used in future research.
There is a need for more carefully developed interventions to curb the obesity epidemic in youth. Prevention of unnecessary weight gain should target energy intake and energy expenditure behaviours, also referred to as energy balance-related behaviours (EBRBs), i.e. physical activity, sedentary, and dietary behaviours . Development of effective interventions requires insight into which specific EBRBs are most relevant and into the correlates and determinants of these behaviours.
Parental and home-environmental factors are crucial determinants of children’s EBRBs and related weight status .
The ENERGY-cross sectional study  surveyed more than 7000 children, their parents, and schools in seven countries representing different regions of Europe using questionnaires aiming children, one of their parents, school staff as well as school environment observations. We wanted to assess a range of parental and home environmental factors with the parent questionnaire. However, no established valid and reliable measures that could be administrated in large population studies across Europe were available.
A similar study was conducted to test the child questionnaire . We found that the ENERGY-child questionnaire has good test-retest reliability and moderate to good construct validity for the large majority of items. Therefore, a parent questionnaire was developed and a test-retest reliability and construct validity study was conducted. In the current paper, the methods and results of the test-retest reliability and construct validity study of the parent questionnaire are presented and discussed.
The ENERGY-parent questionnaire assessed behaviours and potential determinants regarding soft drinks and fruit juices, breakfast, dieting, physical activity, screen viewing, and sleeping of the child. The potential determinants included automaticity of behavioural choices, home availability, parenting practices and (economic) environmental factors.
Most of these concepts were assessed by one or two items. All correlates, except questions about family rules, were assessed on 5-point scales, with response categories ranging from ‘I fully agree’ to ‘I fully disagree’ or ‘Always’ to ‘Never’. Details on the development of the questionnaire, the pre-testing, and translation procedures are described elsewhere . The ENERGY-parent questionnaire is available in English and all languages in which the questionnaire was administered via the ENERGY-website: http://www.projectenergy.eu.
Study population: recruitment and data collection
In six countries the test-retest reliability and construct validity study was conducted (Belgium, Greece, Hungary, the Netherlands, Norway, Spain) according to the standardized protocol described hereafter and in the Additional file 1. Due to deviations from the study protocol, the data from a seventh country (Slovenia) were excluded from the current study.
We recruited parents of children aged 10–12 years old. The recruitment and data collection took place from March until July 2010. Parents were recruited via 1) the schools of children participating in the children test-retest/construct validity study  or 2) other schools that the research team had personal contacts with.
In countries where ethical approval was necessary for such non-intervention studies this was obtained from the relevant ethical committee.
- Test-retest reliability study
When the parents agreed to participate in the test-retest reliability study, we emphasized the importance of the fact that the questionnaires must be completed on a weekday, and once more by the same person, exactly one week later. We sent the questionnaires to the home address of the parents 3–4 days apart; a stamped addressed envelope to return the questionnaire was included with both.
- Construct validity study
Parents who participated in the validity study were asked to fill in the questionnaire and were subsequently interviewed by a researcher or research assistant. The interview was performed using a standard question route. The interviews were recorded and transcribed. Based on the transcribed interview, a second researcher or research assistant (i.e. other than the one doing the interview) filled in a second identical parent questionnaire without knowledge of the answers to the first questionnaire of the parent.
We calculated means and standard deviations (participant characteristics) and medians, 25th, and 75th percentiles values (EBRBs).
For both test-retest reliability and construct validity we assessed agreement at the individual item level. The agreement of categorical, continuous, and dichotomous items was analysed with a two-way random effects single measure intraclass correlation coefficient (ICC 2.1). ICCs were classified as ‘excellent’ (≥ .81), ‘good’ (.61 – .80), ‘moderate’ (.41 - .60), and ‘poor’ (≤ .40) [5–8].
Since the calculation of ICCs depends on the existence of the variability in answering categories, we also calculated percentage agreement, with criteria established as ‘excellent’ (90% - 100%), ‘good’ (75% - 89%), ‘moderate’ (60%-74%), or ‘poor’ (< 60%). If ICC values were lower than .40/.60/.80 but the percentage agreement was equal to or higher than 60%/75%/90%, we reported the percentage agreement .
All statistical tests were performed using SPSS version 18.0 (SPSS Inc., Chicago, IL).
The characteristics of the participants from both the test-retest reliability and construct validity study are shown in Table 1.
The number of participants in the test-retest reliability study ranged from 39 (Norway) to 70 (Belgium). Completion of the questionnaire took 25–50 minutes.
In the construct validity study all countries included 20 parents, except for Spain (9 parents).
The majority of the interviews were carried out face-to-face, except for Belgium and Greece, where almost all interviews were carried out via telephone. The interviews took about 60 minutes.
Table 2 presents descriptive statistics of the parental energy balance-related behaviours.
Test-retest reliability and construct validity study
Test-retest reliability study
Table 3 shows the questionnaire items, their ICC values, and percentage agreement for all countries combined, for the test-retest reliability and construct validity study. Table 4 summarises the findings per category of the ENERGY-parent questionnaire. For the total sample across all countries, the test-retest reliability was good to excellent for 99.2% or all but one of the 121 response items. The test-retest reliability was comparable across the six countries.
Construct validity study
Construct validity appeared to be good to excellent for 92 out of 121 items (76.0%). For the remaining 29 items, construct validity was moderate for 24 (19.8%) and poor for 5 (4.1%) items. Three response items did not show enough variability, resulting in ICCs ≤ .40, but had high (≥ 90%) percentage agreement (Table 4).
The construct validity was comparable across all countries, except for Belgium and Greece. The Belgian and Greek data showed lower construct validity values than the other countries, especially for the constructs that were already more likely to show moderate or poor values in the overall data set.
The results of our study indicate that the ENERGY-parent questionnaire, assessing parenting practices, parental energy balance-related behaviours and their potential determinants, has adequate test-retest reliability and construct validity.
To our knowledge, this is the first study assessing the reliability and validity of a questionnaire on parental behaviours, their determinants and parenting practices for different EBRBs across different countries.
All but one item showed good to excellent test-retest reliability. A large majority of the items had high ICC values and percentage agreement. Some items (e.g. ‘There are breakfast products (e.g. milk, cereal, bread) available at home for my child’) had high agreement with low ICC values, due to little variation in parental responses. Despite these mixed ICC scores, the majority of the percentage agreement between the questionnaires indicated good to excellent test-retest reliability of the questions.
Values for the construct validity were somewhat lower than those for the test-rest reliability. A closer examination indicated that all questions on automaticity (e.g. ‘Drinking soft drinks is something I do without really thinking about’) - often used in questionnaires to assess habit strength regarding a specific behaviour  - showed moderate or even poor values for validity. The lower validity for the habit strength questions is consistent with the findings of the ENERGY-child questionnaire . These findings indicate that these questions should not be used in further research.
Also, with regard to parenting practices, some questions showed lower ICCs across EBRBs. The questions in which parents were asked to indicate whether they negotiate with their child about the EBRBs had moderate or poor construct validity. The interviews indicated that ‘negotiation with the child’ was a concept many respondents did not understand or find relevant. Consequently, these questions should not be used in future research.
The Greek data, and the Belgian data in particular, showed lower construct validity values than the other countries, especially for the constructs that were already more likely to show moderate or poor values in the overall data set. The reason for this deviation may be that in Belgium and Greece almost all interviews (i.e. 17 in Belgium and 15 in Greece) were conducted via telephone, whereas in the other countries the majority of the interviews were conducted face-to-face.
Since the number of respondents per country was relatively small (i.e. 20 in most countries), more value should be attached to the results of the overall data set than to the country-specific analyses.
Our findings suggest that the ENERGY-parent questionnaire has good to excellent test-retest reliability and good construct validity among parents of European children aged 10–12 years.
Based on the results of the validity study, we strongly recommend adapting parts of the ENERGY-parent questionnaire if used in future research.
Availability of supporting data
The data set supporting the results of this article are included within the article and its additional files.
The ENERGY-project is funded by the Seventh Framework Programme (CORDIS FP7) of the European Commission, HEALTH (FP7-HEALTH-2007-B). The content of this article reflects only the authors’ views and the European Community is not liable for any use that may be made of the information contained therein.
Intra-class correlation coefficient
Energy balance-related behaviours.
Golan M, Crow S: Parents are key players in the prevention and treatment of weight-related problems. Nutr Rev. 2004, 62: 39-50. 10.1111/j.1753-4887.2004.tb00005.x.
Brug J, te Velde SJ, Chinapaw MJM, Bere E, de Bourdeaudhuij I, Moore H, Maes L, Jensen J, Manios Y, Lien N, Klepp KI, Lobstein T, Martens M, Salmon J, Singh AS: Evidence-based development of school-based and family-involved prevention of overweight across Europe: the ENERGY-project's design and conceptual framework. BMC Publ Health. 2012, 10: 276-
Singh AS, Vik FN, Chinapaw MJM, Uijtdewilligen L, Verloigne M, Fernández-Alvira JM, Stomfai S, Manios Y, Martens MK, Brug J: Test-retest reliability and construct validity of the ENERGY-child questionnaire on energy balance-related behaviours and their potential determinants: the ENERGY-project. Int J Behav Nutr Phys Act. 2011, 8: 136-10.1186/1479-5868-8-136.
van Stralen MM, te Velde SJ, Singh AS, de Bourdeaudhuij I, Martens MK, van der Sluis M, Manios Y, Grammatikaki E, Chinapaw MJM, Maes L, Bere E, Jensen J, Moreno L, Jan N, Molnár D, Moore H, Brug J: EuropeaN Energy balance Research to prevent excessive weight Gain among Youth (ENERGY) project: Design and methodology of the ENERGY cross-sectional survey. BMC Publ Health. 2011, 11: 65-10.1186/1471-2458-11-65.
Landis JR, Koch GG: The measurement of observer agreement for categorical data. Biometrics. 1977, 33: 159-174. 10.2307/2529310.
McGraw KO, Wong SP: Forming inferences about some intraclass correlation coefficients. Psychol Meth. 1996, 1: 30-46.
Nunnally JC, Bernstein IH: Psychometric Theory. 1994, McGraw-Hill, New York
Kohl HW, Fulton JE, Caspersen CJ: Assessment of physical activity among children and adolescents: A review and synthesis. Prev Med. 2000, 31: S54-S76. 10.1006/pmed.1999.0542.
Saelens BE, Frank LD, Auffrey C, Whitaker RC, Burdette HL, Colabianchi N: Measuring physical environments of parks and playgrounds: EAPRS instrument development and inter-rater reliability. J Phys Act Health. 2006, 3 (Suppl 1): S190-S207.
Aarts H, Paulussen T, Schaalma H: Physical exercise habit: on the conceptualization and formation of habitual health behaviours. Health Educ Res. 1997, 12: 363-374. 10.1093/her/12.3.363.
We would like to thank Manouk de Hooge, who contributed to the data collection during her internship.
There are no financial or non-financial competing interests to declare in relation to this manuscript.
ASS set up and coordinated the study, supervised data analyses and drafted the manuscript. MJMC supervised the study set up and helped to draft the manuscript. LU carried out the statistical analyses. FNV, WvL, JMFA, SS and YM collected the data in the countries or supervised data collection. MvS helped to develop the data collection protocol. CT advised on the statistical methods. JB is the coordinator of the ENERGY-project and helped to draft the manuscript. All co-authors read and approved the final version of the manuscript.