Skip to main content

Insights from using an outcomes measurement properties search filter and conducting citation searches to locate psychometric articles of tools used to measure context attributes



To describe our experience with using a methodological outcomes measurement search filter (precise and sensitive versions of a filter designed to locate articles that report on psychometric properties of measurement tools) and citation searches to locate psychometric articles for tools that can be used to measure context attributes. To compare the precise filter when used alone and with reference list checking to citation searching according to number of records found, precision, and sensitivity.


Using the precise filter, we located 130 of 150 (86.6%) psychometric articles related to 22 of 31 (71.0%) tools that potentially measured an attribute of context. In a subset of six tools, the precise filter alone was more precise than searching with the precise filter combined with reference list searching, or citation searching alone. The precise filter combined with reference list checking was the most sensitive search method examined. Overall, we found the precise filter helpful for our project as it decreased record screening time. For non-patient reported outcomes tools, we had less success with locating psychometric articles using the precise filter because some psychometric articles were not indexed in PubMed. More research that systematically evaluates database searching methods is needed to validate our findings.

Peer Review reports


Evaluation of context is important for the translation of research into policy and practice [1,2,3,4]. Evaluation is the process by which implementation strategies, innovation, or context are tested using valid and reliable tools [5] to determine their ability to influence research uptake or outcomes [6]. Systematic identification of psychometric articles can be resource intensive and time-consuming [7]. Psychometrics are properties of measurement tools that dictate their ability to capture the concept they are intended to measure, minimize measurement error and detect change over time [8] Many validated search filters [9] have been developed to mitigate the workload associated with searching the extensive amount of published literature indexed in bibliographic databases. Methodological search filters are a combination of search terms that ensure the retrieval of a particular type of article [10]. Terwee and colleagues [10] developed a methodological search filter to identify psychometric articles indexed in PubMed for tools that measure patient-reported outcomes (PROM) (tools measuring health outcomes and daily functions reported by patients [10, 11]) and potentially non-PROM measurement tools. This outcome measurement properties (OMP) filter has a sensitive and a precise version, hereafter referred to as the “sensitive filter” or the “precise filter”. The sensitive filter was designed for systematic reviews and the precise filter for gathering psychometric articles without the need for a comprehensive search [10]. It is important to validate search filters to ensure that there is a balance between sensitivity (ability to identify relevant articles) and precision (minimize number of records) [10, 12].

We completed a series of three large international studies to develop the Implementation in CONtext (ICON) framework [1,2,3]: (1) a concept analysis of context as reported in the international published literature (n = 70 articles) [1]; (2) an analysis of 145 interviews conducted with a wide range of healthcare providers in multiple healthcare settings for elements of context relevant to research use in clinical practice [2]; (3) a qualitative interview study with 39 health system stakeholder (individuals responsible for change in a healthcare organization) in 4 countries to elicit their perceptions of important elements of context for implementation [3]. The findings from these studies were then triangulated to produce ICON, a meta-framework where context is conceptualized in three levels (micro, meso, and macro) that are divided into five domains, 22 core attributes and 108 features [13]. We aim to create a repository of tools that measure ICON attributes, and to summarize the psychometric properties of these tools. However, the heterogeneity in the constructs and populations associated with ICON attributes made it difficult to construct a search strategy, guided by traditional systematic review methods [14, 15], that had a good balance between sensitivity and precision. We used Terwee and colleagues’ [10] filter (both versions) to refine our search for psychometric articles. Since a tool name was required to locate psychometric articles of specific tools, we performed citation searches [16, 17] to locate articles for unnamed tools (Additional File 1: search method definitions).

The purpose of this research note is to describe our experience of using an OMP filter and citation searching to identify psychometric articles for 31 tools [18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48] that measure ICON attributes. Our objectives are to: (1) provide an overview of our experience of using an OMP filter (both versions) and citation searching; (2) compare the precise filter with or without reference checking with citation searching according to the number of records located and time required to complete article selection, precision, and sensitivity; (3) discuss the benefits of and the insights garnered from using an OMP filter and citation searching.


In consultation with a health information specialist (TR), we performed searches in PubMed from inception to July/December 2021 using the precise filter combined with the tool’s name (Additional File 2: search terms). We used the sensitive filter only if the search using the precise filter produced zero results. We performed citation searches using an indexed development article when: (1) tools had no name; or (2) the OMP (both versions) filter search did not locate a relevant article. Citation searching was conducted in Scopus and Web of Science. We checked the reference lists of included articles to supplement our database searches [17, 49]. We (LKC, MG, WJS or VW) screened records and full text articles independently and in duplicate. Discrepancies were resolved through consensus or in consultation with a senior researcher. We included articles that aimed to evaluate the psychometric properties of any of the 31 tools [18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48].

Nineteen of 31 (61.3%) tools were non-PROM [22, 24,25,26,27, 29,30,31, 33, 34, 36, 38, 39, 41,42,43,44, 46, 47], and the remaining 12 (38.7%) were PROM [18,19,20,21, 23, 28, 32, 35, 37, 40, 45, 48]. We conducted a search on PubMed using an OMP filter for 29 of 31 (93.5%) tools which had names; 12 PROM [18,19,20,21, 23, 28, 32, 35, 37, 40, 45, 48] and 17 non-PROM [22, 25,26,27, 29,30,31, 33, 34, 36, 38, 39, 41,42,43,44, 46]. We used the precise filter for 28 of 29 named tools [18,19,20,21,22,23, 25,26,27,28,29,30,31,32,33,34,35, 37,38,39,40,41,42,43,44,45,46, 48], and the sensitive filter for one tool [36] (precise version found zero records). We performed a citation search on 2 of 31 tools (6.5%) (both non-PROM [24, 47]) because they were unnamed.

Methods for sub-analysis

We compared the search results of the precise filter (alone or combined with reference checking), with citation searching in a sub-analysis of six randomly selected tools (three PROM [28, 35, 40] and three nonPROM [26, 31, 43]). We compared the search methods according to number of records identified, time to complete record screening, precision, and sensitivity. One researcher appraised the inclusion of articles for the citation searches. Precision is the total number of included articles divided by the total number of records identified by a search method [14]. Sensitivity is the number of included articles divided by the total number of unique articles included across the three search methods [14].


We found at least one relevant article for 27 of 31 (87.1%) tools [20,21,22,23,24,25,26,27,28, 30,31,32,33, 35,36,37,38,39,40,41,42,43,44,45,46,47,48] (Table 1: searches and search results). Across all search methods, we screened 10,303 records, appraised 546 full-text articles for eligibility and included 150 articles (Additional file 3 and Additional file 4: PRISMA diagram and search results for each tool respectively).

Table 1 Description of search results according to number of tools

OMP filter (N = 29 named tools) and reference checks (N = 22 tools)

Our searches using the precise filter successfully identified 130 relevant articles across 22 of 29 (75.9%) named tools [20,21,22,23, 26,27,28, 30,31,32,33, 35, 37, 39,40,41,42,43,44,45,46, 48]. We identified a relevant article for a slightly higher proportion of PROMS (10 of 12; 83.3%) [20, 21, 23, 28, 32, 35, 37, 40, 45, 48] than non-PROMS (12 of 17; 70.6%) [22, 26, 27, 30, 31, 33, 39, 41,42,43,44, 46]. Our OMP (both versions) filter search did not locate a relevant article for 7 of 29 tools (24.1%), a larger proportion of non-PROM (5 of 17; 41.7%) [25, 29, 34, 36, 38] than PROM (2 of 12; 16.7%) [18, 19].

By searching the reference lists of included psychometric articles, we found 12 additional psychometric articles [43, 50,51,52,53,54,55,56,57,58,59,60] for 5 (two PROM [21, 45] and three non-PROM [30, 39, 43]) of the 22 (22.7%) tools identified using the precise filter. Ten of the 12 (83.3%) articles (one article [50] for a PROM [45] and nine articles [43, 51,52,53, 56,57,58,59,60] across three non-PROMs [30, 39, 43]) were not indexed in PubMed. The remaining two articles [54, 55] (for one PROM [21]) were indexed in PubMed but not captured by the precise filter search.

Citation searches (N = 9 tools) and reference checks (N = 5 tools)

Citation searches were performed using the index tool development article for 9 of 31 (29.0%) tools (two PROM [18, 19] and seven non-PROM [24, 25, 29, 34, 36, 38, 47]). Citation searching retrieved eight psychometric articles across 5 of 9 (55.6%) [24, 25, 36, 38, 47] tools. We did not locate additional articles from reference checking. We were unable to perform a citation search for 3 of 9 (33.3%) tools (two PROM [18, 19] and one non-PROM [29]) because the development study was in a book or an online report and not indexed on Scopus or Web of Science. With citation searching we did not locate a psychometric article for the non-PROM [34] tool.

Sub-analysis results

Amongst a subset of six tools, the precise filter alone identified 500 records, while the precise filter search combined with reference checks located 1882 records, and citation searching found 2340 records (Table 2: detailed sub-analysis results). The precise filter search alone was also the most time-efficient method in our sub-analysis. We completed screening of records and full-text articles for the six tools four times faster when using the precise filter alone, compared to citation searches (~ 350.5 min versus 1386 min, respectively). The addition of reference checks to the precise filter search increased the time required for article selection (~ 1079.8 min) but was still more time efficient than citation searching.

Table 2 Sub-group Analysis of OMP filter (with or without reference checking) and Citation Search

The precise filter combined with reference checks located the most included articles (n = 56 (96.6%)). The precise filter when combined with reference checking was the most sensitive search method (mean, standard deviation = 96.7%, 8.2%) and the precise filter alone was the most precise search method (precision = 10.8%). The precise filter (alone or combined with reference checks) was more sensitive and precise when searching for psychometric articles for the three PROM [28, 35, 40] than for the three non-PROM [26, 31, 43] tools.

The precise filter in combination with reference checks identified nine articles, across four tools [26, 28, 40, 43]; all these articles were previously unidentified through the citation search (Additional file 5: missed articles and reasons they were missed by one of the methods). One [61] of 9 (11.1%) articles relating to one tool [26] was not identified by the citation search because an article [62] other than the development article was cited. Two of 9 (22.2%) articles [51, 63] relating to two tools [40, 43] were missed by our citation search because they were not indexed as a “cited” article in Scopus or Web of Science even though the authors referenced the article we used for citation searching. Two of 9 (22.2%) articles [64, 65] relating to two tools [28, 43] were not identified by citation searching because they were earlier reports about the tools’ development and were not indexed in either Scopus or Web of Science. Four of 9 (44.4%) articles [66,67,68,69] relating to two tools [28, 40] were missed by citation searching because the authors cited a different article reporting on the tool’s development from the one we used in our citation search. Additional citation searches were performed for these two tools [28, 40] using the newly discovered development articles [65, 70].

In summary, the additional searches did not change the precision of citation searching across the six tools (2.1%) but did increase the overall mean sensitivity of citation searching from 93.3% to 94.1% (Additional file 6: sub-analysis results with the additional citation searches considered). In contrast, the citation search captured two articles [71, 72], relating to one tool [43], that were not located by the precise filter search because they were not indexed in PubMed.


Our study provides a greater understanding of the benefits of using the precise filter and conducting citation searches (see Table 3).

Table 3 Benefits and insights gained from using the outcomes measurement properties filter and conducting a citation search to locate psychometric articles

OMP filter

The most important benefit of searching using the precise filter was the decreased time required to screen records. Based on our sub-analysis of six tools, searching with the precise filter alone proved to be the most precise method, and the combination of the precise filter with reference checks was the most sensitive search method. However, before using the OMP filter (both versions), the following should be considered: 1) the OMP filter cannot locate articles that are not indexed in PubMed; 2) the search for psychometric articles of particular tools is dependent on knowing the name of the tool; and 3) the unknown implications of recent changes to the PubMed interface [73, 74] (detailed in Table 3) on the OMP filter.

We found that the search conducted with the precise filter successfully identified more psychometric articles for PROM than non-PROM. However, further systematic research comparing the efficacy of the OMP filter at locating psychometric articles for PROM and non-PROM is needed to confirm our findings. Systematic testing of the OMP filter in the context of recent developments in the field of measurement is an avenue for future research (e.g. how the addition of search terms related to content validity (relevance, comprehensiveness, comprehensibility [75, 76]) and pragmatic properties (usefulness, compatibility, acceptability, etc. [77]) affect sensitivity and specificity).

Citation searching

Citation searching was helpful in identifying psychometric articles of tools that had not been named or tools for which the search with the OMP filter (both versions) did not locate a psychometric article (the majority being non-PROMs). When planning a citation search to locate psychometric studies, the following should be considered: (1) tool development study might be reported in multiple articles; (2) multiple citation searches on several development articles may be more sensitive but increase the number of retrieved records; (3) citation searching is limited to the articles indexed in the utilized database(s); (4) citation searching may identify more records than the precise filter.


We did not intend to conduct a systematic evaluation of the OMP filter or citation searching. Furthermore, we did not evaluate other search methods that are used in measurement reviews (e.g., a traditional search strategy). Our sub-analysis, comparing the precise filter and citation searching, was based on only six tools. The references we used for citation searching were originally retrieved using the OMP filter and only one researcher screened records retrieved by citation searching. Considering these limitations, our findings may reflect trends in each methods’ search results, precision and sensitivity that require further investigation.


The precise filter decreased the number of records required for screening for our project. Based on a sample of six tools, searching with the precise filter alone was more precise, sensitive, and efficient than citation searching. More systematic research is required to evaluate the usefulness of OMP filters and other search methods in measurement reviews, especially reviews on tools that are developed/used across multiple disciplines and specialties and address a diverse range of theoretical constructs.

Availability of data and materials

The definition of search methods, search strategy, PRISMA diagram, the search results for each tool, reasons for discrepancies between search methods during the sub-analysis, changes in results with additional citation searches for the sub-analysis are provided as additional files. The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.


  1. Squires JE, Graham I, Bashir K, et al. Understanding context: a concept analysis. J Adv Nurs. 2019;75(12):3448–70.

    Article  PubMed  Google Scholar 

  2. Squires JE, Aloisio LD, Grimshaw JM, et al. Attributes of context relevant to healthcare professionals’ use of research evidence in clinical practice: a multi-study analysis. Implement Sci. 2019;14(1):52.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Squires JE, Hutchinson AM, Coughlin M, et al. Stakeholder perspectives of attributes and features of context relevant to knowledge translation in health settings: a multi-country analysis. Int J Health Plann Manage. 2021;11(8):1373.

    Google Scholar 

  4. Kivimäki M, Elovainio M. A short version of the team climate inventory: development and psychometric properties. J Occup Organ Psychol. 1999;72(2):241–6.

    Article  Google Scholar 

  5. Bowen S. A Guide to evaluation in health research. 2012; Accessed July 27. 2022.

  6. Straus SETJ, Bhattacharyya O, Zwarenstein M, Graham ID. Chapter 3 5 Monitoring knowledge useand evaluating outcomes. In: Straus SETJ, Graham ID, editors. Knowledge translation in health care: moving from evidence to practice. Toronto: Wiley; 2013.

    Chapter  Google Scholar 

  7. Polanin JR, Pigott TD, Espelage DL, Grotpeter JK. Best practice guidelines for abstract screening large-evidence systematic reviews and meta-analyses. Res Synth Methods. 2019;10(3):330–42.

    Article  PubMed Central  Google Scholar 

  8. Mokkink LB, Prinsen C, Patrick DL, et al. COSMIN methodology for systematic reviews of patient-reported outcome measures (PROMs). User Manual. 2018;78(1):63.

    Google Scholar 

  9. The InterTASC Information specialists' sub-Group. ISSG search filter resource. 2006. Accessed Sep 7 2022.

  10. Terwee CB, Jansma EP, Riphagen II, de Vet HC. Development of a methodological pubmed search filter for finding studies on measurement properties of measurement instruments. Qual Life Res. 2009;18(8):1115–23.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Quittner AL, Nicolais CJ, Saez-Flores E. Integrating patient-reported outcomes into research and clinical practice. Kendig’s Disord Respir Tract Children. 2019;231–240:233.

    Google Scholar 

  12. Durao S, Kredo T, Volmink J. Validation of a search strategy to identify nutrition trials in pubMed using the relative recall method. J Clin Epidemiol. 2015;68(6):610–6.

    Article  PubMed  Google Scholar 

  13. Squires JE, Brehaut J, Curran J, et al. Understanding context in knowledge translation: development and consensus of a conceptual framework (phase I). In: Canadian Institutes of Health Research 2014–2015.

  14. Lefebvre C, Glanville J, Briscoe S, et al. Searching for and selecting studies. Cochrane Handbook Syst Rev Interv. 2019;23:67–107.

    Article  Google Scholar 

  15. Aromataris E, Munn Z. Chapter 1: JBI Systematic Reviews. JBI Manual for Evidence Synthesis 2020. Accessed 2 Oct 2022

  16. Wright K, Golder S, Rodriguez-Lopez R. Citation searching: a systematic review case study of multiple risk behaviour interventions. BMC Med Res Methodol. 2014;14(1):1–8.

    Article  CAS  Google Scholar 

  17. Lefebvre C, Glanville J, Briscoe S, et al. Technical supplement to chapter 4: searching for and selecting studies. cochrane handbook for systematic reviews of interventions version 6.3 (updated February 2022) 2022. Accessed 15 Aug 2022.

  18. Webster G. Final report on the patient satisfaction questionnaire project. 1989.

  19. Weiss K. Measuring the reliability and validity of the art of medicine survey. Denver: Denver (CO). HealthCare Research Inc 2001.

  20. Coleman EA, Mahoney E, Parry C. Assessing the quality of preparation for posthospital care from the patient’s perspective: the care transitions measure. Med Care. 2005;43(3):246–55.

    Article  PubMed  Google Scholar 

  21. Barr PJ, Thompson R, Walsh T, Grande SW, Ozanne EM, Elwyn G. The psychometric properties of collaborate: a fast and frugal patient-reported measure of the shared decision-making process. J Med Internet Res. 2014;6(1):e2.

    Article  Google Scholar 

  22. King G, Servais M, Kertoy M, et al. A measure of community members’ perceptions of the impacts of research partnerships in health and social services. Eval Program Plann. 2009;32(3):289–99.

    Article  PubMed  Google Scholar 

  23. Campbell J, Narayanan A, Burford B, Greco M. Validation of a multi-source feedback tool for use in general practice. Educ Prim Care. 2010;21(3):165–79.

    Article  PubMed  Google Scholar 

  24. Ehrhart MG. Leadership and procedural justice climate as antecedents of unit-level organizational citizenship behavior. Pers Psychol. 2004;57(1):61–94.

    Article  Google Scholar 

  25. Carless SA, Wearing AJ, Mann L. A short measure of transformational leadership. J Bus Psychol. 2000;14(3):389–405.

    Article  Google Scholar 

  26. Aarons GA, Ehrhart MG, Farahnak LR. The implementation leadership scale (ILS): development of a brief measure of unit level implementation leadership. Implement Sci. 2014;9(1):45.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Körner M, Wirtz MA. Development and psychometric properties of a scale for measuring internal participation from a patient and health care professional perspective. BMC Health Serv Res. 2013;13(1):1–11.

    Article  Google Scholar 

  28. Stewart AL, Nápoles-Springer AM, Gregorich SE, Santoyo-Olsson J. Interpersonal processes of care survey: patient-reported measures for diverse groups. Health Serv Res. 2007;42(3 Pt 1):1235–56.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Stogdill RM. Manual for the leadership behaviour development questionnaire: form XII Columbus: Bureau of business research. Ohio: Ohio State University; 1963.

    Google Scholar 

  30. Bass BM, Avolio BJ. Transformational leadership development: Manual for the multifactor leadership questionnaire. Palo Alto: Consulting Psychologists Press; 1990.

    Google Scholar 

  31. Kenaszchuk C, Reeves S, Nicholas D, Zwarenstein M. Validity and reliability of a multiple-group measurement scale for interprofessional collaboration. BMC Health Serv Res. 2010.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Frank C, Asp M, Fridlund B, Baigi A. Questionnaire for patient participation in emergency departments: development and psychometric testing. J Adv Nurs. 2011;67(3):643–51.

    Article  PubMed  Google Scholar 

  33. Nahm E, Resnick B, Mills ME. Development and pilot-testing of the perceived health web site usability questionnaire (PHWSUQ) for older adults. Stud Health Technol Inform. 2006;122:38.

    PubMed  Google Scholar 

  34. Hatcher M. Survey of acute care hospitals in the United States relative to technology usage and technology transfer. J Med Syst. 1997;21(5):323–36.

    Article  CAS  PubMed  Google Scholar 

  35. Risser NL, Batey V. Development of an instrument to measure patient satisfaction with nurses and nursing care in primary care settings. Nurs Res. 1975;24(1):45–52.

    Article  CAS  PubMed  Google Scholar 

  36. Welbourne TM, Johnson DE, Erez A. The role-based performance scale: Validity analysis of a theory-based measure. Acad Manag J. 1998;41(5):540–55.

    Article  Google Scholar 

  37. Scholl I, Kriston L, Dirmaier J, Buchholz A, Härter M. Development and psychometric properties of the shared decision making questionnaire—physician version (SDM-Q-Doc). Patient Educ Couns. 2012;88(2):284–90.

    Article  PubMed  Google Scholar 

  38. Barbuto JE Jr, Wheeler DW. Scale development and construct clarification of servant leadership. Group Organ Manag. 2006;31(3):300–26.

    Article  Google Scholar 

  39. van Dierendonck D, Nuijten I. The servant leadership survey: development and validation of a multidimensional measure. J Bus Psychol. 2011;26(3):249–67.

    Article  PubMed  Google Scholar 

  40. Kriston L, Scholl I, Hölzel L, Simon D, Loh A, Härter M, The 9-item Shared Decision Making Questionnaire (SDM-Q-9). Development and psychometric properties in a primary care sample. Patient Educ Couns. 2010;80(1):94–9.

    Article  PubMed  Google Scholar 

  41. Singer S, Meterko M, Baker L, Gaba D, Falwell A, Rosen A. Workforce perceptions of hospital safety culture: development and validation of the patient safety climate in healthcare organizations survey. Health Serv Res. 2007;42(5):1999–2021.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Schutz AL, Counte MA, Meurer S. Development of a patient safety culture measurement tool for ambulatory health care settings: analysis of content validity. Health Care Manag Sci. 2007;10(2):139–49.

    Article  PubMed  Google Scholar 

  43. Anderson NR, West MA. Measuring climate for work group innovation: Development and validation of the team climate inventory. J Organ Behav. 1998;19(3):235–58.

    Article  Google Scholar 

  44. Berk RA, Berg J, Mortimer R, Walton-Moss B, Yeo TP. Measuring the effectiveness of faculty mentoring relationships. Acad Med. 2005;80(1):66–71.

    Article  PubMed  Google Scholar 

  45. Glasgow RE, Wagner EH, Schaefer J, Mahoney LD, Reid RJ, Greene SM. Development and validation of the patient assessment of chronic illness care (PACIC). Med Care. 2005;43(5):436–44.

    Article  PubMed  Google Scholar 

  46. Jones J, Barry M. Developing a scale to measure trust in health promotion partnerships. Health Promot Int. 2011;26(4):484–91.

    Article  CAS  PubMed  Google Scholar 

  47. Costa AC, Anderson N. Measuring trust in teams: development and validation of a multifaceted measure of formative and reflective indicators of team trust. Eur J Work Organ Psychol. 2011;20(1):119–54.

    Article  Google Scholar 

  48. O’Cathain A, Knowles E, Nicholl J. Measuring patients’ experiences and views of the emergency and urgent care system: psychometric testing of the urgent care system questionnaire. BMJ Qual Saf. 2011;20(2):134–40.

    Article  PubMed  Google Scholar 

  49. Horsley T, Dingwall O, Sampson M. Checking reference lists to find additional studies for systematic reviews. Cochrane Database Syst Rev. 2011.

    Article  PubMed  PubMed Central  Google Scholar 

  50. Cramm JM, Nieboer AP. Development and validation of the older patient assessment of chronic illness care (O-PACIC) scale after hospitalization. Soc Indic Res. 2014;116(3):959–69.

    Article  Google Scholar 

  51. Kivimäki M, George K, Elovainio M, Thomson L, Kalliomäki-Levanto T, Heikkilä A. The team climate inventory (TCI)—four or five factors? Testing the structure of TCI in samples of low and high complexity jobs. J Occup Organ Psychol. 1997;70(4):375–89.

    Article  Google Scholar 

  52. Kivimaki M, Elovainio M. A short version of the team climate inventory: development and psychometric properties. J Occup Organ Psychol. 1998;72:241.

    Article  Google Scholar 

  53. Bobbio A, Dierendonck DV, Manganelli AM. Servant leadership in Italy and its relation to organizational variables. Leadership. 2012;8(3):229–43.

    Article  Google Scholar 

  54. Elwyn G, Barr PJ, Grande SW, Thompson R, Walsh T, Ozanne EM. Developing CollaboRATE: a fast and frugal patient-reported measure of shared decision making in clinical encounters. Patient Educ Couns. 2013;93(1):102–7.

    Article  PubMed  Google Scholar 

  55. Forcino RC, Bustamante N, Thompson R, et al. Developing and pilot testing a Spanish translation of collaboRATE for use in the United States. PLoS ONE. 2016;11(12): e0168538.

    Article  PubMed  PubMed Central  Google Scholar 

  56. Bycio P, Hackett RD, Allen JS. Further assessments of Bass’s (1985) conceptualization of transactional and transformational leadership. J Appl Psychol. 1995;80(4):468.

    Article  Google Scholar 

  57. Den Hartog DN, Van Muijen JJ, Koopman PL. Transactional versus transformational leadership: an analysis of the MLQ. J Occup Organ Psychol. 1997;70(1):19–34.

    Article  Google Scholar 

  58. Garman AN, Davis-Lenane D, Corrigan PW. Factor structure of the transformational leadership model in human service teams. J Organ Behav. 2003;24(6):803–12.

    Article  Google Scholar 

  59. Tejeda MJ, Scandura TA, Pillai R. The MLQ revisited: Psychometric properties and recommendations. Leadersh Q. 2001;12(1):31–52.

    Article  Google Scholar 

  60. Vandenberghe C, Stordeur S, D’hoore W. Transactional and transformational leadership in nursing: structural validity and substantive relationships. Eur J Psychol Assess. 2002;18(1):16.

    Article  Google Scholar 

  61. Lyon AR, Cook CR, Brown EC, et al. Assessing organizational implementation context in the education sector: confirmatory factor analysis of measures of implementation leadership, climate, and citizenship. Implement Sci. 2018;13(1):1–14.

    Article  Google Scholar 

  62. Finn NK, Torres EM, Ehrhart MG, Roesch SC, Aarons GA. Cross-validation of the implementation leadership scale (ILS) in child welfare service organizations. Child Maltreat. 2016;21(3):250–5.

    Article  PubMed  Google Scholar 

  63. Goto Y, Miura H, Son D, et al. Psychometric evaluation of the Japanese 9-item shared decision-making questionnaire and its association with decision conflict and patient factors in Japanese primary care. JMA journal. 2020;3(3):208–15.

    Article  PubMed  PubMed Central  Google Scholar 

  64. Anderson N, West MA. The team climate inventory: development of the TCI and its applications in teambuilding for innovativeness. Eur J Work Organ Psychol. 1996;5(1):53–66.

    Article  Google Scholar 

  65. Stewart AL, Nápoles-Springer A, Pérez-Stable EJ. Interpersonal processes of care in diverse populations. Milbank Q. 1999;77(3):305–39.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  66. Kasper J, Heesen C, Köpke S, Fulcher G, Geiger F. Patients’ and observers’ perceptions of involvement differ validation study on inter-relating measures for shared decision making. PLoS One. 2011;6(10):26255.

    Article  Google Scholar 

  67. Beaulieu MD, Haggerty JL, Beaulieu C, et al. Interpersonal communication from the patient perspective: comparison of primary healthcare evaluation instruments. Healthc Policy. 2011;7:108–23.

    PubMed  PubMed Central  Google Scholar 

  68. Haggerty JL, Beaulieu C, Lawson B, Santor DA, Fournier M, Burge F. What patients tell us about primary healthcare evaluation instruments: response formats, bad questions and missing pieces. Healthc Policy. 2011;7:66–78.

    PubMed  PubMed Central  Google Scholar 

  69. Nápoles-Springer AM, Santoyo-Olsson J, O’Brien H, Stewart AL. Using cognitive interviews to develop surveys in diverse populations. Med Care. 2006;44(11 Suppl 3):S21-30.

    Article  PubMed  Google Scholar 

  70. Simon D, Schorr G, Wirtz M, et al. Development and first validation of the shared decision-making questionnaire (SDM-Q). Patient Educ Couns. 2006;63(3):319–27.

    Article  CAS  PubMed  Google Scholar 

  71. Antino M, Gil-Rodriguez F, Martí M, Barrasa A, Borzillo S. Development and validation of the Spanish version of the team climate inventory: a measurement invariance test. Annu Rev Psychol. 2014;30(2):597–607.

    Google Scholar 

  72. Tseng H-M, Liu F-C, West MA. The team climate inventory (TCI) a psychometric test on a Taiwanese sample of work groups. Small Group Res. 2009;40(4):465–82.

    Article  Google Scholar 

  73. National Library of Medicine. New pubmed to replace legacy PubMed in Mid-May. NLM Tech Bull. 2020. Accessed 15 Aug 2022.

  74. National Library of Medicine. New pubmed transition FAQs. 2021. Accessed 14 Aug 2022.

  75. Terwee CB, Prinsen CA, Chiarotto A, et al. COSMIN methodology for evaluating the content validity of patient-reported outcome measures: a Delphi study. Qual Life Res. 2018;27(5):1159–70.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  76. Terwee CB, Prinsen C, Chiarotto A, et al. COSMIN methodology for assessing the content validity of PROMs–user manual. Amsterdam: VU University Medical Center; 2018.

    Google Scholar 

  77. Lewis CC, Mettert KD, Stanick CF, et al. The psychometric and pragmatic evidence rating scale (PAPERS) for measure development and evaluation. Implement Res Pract. 2021;2:26334895211037390.

    Google Scholar 

  78. Bakshi AB, Wee SL, Tay C, et al. Validation of the care transition measure in multi-ethnic South-East Asia in Singapore. BMC Health Serv Res. 2012;12:256.

    Article  PubMed  PubMed Central  Google Scholar 

  79. Cao X, Chen L, Diao Y, Tian L, Liu W, Jiang X. Validity and reliability of the Chinese version of the care transition measure. PLoS ONE. 2015;10(5): e0127403.

    Article  PubMed  PubMed Central  Google Scholar 

  80. Coleman EA, Smith JD, Frank JC, Eilertsen TB, Thiare JN, Kramer AM. Development and testing of a measure designed to assess the quality of care transitions. Int J Integr Care. 2002;2: e02.

    Article  PubMed  PubMed Central  Google Scholar 

  81. Shadmi E, Zisberg A, Coleman EA. Translation and validation of the care transition measure into Hebrew and Arabic. Int J Qual Health Care. 2009;21(2):97–102.

    Article  PubMed  Google Scholar 

  82. Charalambous A. Validation and test-retest reliability of the Risser patient satisfaction scale in Cyprus. J Nurs Manag. 2010;18(1):61–9.

    Article  PubMed  Google Scholar 

  83. Charalambous A, Adamakidou T. Risser patient satisfaction scale: a validation study in Greek cancer patients. BMC Nurs. 2012;11:27.

    Article  PubMed  PubMed Central  Google Scholar 

  84. Ventura MR, Fox RN, Corley MC, Mercurio SM. A patient satisfaction measure as a criterion to evaluate primary nursing. Nurs Res. 1982;31(4):226–30.

    Article  CAS  PubMed  Google Scholar 

  85. Biering P, Becker H, Calvin A, Grobe SJ. Casting light on the concept of patient satisfaction by studying the construct validity and the sensitivity of a questionnaire. Int J Health Care Qual Assur Inc Leadersh Health Serv. 2006;19(2–3):246–58.

    Article  PubMed  Google Scholar 

  86. Yellen E, Davis GC, Ricard R. The measurement of patient satisfaction. J Nurs Care Qual. 2002;16(4):23–9.

    Article  PubMed  Google Scholar 

  87. Stanick CF, Halko HM, Nolen EA, et al. Pragmatic measures for implementation research: development of the psychometric and pragmatic evidence rating scale. Transl Behav Med. 2021;11(1):11–20.

    Article  PubMed  Google Scholar 

Download references


JES holds a University Research Chair in Health Evidence Implementation. AMH holds a Chair in Nursing at the Centre for Quality and Patient Safety Research at Barwon Health Partnership and is the Director for the Centre for Quality and Patient Safety Research (QPS). IDG holds a CIHR Foundation Grant (FDN# 143237) in Integrated Knowledge Translation.


This research was supported by a Canadian Institutes of Health Research (CIHR) Project Grant (Award #: 426122). The CIHR had no role in the study’s design, conduct and reporting.

Author information

Authors and Affiliations



All the authors (JES, WJS, AMH, TR, IDG, VW, LKC, MG) participated in the conception of the study, its method and analysis. WJS conducted the searches on PubMed using the OMP (precise and sensitive) filter and the citation searches. WJS, VW, LKC, and MG conducted the title and abstract screening, located full text articles, completed full text screening. WJS analyzed the data with consultation from all the authors. WJS drafted the manuscript. All the authors (JES, WJS, AMH, TR, IDG, VW, LKC, MG) critically reviewed and revised the manuscript regarding its content and approved of the version that is to be published and agree to be accountable for all aspects of the work. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Janet E. Squires.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1:

Definition of search methods.

Additional file 2:

Tool names and search terms combined with terwee filter.

Additional file 3:

PRISMA diagram of identified records screened and full texts appraised.

Additional file 4:

Search results breakdown for each tool.

Additional file 5:

Reasons for discrepancies between the precise outcomes measurement properties filter search and citation sear.

Additional file 6:

Sub-group analysis of terwee filter (with or without reference checking) and citation search.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Santos, W.J., Hutchinson, A.M., Rader, T. et al. Insights from using an outcomes measurement properties search filter and conducting citation searches to locate psychometric articles of tools used to measure context attributes. BMC Res Notes 16, 34 (2023).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI:


  • Psychometric
  • Context
  • Filter
  • Outcomes
  • Review
  • Search