Received date: January 27, 2017; Accepted date: February 06, 2017; Published date: February 08, 2017
Citation: Dikken J, Hoogerduijn JG, Klaassen S, Schuurmans MJ (2017) Construct Validity and Reliability of the Dutch Older Patient in Acute Care Survey (OPACS), Measuring Nurses’ Attitude towards Older Patients. J Gerontol Geriatr Res 6:393. doi: 10.4172/2167-7182.1000393
Copyright: © 2017 Dikken J, et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Visit for more related articles at Journal of Gerontology & Geriatric Research
Background: In clinical practice, nurses’ attitudes regarding older patients are important in relation to quality of care. The Older People in Acute Care Survey (OPACS) is an instrument measuring hospital nurses attitudes regarding older patients and validated in Australia and the USA. The OPACS is translated in Dutch language and content validity of this translation is previously assessed, presenting questionable results. Measurement instruments, however, cannot be “validated” based on content validity evidence alone. Judgmental evidence and statistical analysis should be combined to fully evaluate content domain definition and representation and guide further development.
Objective: Assess structural validity and reliability to fully evaluate the OPACS for use in the Netherlands, complementing previous conducted content validity results.
Setting: Three general hospitals in the Netherlands.
Participants: 201 registered nurses.
Methods: Confirmatory factor analysis was used to assess the structural validity. Reliability was assessed with Cronbach’s alpha.
Results: OPACS Section A (measuring practice experiences) demonstrated to have acceptable structural validity- and good reliability outcomes after exclusion of two items (model fit: x² [df=537]=8475.40, p <0.001, CFI=0.96, TLI=0.96, RMSEA=0.21; Cronbach’s alpha=0.82). Section B (measuring general opinion) demonstrated to have inadequate structural validity outcomes (model fit: x² [df=1127]=9200.29, p<0.001, CFI=0.68, TLI=0.67, RMSEA=0.15). None of the items contributed significant to the factor and therefore no further analysis could be performed (range p(>|z|)= 0.551 -0 .788).
Conclusion: Even though structural validity for section A was acceptable, content validity scores of a majority of items in this subscale were low, resulting in questionable use of this subscale for the Dutch context. The findings of this study, in relation to the earlier findings regarding content validity, justify the conclusion that use of the Dutch OPACS in clinical practice and research is not recommended. Given these findings, future research should pursue the development or (cross-cultural) validation of other instruments measuring hospital nurses attitudes towards older patients for the Dutch cultural context. Furthermore, this study demonstrated the influence of cultural differences on measurement instruments and the need for rigorous research before using a measurement instrument in a new culture or context.
OPACS; Attitude; Experience; Opinion; Netherlands; Nurses; Hospital; Cross-cultural
Worldwide, people are aging . This demographic change results in an increase of older people admitted in hospitals. A growing number of registered nurses will encounter older patients in their daily work and a positive attitude is often promoted [2,3]. Healthcare professionals need to understand current attitudes regarding older patients when workforce strategies for promoting positive attitudes are to be implemented [4,5].
The Older Patient in Acute Care Survey (OPACS), developed in Australia, measures hospital nurses practice experiences and general opinion regarding older patients which are considered aspects of attitude [6,7]. The OPACS was developed using focus groups with 16 nurses discussing their experience of caring for older patients in the acute care setting. The final OPACS consisted of 86 items related to 13 different aspects influencing the nursing care of older patients. Verbal statements regarding these 13 aspects are scored on a 5 point Likert scale ranging from strongly disagree – strongly agree . The OPACS has been translated and content validity is assessed in the USA. Results demonstrated excellent content validity scores . Furthermore, structural validity and reliability outcomes for the American OPACS proved to be good . After translation towards the Dutch language, however, a majority of items were considered “not to be relevant” by experts, resulting in low content validity scores even though translation was considered good . Why the content was not considered relevant for the Netherlands, and whether or not this is reflected in the construct is unknown, making it difficult to adjust the OPACS to the Dutch context.
In literature, the concept of content validity has been controversial since its inception and it is described that although content validity is a fundamental requirement of all assessment instruments, measures cannot be considered “valid” based on content validity evidence alone . Both judgmental and statistical analysis of test content provide important information regarding content- and construct validity and both approaches have their limitations. Therefore it is recommended to use both types of analysis to fully evaluate content domain definition and representation [11-13]. With assessment of the structural validity and reliability of the Dutch OPACS, content validity results of a previous study will be complemented, resulting in a full evaluation of the OPACS content and use for the Dutch cultural context.
The aim of this study is therefore evaluating the construct validity and reliability of the Dutch OPACS, complementing previous study results.
This study followed a multicenter cross-sectional design.
Setting and subjects
Data of nurses from the Netherlands were derived over a six-month period. Registered nurses working in three general hospitals located in the middle of the Netherlands were recruited and included after informed consent was obtained. Nurses were invited to participate through e-mail from their ward manager, flyers, and a message on the online hospital communication boards. Nurses completed the Dutch OPACS and several questions regarding their socio- demographic characteristics online. The study was approved by the medical review board of the University Medical Center Utrecht, the Netherlands (METC protocol number: 14-345/C).
The OPACS is originally developed in Australia . It consists of two scales; section A measuring practical experience (36 items) and section B measuring general opinions towards older patient’s needs (50 items) on a 5 point Likert scale. The Australian OPACS demonstrated good face validity and high reliability scores (Kappa 0.76). The United States version of the OPACS showed a high Scale-Content Validity Index/universal agreement (S-CVI/ua) score (S-CVI/ua=0.92)  and good structural validity and excellent reliability scores (Cronbach’s alpha=0.93) . The American OPACS was translated and validated on content in the Netherlands, demonstrating positive translation but alarming content validation results (S-CVI/average 0.62) with major differences in rating of relevance between experts (S-CVI/ua=0.13). The same Dutch OPACS was used for data collection in the present study to assess the construct validity.
Confirmatory factor analysis (CFA) was used to assess the construct validity of the Dutch OPACS. The aim of CFA is to test a hypothesized factor structure or model and assess its fit to the data. Relations of indicators (observed variables) to factors (latent variables) as well as the correlations among the latter are tested in the measurement model .
First, missing values were assessed to determine whether list-wise deletion could be used. Then CFA was performed for Dutch OPACS section A and section B by testing several models using Lavaan: an R package for structural equation modeling . Evaluation of each model was based on considering a variety of fit measures: the X² minimum fit function test; the Comparative Fit Index (CFI); the Tucker Lewis Index (TLI) and the Root Mean Square Error of Approximation (RMSEA). Values of >0.95 for the CFI/TLI indicate a good fitting model. The RMSEA should be <0.06 indicating a good fitting model [14,16]. All analysis are performed using R .
Of the participating sample, only complete cases were included in this study (73.6%). The socio-demographic characteristics of respondents with missing values were not significantly different from complete cases (all p>0.05). Socio-demographic characteristics are presented in (Table 1).
|NL respondents (n= 201)|
|Gender, female n (%)||185 (92.0)|
|Age, mean (SD)||38.7 (12.3)|
|Highest education, n (%)|
|Missing, n||2 (1.0)|
|Years of experience, mean (SD)||16.0 (12.0)|
|Hours a week working, mean (SD)||26.7 (8.8)|
AAS=An Associate of Science in Nursing BSN=Bachelor of science in nursing, PhD=completed a doctoral program in nursing or related fields, SD=Standard deviation
Table 1: Sample characteristics.
Validity and reliability of the Dutch OPACS section A (Practice experiences)
In Table 2, the different CFA models assessing section A (practice experiences) are presented. The unidimensional model for the Dutch OPACS section A (Model 1) did fit the data (x2 [df=594]=9088.53, p<0.001, CFI=0.96, TLI=0.96, RMSEA=0.21). There were no items with a negative loading on the factor “practice experiences”. However, 2 items (items 20, 22) did not significantly contribute to the factor and were therefore excluded. As expected, exclusion of these two items did not worsen the model fit to the data in Model 2 (x2 [df=537]=8475.40, p<0.001, CFI= 0.96, TLI= 0.96, RMSEA= 0.21). Internal consistency was considered good (Cronbach’s alpha=0.82 [0.79 – 0.84]) (Table 2).
|Model||Model fit statistics||Items deleted|
|X2||df||p||CFI||TLI||RMSEA (95% CI)|
|Model 1. 36 items, only factor structure constrained||9088.53||594||<0.001||0.96||0.96||0.21 (0.20 - 0.21)||[20,22]|
|Model 2. 34 items, exclusion of non-significant loading of items on construct||8475.40||527||<0.001||0.96||0.96||0.21 (0.21 - 0.22)|
X2=Chi-square statistics, df=degree of freedom, CFI=Comparative Fit Index, TLI=Tucker Lewis Index, RMSAE=Root Mean Square Error of Approximation, CI=Confidence Interval.
Table 2: Confirmatory Factor Analysis model fit statistics for OPACS-NL section A (practice experiences) 36 items.
Validity and reliability of the Dutch OPACS section B (General opinion)
The same CFA model was used to assess the Dutch OPACS section B: general opinion. The unidimensional model for the Dutch OPACS section B (Model 1) did not fit the data well (x2 [df=1127]=9200.29, p<0.001, CFI=0.68, TLI=.67, RMSEA=0.15). When looking at the items separately to assess which items should be excluded to improve the model fit to the data, it appeared that none of them contributed significant to the factor (Table 3) and therefore no further analysis could be performed (range p(>|z|)=0.551 - 0.788).
None of the items contribute significant to the factor, see P (>|z|).
Table 3: Final item loadings and test statistics for the Dutch OPACS section B (generalopinion) 50 items.
This study assessed the construct validity and reliability of the Dutch OPACS measuring practice experiences and general opinion of hospital nurses regarding older patients.
The items of section A: practice experiences, demonstrated to measure one construct. Only two items did not contribute to the construct and were therefore excluded. None of the items in OPACS section B, contributed significant to the factor general opinion, meaning that none of the items measured the construct solely making it impossible to include good items and exclude bad items using statistics.
In a previous study by van Schelven et al. , low content validity scores for 20 (58.8%) of the 34 items were presented for subscale A: practice experiences. By assessment of items with low content validity, several cultural reasons were found explaining the low rating by experts. First, 26 items (72.2%) mentioned a difference in care giving between old and young patients with only 5 (19.2%) of these items considered relevant. Focusing on the difference in care giving between old and young patients undermines the Dutch vision that care should be adjusted to the need of the individual patient (the same basic principle for old and young) which is taught in education and in clinical practice in the Netherlands. For example the item: “I ask older patients if they have pain more often than I ask younger patients”, with total agreement reflecting a positive attitude, is considered not to be relevant because nurses should assess pain three times a day in every hospitalized patient regardless their age according to Dutch quality systems. Second, the relation between several items and “positive or negative attitude” was unclear for experts resulting in a questionable scoring system. For example, the question: “I am more likely to speak in simple language to an older patient than to a younger patient” with total agreement reflecting a positive attitude. However, language used by nurses should always be adjusted to the individual patient, not be based on age alone because this can lead to a feeling of stereotyping by the older patient influencing the perceived quality of care . Therefore, not agreeing on this item can also be explained as “good attitude” by nurses respecting the older patient and approach him/her as an adult. This makes it disputable what “good attitude” is in relation to the item as presented by the OPACS. Before this subscale can be used in clinical practice in the Netherlands, items should be reexamined, discussed and adjusted by experts on content. The number and form of adjustments needed is so rigorous that this will result in a new instrument which means that 1) it will not be comparable with the OPACS-US or any other existing instrument making cross-national comparisons impossible and 2) might not be worth the effort with other existing instruments possibly more suited to the Dutch culture in its origin. The study by van Schelven et al.  also presented low content validity scores for 60% of the items in section B: general opinion. Results from the content validity study by van Schelven , combined with this study suggest that the translated version of the OPACS in the Netherlands should not be used.
This study demonstrated that assessment of both content- and structural validity are necessary to fully comprehend the validity of an instrument in a particular culture . Content validity is considered a fundamental requirement . Our results support this, demonstrated by the Dutch OPACS section A which would be assess valid if only structural validity and reliability scores would have been conducted. Too often, instruments are tested only using quantitative tests to assess validity and reliability when used in different cultures, settings and groups. Our studies demonstrate that validity and reliability of instruments can differ substantial between countries emphasizing the importance for rigorous cross-cultural validation before an instrument should be used in clinical practice in different cultures and in research. Researchers should therefore always assess content validity and describe possible (cultural) differences on item and scale level, as this influences the results of the study conducted.
Some considerations regarding this study should be discussed. Missing data were not imputed and cases were excluded (even though missing values were completely at random) to maximize the validity of the item selection during the item reduction process. This is considered acceptable as no differences were found in characteristics between full cases and cases having missing values and performance of analysis was not affected by sample size. Furthermore, considering the response rate and sample size of nurses from the different centers, the representativeness (having an convenience sample) can be questioned and selection bias could have led to an overestimation of effect as nurses with interest in older patients are more likely to participate. However, this is not considered a problem as the primary focus was on structural validation of the Dutch OPACS and not an exploration of attitudes of Dutch hospital nurses in the Netherlands. Furthermore, no sample size problems were indicated in analysis of the data. Third, OPACS section A proved unidimensional. However, whether the same construct is measured in the United States as in the Netherlands is not assessed in this study. Measurement invariance between items should always be assessed before comparisons between countries can be performed . With regards to the Dutch OPACS, it is likely that the Dutch subscale measures a different construct taking content validity results into account.
In conclusion, in clinical practice, identifying attitude problems is an important step to improve the quality of care for older patients . However, it is important that valid and reliable instruments are used to do so. The results from this study cannot justify the use of the Dutch OPACS in clinical practice and/or research. Even though section A (measuring practice experiences) demonstrated to have good structural validity results, items measuring practice experiences are considered unclear in interpretation and scoring and therefore not ready for use in clinical practice and research in the Netherlands. Section B (measuring general opinion) also demonstrated not to be applicable for use in the Netherlands as a result of low structural validity and reliability. Although section A might have some pointers for developing a new instrument, it might not worth the effort having other instruments potentially more suited to the Dutch culture in its origin.
This work was supported by the Foundation of Innovation Alliance, Regional Attention and Action for Knowledge circulation (SIA RAAK) and by an internal grant of the University of Applied Sciences Utrecht, the Netherlands.